Steve's Evolution Payroll Blog

Evo error "Out of resources" caused by bad Windows Update

A recent Windows Update from Microsoft has broken something in Evolution, provoking "Out of System Resources" (or sometimes just "Out of Resources") messages when running Evolution reports. These could show up with full error info:

Or with just a single unhelpful dialog box:

However manifested, these errors prevent the report from completing, and restarting Evolution (or the computer) has no impact.

This is caused not by anything Evolution is doing wrong (and is completely unrelated to Evolution's 64-bit readiness), but by a bad Windows Update by Microsoft; this blog post describes in more detail how to work around it by either removing the update, or applying a hotfix.

Microsoft was attempting to fix two legitimate security issues (CVE-2021-1640 and CVE-2021-26878), which were elevation of privilege vulnerabilities in the Windows print spooler; these are serious and need to be fixed.

These fixes broke a lot of things besides Evolution, and in some cases—depending on your specific print drivers installed—can cause full BSOD ("blue screen of death") Windows crashes.

Removing the bad updates

Fortunately, it's relatively easy to remove the errant updates in question until they are re-released without the bugs.

First: be sure to identify which computer this it's happening on (and which one has the bad update).

If you see the full error trace, then look at the "Computer Name" field, which will either be the name of your own desktop, or the name of one of the Evolution middle-tier servers (if the "user name" field shows system, this is almost certainly a middle-tier Evo server).

But if you see the small popup dialog box, then it's your own desktop that has the bad update and not the server.

In either case, you or your IT folks should go into Windows Update history and uninstall the update found in the list here based on the operating system. All updates are identified with KB (Knowledge Base) article numbers.

Windows 7 SP1 and Windows Server 2008 R2

• Monthly Rollup: KB5000841
• Security-only: KB5000851

Windows 8.1 and Windows Server 2012 R2

• Monthly Rollup: KB5000848
• Security-only: KB5000822

Windows 10 version 1909

• Support Page: KB5000808

Windows 10 version 2004 and 20H2

• Support Page: KB5000802

If you see any of these KB numbers in your Windows Update history, remove them.

Note: users on a corporate domain network might not have the permissions to do this; see your local IT department for this.

Alternate: apply an updated Microsoft Update ... maybe?

I've seen reports that Microsoft has released some out-of-band builds (KB5001648 and KB5001649, at least) of the errant update that fixes this issue, but I'm getting conflicting information about which update applies to which OS, and I'm fairly sure that not all impacted operating systems have one of these fixes yet.

There are likely experts with Windows Updates that know how to drill down on the details and can be really sure to pick the right options, but I'm definitely not one of those, and it makes me nervous to experiment here.

So my best advice at the moment: if you're having the issue, just uninstall the errant update and try again during next month's Patch Tuesday, when—hopefully—Microsoft has sorted this out.

I'll update this post if I find more on-point information.

Evo GUI update problems re: 64-bit Readiness

A few Evolution GUI desktop users have reported problems with their version auto-update ("Version update has failed"), with an additional error message:

This appears only in certain update situations, appears only once, and is due to a minor mis-step in Asure's move to 64-bit readiness. It's unfortunate, but easy to work around.

A user receiving this message should use the standard Windows file explorer, navigate to the path C:\iSystems\Evolution\Client\ and create a new empty folder Win64 and then launch Evolution again. This new folder fills in the missing step, and the update should complete normally.

What's this about?

The Win64\ folder is preparing the way for 64-bit support coming down the road, and—as I understand it—one Evo update got this update logic wrong, so it will show this error. Subsequent software releases do it all correctly.

All AWS bureaus get all Evolution software releases every time, while self-hosted bureaus pick-and-choose, but the end payroll customer only updates themselves when they fire up the desktop client. No bureau has control over this, so some customers who live in Evolution every day get every update, while (perhaps) small monthly payrolls skip numerous intermediate updates.

This is unfortunate, and though I did start investigating which exact specific versions were involved—Asure was very helpful on this front—in practice it doesn't matter: if the customer gets this message, they just have to create the folder and move on.

Are we on/do we require 64 bits now???

No, not yet.

This is just paving the way for the future, and I'm surprised it's going as smoothly as it has been. Asure has put an enormous effort into what I can only call a Herculean task of upgrading this complex application.

Asure has stated that 64 bits on the desktop is supported at least through the end of June, though self-hosted bureaus with 32-bit servers have to start preparing now (I'm working with several bureaus on exactly this).

64-bit Evolution is coming soon!

Since the beginning of time, Evolution has been a 32-bit computer application, but more modern 64-bit CPUs have been available for more than a decade, leaving a great deal of enhanced performance out of reach. Evolution runs just fine on a 64-bit system, but won't take advantage of any of the 64-bit features (such as far more memory).

This is about to change, and it's unequivocally a good thing.

Note: this has no impact on web-only users, whether Evo Payroll, Employee Portal, or any of the Advanced HR properties.

Table of Contents:

• Evolution GUI 64-bit readiness notification
• The "Client Info" Tab
• Customer Upgrade Options
• Is the bureau responsible for this?

Initially the changes will be on the server side, and this only impacts you if you're self hosted and you have 32-bit servers; a few of my customers are in this position and are either upgrading, replacing servers, or migrating to AWS.

Evolution pods in AWS have always been 64 bits, so bureaus there don't have to think about the server part.

Evolution classic GUI desktop: 64-bit readiness notification

Upgrades to the desktop - the classic Evolution GUI client - won't happen until mid-year, but this is where it's far more likely to impact bureaus or payroll customers: there are certainly many users still running older 32-bit operating systems that will not support Evolution once the 64-bit desktop upgrade hits at the end of June: the application will simply fail to start.

Fortunately, Asure has provided a facility to warn users that they are running on a soon-to-be-unsupported operating system. When they launch Evolution.exe on a 32-bit machine, it will show this popup dialog box:

I've been told that for now, the user can check the [X] box to silence the warning, but after a certain point - I don't know when - that box will continue to pop up and can't be silenced. The goal here is to avoid an unhappy surprise when a user tries to do payroll and finds that they forgot about the warning from a few months ago, and now can't run Evolution. I hate unhappy surprises.

It's important to understand that the vast majority of users running 32-bit systems are also running unsupported operating systems (such as Windows 7 or even Windows XP), which are no longer receiving updates, including security updates. These users should have updated a long time ago.

Some users may have the ability to perform an in-place upgrade of their operating system, others will need to get a new machine, but this ought to be plenty of notice for everybody.

Kudos to Asure for providing this feature: the alternative is crafting an announcement to your customers with instructions for how to tell what kind of OS they are running, and then walking people through this over the phone, and it makes my head hurt thinking about it.

The "Client Info" tab

An additional feature can be found in the menu SB Admin - Maintenance in the Client Info tab, which shows remote user activity and their 64-bit readiness.

The important field here is Windows64; you're looking for a Y in this column to indicate 64 bit status: they will be able to run the updated Evolution even if they have an older operating system that really ought to be upgrades (Windows 7 or Windows 7).

But users who show up with Windows64=N will receive this popup at startup, and it would be good bureau practice to keep an eye on this and touch base with customers in this circumstance. Seen the next section for possible options.

Of additional note here is the IP Address column, which is somewhat misleading: this is the IP address of the computer on their own local network, and it's only going to be useful to their own IT department in locating that machine. This will virtually always be an RFC 1918 private address (aka: 192.168.X.X, 10.X.X.X, etc.), and it's specifically not going to be useful to find out which public IP they logged in from.

This Client Info tab is for 64-bit readiness, not for keeping an eye on general user activity or login history, though I suppose some future enhancement for this wider purpose might be possible.

Customers Upgrade Options

It's a certainty that computers with 32-bit operating systems will refuse to run the Evolution.exe classic desktop GUI by mid-year, so for customer in this position, what to do?

Users running 32-bit operating systems are already struggling under these limited machines, especially with a limit of 4 gigabytes of memory. This amount of RAM is possibly sufficient for very casual web surfing and email, but it's not enough for any meaningful business applications; when I specify machines for customers, I always start with a minimum of 8 gigabytes of RAM and go up from there depending on the intended use (I've had 32 gigabytes of RAM on my workstation for 5 years).

Options available to the customer:

• Get a new machine; this is probably the cleanest option and the customer was probably thinking about it already. This might start a chain of hand-me-down at the customer where the Evolution user gets the new machine, and the old machine goes to someone else.
• Reinstall a 64-bit operating system on the same machine; this requires that the underlying CPU supports 64 bits, and it might require a paid Windows 10 OS upgrade from Microsoft. I don't believe there's any way to convert in place, it requires a full clean install of the OS, so this will certainly requires help from their own IT people.
• Consider Evolution Payroll; users who can't or won't upgrade their computers might be able to switch to using the web-based Evo Payroll, though it's possible that their out-of-date computer is also running an out-of-date web browser. This might require investigation.

Who's responsible for this upgrade?

To answer one question that I'm sure is going to come up: is the bureau responsible for these required customer upgrades?

My opinion: No way.

The vast majority of customers with 32-bit machines are running obsolete and out-of-support operating systems that have not been receiving critical security updates in some time. These machines should have been replaced a long time ago, and there's simply no way this can reasonably be pinned on the bureau.

As I type this, it occurs to me that there's a fair case that Evolution should refuse to run on unsupported operating systems because it's just dangerous to run financial applications on machines that are more likely to get hacked (due to lack of security updates).

Only customers running 32-bit Windows 10 are able to play the "we have a current operating system" card, but these are not common, and my guess is most of them are also old machines that were previously upgraded from Windows 7. Microsoft offered a free Windows 7 → Windows 10 upgrade for a long time and repeatedly nagged the user about it.

It's unthinkable to me that anybody intentionally purchased a new machine in the last 5 years with a 32-bit operating system, but I guess the world is full of people who make unfortunate choices.

In any case, it's increasingly common that line-of-business software is going 64-bit only, and this has been the case for quite a while, and is completely reasonable for Evolution even if only for the availability of a larger address space.

Many Evolution users have struggled with "Out of memory" errors, especially for large clients, and this was strictly a limitation of the 32-bit address space: going to a 64 bits application means you can use all the memory in your machine.

By way of example, the outstanding Fusion 360 software I used for 3D modeling and design has been 64-bit only, and it's been out for almost 10 years.

I suppose it's inevitable that some customers will push back and/or throw a fit, but I don't believe the bureau (or Asure) is responsible for end customers having to upgrade their computers to perform an important business functions in the year 2021. 64 bits is not a new thing.

It seems likely to me that some customers might even leave a bureau over this, though I suspect they were already unhappy and were already looking for an excuse. Some bureaus might even welcome this for some customers :-)

All bureaus have made accommodations to customers who make unreasonable demands just to keep the peace; this would be one of them if you care to.

Report types in Evolution (S/B/C)

On the evolution-talk mailing list, there often comes up discussions about reports and how one bureau can't find the same report another bureau has mentioned. I hope to clear some of this up.

As far as I understand, there are three kinds of reports in Evolution:

"S" reports (universal)

These are System reports provided by Evolution itself - they are universally available to all bureaus, and each report has the same name and number everywhere. Example: S4015 is the "Direct Deposit Audit" report everywhere.

"B" reports (custom)

These are custom Bureau-wide reports created by a bureau and were manually loaded into Evolution; they do not come from Asure.

"C" reports (custom)

These are custom Client reports created by the bureau, but are attached to individual clients rather than applying to the bureau as a whole. These do not come from Asure either.

Creating / Obtaining custom reports

As noted, System reports are available to everybody but custom reports are not, so if you see mention of a B or C report on the list, you probably don't have it.

There are three ways to get a custom report:

• Write it yourself; there are several tools available for reporting writing, and at a certain point it may be worth developing report-writer expertise within your bureau just to service these requests.
• Hire it out; there are several excellent consultants who do custom-report development and are widely used in the Evolution community (Teri and Dana), and they are universally respected. Personally, I don't do reports stuff and have hired Teri to do this for me.
• Obtain from another bureau; some bureaus are happy to provide their custom reports to anybody, while others may ask you to share the cost that they paid to have it developed. Always start by asking nicely (privately) and be willing to dicker a bit.

I believe that in the past, iSystems/Asure hosted a collection of bureau-provided custom reports in some portion of the support website, but I don't know if this is still a thing.

What's in a (report) name?

Because System reports are universal, the same number ("S123") means the same thing to everybody, so it's fair game for use on the evo-talk list.

But because all custom reports get a unique internal number per bureau, the number doesn't mean anything to anybody at any other bureau; these are internal numbers only.

You might have a useful B123 "Customer Demographic Report" (just making this up), so even if you share it with another bureau, it won't have the same number except by rare random chance. They're internal numbers only.

So, if you're asking about a B### or C### report on the Evo talk list, always mention it by *name* and not by number, though keep in mind the name doesn't always have to be the same either; when you import a report, you can give it any name you want.

"Binary file was modified: Evolution.exe"

When the Evolution desktop client is launched, it does a self-check to make sure it hasn't been tampered with by doing an internal checksum of itself,. and when this check fails, we see a "Binary filed was modified" error dialog:

(sorry for fuzzy quality)

This functionality has been part of Evolution for more than a decade without being triggered, but in early 2020 I started seeing a few of these pop up, and was finally able to track down what's going on: it's Comodo in the two cases I've been able to study.

The customer is using "Comodo One Enterprise Security", and in additional to the customary firewall and antivirus protections, it seems to have an additional "containment" module that tries to run software in a kind of application sandbox. I don't know the details yet, but it's definitely responsible for whatever is happening to Evolution.

Evolution uses an unusual—but not improper—mechanism for updating itself, in part because payroll updates happen much more often than with other kinds of software. This has been especially the case during this coronavirus pandemic; it's a lousy time to be in the payroll software business.

For the desktop user

As far as I know, the only way to get around this at the desktop is to temporarily disable Comodo while the update is running, though I would imagine that many installations don't give the end user that capability. If you get this error, you probably have to engage your IT department to have them pause the containment feature so that Evolution will update itself.

For the IT department

Evolution typically runs in the C:\iSystems\Evolution\Client\ directory, launching as Evolution.exe, but during self-updates it also uses a directory under %TEMP%\iSystems\Update\ in the process, including possibly launching itself from that directory.

I don't know the Comodo software at all, but two helpful customers have been giving me tidbits

First, it appears that trying to safelist the Evolution.exe on the particular user's machine does not work, because I suspect Comodo is using the hash of the executable rather than the path. This means that once Evolution.exe is running, it's fine, but the next update—which changes the file's hash—fails the test.

Another customer is attempting to create a systemwide profile that (temporarily) safelists all .EXE—which I do not recommend other than for testing— just to see if that does the trick. If so, then the profile can be refined to just the narrow minimum required.

I'm very anxious to find comprehensive instructions for this.

Getting EvoTime to work on modern systems; MSSTDFMT not registered

"EvoTime" is a tool used to convert timeclock data from outside formats (say, from various point-of-sale systems) into something that Evolution payroll can import, and though it's been popular with users, it's not been supported by anybody (either the original developer or by Asure) in a very long time, and there are challenges running it on modern machines.

Note: this post is meant for Evolution service bureaus; if you're an end payroll customer, please contact your payroll company for help with this. I don't typically support payroll users except at the direction of my customers.

"Installation" is mainly just extracting EvTime.exe from the ZIP file—presumably obtained from Asure Software?—and running it, but on modern machines, it reports:

Class not registered:
You need the following file to be installed on your machine.
MSSTDFMT.DLL

This file supports the Visual Basic 6 language, which has been obsolete for a long time and is no longer included in modern Windows systems; you'll have to locate and install this file yourself.

DANGER ZONE: DO NOT search the internet for this file and download it from any source you do not completely trust. In my book, this would be either Microsoft or Asure; all other download sites, no matter how reputable looking, are a security risk. Bad actors are known to publish these same files but with viruses included, and it's just crazy dangerous.

As of this writing, Asure Software hosts this on its download site; I'm not willing to provide a direct URL link, but it's in the downloads/Utility/ folder as MSSTDFMT.zip. If you're an Evolution service bureau, you should be able to get this from support.

Once you obtain the file—MSSTDFMT.DLL— you'll need to install and register it, and this depends on what kind of computer you're using. EvTime is a 32-bit program, so its location depends on whether your Windows operating system is 32-bit or 64-bit.

1. Download the file to a place whose path you know; the downloads folders is as good as any. Be sure to note the full path of the file (ex: C:\Users\Steve\Downloads\MSSTDFMT.dll).
2. Start an administrative command prompt; even if you think you're an administrator on your own machine, you still need to elevate to actually use those full permissions. From the start bar, locate either command or powershell, then right-click and select "Run as administrator":

   

   When prompted, either click OK to approve elevation to full administrator, or (if requested) type your administrative password to approve elevations.

   You'll get a console window in response.

3. Copy the MSSTDFMT.DLL file to the proper directory, which depends on the type of Windows you're running:
   • 64-bit Windows: C:\Windows\SysWow64\
   • 32-bit Windows: C:\Windows\system32\
4. "Register" the DLL with Windows, which also depends on the version of windows you're using:
   • 64-bit Windows: regsvr32 C:\Windows\SysWow64\MSSTDFMT.dll
   • 32-bit Windows: regsvr32 C:\Windows\system32\MSSTDFMT.dll
   It will probably pop up a small dialog box showing you that it worked. If it fails, you're probably not a full administrator.
5. Try to run EvTime again.

It's unfortunate that this is all necessary, but it seems that nobody's stepped up to create a more modern tool that can perform this task.

Evo client: "Run as administrator" may prevent saving to network drives

Scenario: Service bureau CSR working remotely but connected to the office via secure VPN, running the Evolution desktop GUI client while logged into a Windows terminal services machine. Evo runs fine, but when trying to save files from the application, the save-file-as dialog box would not show any network mapped drives, only the local ones.

Hmmm.

Resolution: uncheck the "Run this program as an Administrator" box in the properties for the Evolution.exe:

We're not sure how this got set in the first place, possibly it was the installer taking special note of terminal services, but unchecking this box made the problem go away.

Evo Server guidelines for your new IT people

Customers who host their own Evolution gear sometimes handle all their own IT, but many outsource to local IT firms, and the MSP (Managed Service Provider) model has become very popular. In this model, customers pay a flat fee per month for each server or workstation, allowing for predictable cost projections.

In my experience, the first thing that any MSP does is install their remote management software on all the machines, including the Evolution servers. This software provides remote access for their technicians, it collects monitoring and alert data that phones home to their mothership, and allow for keeping a close eye on the machine.

You want this: Knowing that the C: drive is filling up, that there are critical errors in the event log, and any of the many other things that require attention, gives far better ongoing monitoring than a consultant like I provide: I'm not an MSP.

But the line has to be drawn between monitoring/remote access for the technicians (which helps your business), and proactive intervention on the machine (which is often bad).

Evolution servers have special requirements for reboots, because the Evolution application services have to be started in a certain order, and this almost never happens correctly by random chance.

Managed Service Providers must not configure any automated process that changes the system state, and must not ever perform reboots without coordinating with the Evolution software, whether automated or scheduled by a technician.

Almost every time a customer gets a new IT provider, we find Evolution problems in the morning that are traced back to some MSP-initiated action, such as installation of Windows updates overnight, but without any of the special knowledge of Evolution services.

The "aha!" moment while researching today's customer problem came from the Windows Event Log:

"LTSVC.exe" is LabTech (very popular MSP software), and it's initiated a restart of the server EVO6 just before 4 in the morning after installing a Windows update. Today, three of four Evo servers for this customer were rebooted, and it caused several hours of havoc until we figured out what happened (thankfully, it wasn't a repeat of the bad power problems we'd recently fixed, but we didn't know that without research).

This can be caused by fully automatic behavior of the LabTech software, or it could be that an MSP technician surveyed the systems yesterday and manually scheduled a 4AM install+restart; in this case, the lack of entries in the Task Scheduler suggests that this was LabTech running on autopilot.

Guidelines for MSPs / outside IT consultants

The Evolution line-of-business application servers are central to the business of your customer, and they are unfortunately kind of picky about things that they shouldn't be, but we still have to work around them.

Evolution systems are generally working 24x7

Though humans are not typically logging in to do payrolls at all hours of the day, service bureaus schedule many tasks to run overnight so that they're available in the morning, so it's very common to find the system busy at 11:30PM or 3AM or whatever. There is no time of day that's inherently "safe" for an automatic reboot.

Also keep in mind that many bureaus service customers in all timezones, so 8PM on the East coast is still only 5PM in California.

Evolution application services must be started in a certain order

Just rebooting an Evolution server, especially the one hosting the Request Broker application service will almost always come back in a unusable state because the services are unlikely to happen to start in the proper order. I've written about that here:

Evo Tip: Best Practices for Managing Evolution Services

This is really unfortunate, and I believe it borders on a bug in the software, but we work with the software we have, not what we wish we had. There's no obvious or trivial way to work around this in an automated manner.

Be sure your MSP software never automatically reboots an Evo server for any reason

This includes a for-good-measure reboot, or one that applies a Windows update.

Be sure your technicians know about this

I find conscientious technicians schedule overnight reboots frequently, and those break Evolution just as an automated one does.

DO NOT configure any automatic temporary file cleanups

Evolution uses many temporary files, and some of them necessarily have to hang around for a long time, and (in particular) LabTech MSP software has a module to "clean up" older temporary files. This breaks Evolution badly - don't enable this module at all.

DO NOT configure any automatic network tests/probes of the Evolution services

Evolution services do not always respond well to random TCP/IP probes, which is unfortunate but still a fact of life: tests just to make sure the service is up typically cause more problems than good.

And though properly-configured probes can be done in some cases, I've seen local mail relay (often running on one of the Evo middle tier machines) effectively disabled by port 25/tcp tests done badly, causing chronic email delivery problems until these unnecessary probes were disabled.

DO NOT configure any realtime antivirus without coordinating file exclusions

Evolution, especially the Request Broker, reacts very badly if one of its many working files gets locked while performing realtime scanning.

In addition, if the AV engine mis-characterizes a valid Evolution process as malware, it's going to quarantine it and essentially bring down the entire system.

This has happened to customers in the past, and it's very painful.

DO NOT install any third-party network firewall protections.

I generally leave the standard Windows native firewall enabled because I've got a good handle on the dozen or so exclusions required, but any third party firewall product is likely to do far more harm than good.

In particular, there cannot be any SSL certificate inspection on the network stream arriving from the outside world into Remote Relay; some special sauce of how Evo handles SSL means no third party software's ever going to get it right.

Watch out for Evo licensing landmines

Evolution ties its licensing to the underlying hardware, and many have found themselves with a broken license due to completely innocent changes in the server configuration, the very definition of an unhappy surprise.

Do not change:

• The hard drive configuration, including plugging in an external USB hard drive. If it presents itself as "removable", it's fine, but if it tells the OS it's a permanent drive, that changes the overall hard drive config and will break the license.

• Be super careful doing anything with the network interfaces; Evo includes the MAC address of the first network card it finds, but the order that NICs appear is highly unpredictable, and merely disabling an unused NIC is a common source of license breakage.

  However, all the IP address parameters can be changed at any time: IP, DNS, gateway, DHCP configuration - no problem. Server hostname can be changed as well without concern.

• Do not change the "Registered Owner" or "Registered Organization" of the server; these encode into the license as well.

Note that making a license-breaking change is not usually evident right away: it's only when a service (or the overall server) restart that it shows up, and it's then an emergency that can only be resolved with the Evolution support department.

Tread very, very carefully with anything on the Linux database server.

Other than passive monitoring, there's almost nothing that any MSP will be able to usefully do on the Linux database server.

DO NOT run network security/vulnerability/portscan tests during working hours

It's always fun to throw Nessus at the local network to see what ports are open and what vulnerabilities there might be, but Evolution does not react well to the unusual traffic presented.

If a network vulnerability scan is required by an audit, or just by good practice, be sure to do it at a time when Evolution can be stopped and then restarted after the test. The scans won't actually damage any data, but Evo may not work well until its head is cleared.

DO provide monitoring services and alerts

It's a value to your customer to give a heads up that a hard drive is filling up, that some process is eating 100% of CPU, or that a power supply has gone bad.

As you work with your customer's Evolution consultant, you'll get a sense for what things you should be watching for, and which things either don't matter, or are taken care of by somebody else.

Did the system reboot?

For the Evolution user finding an unexplained error showing up first thing in the morning, it's good to take a peek at the error message itself to see: did the machine reboot last night?

In this example:

Invalid property value
date/time         : 2019-07-23, 08:35:51, 769ms
computer name     : EVO6
user name         : SYSTEM 
operating system  : Windows NT New Service Pack 1 build 7601
system language   : English
system up time    : 4 hours 37 minutes
program up time   : 4 hours 37 minutes
processors        : 16x Intel(R) Xeon(R) CPU E5-2440 0 @ 2.40GHz
...

Hmmm, at 8:30 AM the machine had been up for 4 hours 37 minutes, so something happened just before 4AM; this is what has to be investigated (in addition to whatever is getting in the way of this particular Evolution task).

• Was somebody in the office that early doing maintenance?
• Did the machine have an operating system panic/bluescreen?
• Is there bad power?
• Was somebody in the server room and kicked a power cord?
• Did somebody turn on the "install updates and reboot" setting in Windows Update?
• Did a new MSP configure its own automatic update/reboot?

In our case, it was the latter, and it's been addressed with the MSP, and we believe that all the problems this morning were due to these inadvertent restarts.

Your UPS: "Replace Batteries" is not a suggestion

Almost all Evo service bureaus have at least some server gear inhouse: even if Evolution itself is hosted by Asure in AWS, you may still have a file server, print server, or the like.

And you probably have an Uninterruptible Power Supply (UPS), a battery backup, to keep your gear running if the power goes out. It's most helpful so a brief flicker on the power lines doesn't take down your servers, but for real outages all it really needs to do is keep the machines running long enough to be shut down gracefully.

Ungraceful shutdown is catastrophic to your gear and to your key business data held within.

Like anything else electronic, a UPS has limited life, but unlike other electronic gear, it can predict when it's going to fail: the "Replace batteries" light will come on when the UPS determines it's not holding a charge well.

UPS battery replacement is pretty straightforward, something most IT service firms can do for you easily, though the heavy batteries do have higher than usual shipping costs.

Having old/bad batteries means you have no protection from a real electrical power loss, and this is no surprise, but it's not the only risk involved. Having bad batteries can actually exacerbate the existing electrical problems by introducing new ones.

That tiny flicker on the power line that you don't see, and one that the redundant power supplies on your server gear could easily (ahem) power through, is correctly detected by the UPS as a power issue, but when it attempts to switch to battery, the wheels come off and your gear crashes.

These hard, ungraceful shutdowns can be devastating to your data; even if they don't physically damage the hardware, it's exceptionally common that a hard crash leaves the filesystem in a bad state such that the files on the filesystem—such as your Evolution databases on the /db/ partition— are corrupted.

A UPS with bad batteries is worse than having no UPS at all.

When your UPS tells you to replace the batteries, DO IT; your gear is at risk.

Temporary workaround

A workaround that I recommend while waiting for the batteries (or a new UPS if that's the issue) is—for gear with redundant power supplies—to plug one power cord into the UPS, and the other power cord into a regular wall socket.

The wall plug connection doesn't protect you from a true loss of electrical power, but it can protect you from a bad UPS.

This is not speculation: this happened today at a customer.

Four Evo servers were plugged into a UPS with bad batteries, but one of them (the request broker, as it happens) had its second power supply plugged into a regular wall socket.

When the UPS decided to crash all the gear plugged into it, only the request broker machine kept running. Evo was down, of course, but that machine didn't suffer any potentially catastrophic data corruption that the other servers were subject to.

The best solution for the long term, of course, is redundant UPS units, where each server with redundant gear plugs into both of them; this provides the clean, conditioned power that a UPS can provide (and the wall outlet cannot), as well as protection from a flakey UPS.

Power budgets

An important concern here: each UPS can only deliver so much juice, and there's a tradeoff of power versus runtime, and when running two UPS's, you have to be sure you know where all the power is going.

Machines with redundant power supplies typically only take power from one cord or the other, and you can't easily tell which of two UPS units it's drawing on. This means that when you look at the current load on a UPS, you can't tell if it's doing all the heavy lifting for all the servers, or if chance is drawing one server's power from the other.

The technical discussion of how to allocate your power budgets is beyond the scope of a simple blog post, but a simple rule is: if you have multiple UPS's, you cannot ever run with a combined load of more than 100%, because if one UPS fails, the other won't be able to pick up the slack.

Be sure to engage a qualified advisor when planning your UPS strategy to make sure you get your power budgets right.

In any case: do not volunteer for an emergency by neglecting the battery warning.

Evolution and Virtual Machines - it's ok!

For a very long time, iSystems offered only limited support for Evolution instances running in a virtual environment due to concerns about performance or configuration. After having seen some very poor VM installations in the past, I understood this.

But this has not been the case in years; Evolution works great in a properly-configured virtual machine, especially on the Windows side, and customers have used both VMware and Hyper-V very successfully.

The classic SaaS environment built by iSystems was entirely VM on the Windows side, and in the current AWS pods, all parts (db and middle tiers) are virtual.

If a customer is purchasing new gear to host more middle-tier resources, I always recommend getting a good solid machine with SSD storage, plenty of RAM, and to split it into VMs of 8 CPU each; a 24-core machine would conveniently host three VMs.

Evolution is still a 32-bit program with a two gigabyte per-process memory limit, and it runs only as many tasks as there are CPU cores. Though technically it can use all 24 cores of a big machine, all 24 concurrently-running tasks have to fit in that limited 2 gigabyte memory space, and it's very, very common to run out of memory if the mix of jobs just happens to go over the 2G limit. This is painful.

In practice, bureaus with memory issues have to artificially lower the max-tasks that Evolution will run at one time. This alleviates the memory issue, but everybody hates it that a 24- or 32-core machine is only using 8 of them for Evo. What a waste!

But turning this same machine into three VMs - of 8 cores each - then you now have three request processors and three separate 2 gigabyte process limits, and it can well and truly use all the power of the machine.

The Linux DB server requires much more thought, as Evo will actually use a very very powerful machine without the artificial 32-bit process constraints that limit the middle tier.

Most customers are no longer purchasing new gear for their self-hosted Evolution instances, as they're planning to move to an AWS pod sooner or later, but those doing a hardware refresh may well wish to consider going the VM route, especially if they're a VM shop already.