Steve's Evolution Payroll Blog

"Binary file was modified: Evolution.exe"

When the Evolution desktop client is launched, it does a self-check to make sure it hasn't been tampered with by doing an internal checksum of itself,. and when this check fails, we see a "Binary filed was modified" error dialog:

(sorry for fuzzy quality)

This functionality has been part of Evolution for more than a decade without being triggered, but in early 2020 I started seeing a few of these pop up, and was finally able to track down what's going on: it's Comodo in the two cases I've been able to study.

The customer is using "Comodo One Enterprise Security", and in additional to the customary firewall and antivirus protections, it seems to have an additional "containment" module that tries to run software in a kind of application sandbox. I don't know the details yet, but it's definitely responsible for whatever is happening to Evolution.

Evolution uses an unusual—but not improper—mechanism for updating itself, in part because payroll updates happen much more often than with other kinds of software. This has been especially the case during this coronavirus pandemic; it's a lousy time to be in the payroll software business.

For the desktop user

As far as I know, the only way to get around this at the desktop is to temporarily disable Comodo while the update is running, though I would imagine that many installations don't give the end user that capability. If you get this error, you probably have to engage your IT department to have them pause the containment feature so that Evolution will update itself.

For the IT department

Evolution typically runs in the C:\iSystems\Evolution\Client\ directory, launching as Evolution.exe, but during self-updates it also uses a directory under %TEMP%\iSystems\Update\ in the process, including possibly launching itself from that directory.

I don't know the Comodo software at all, but two helpful customers have been giving me tidbits

First, it appears that trying to safelist the Evolution.exe on the particular user's machine does not work, because I suspect Comodo is using the hash of the executable rather than the path. This means that once Evolution.exe is running, it's fine, but the next update—which changes the file's hash—fails the test.

Another customer is attempting to create a systemwide profile that (temporarily) safelists all .EXE—which I do not recommend other than for testing— just to see if that does the trick. If so, then the profile can be refined to just the narrow minimum required.

I'm very anxious to find comprehensive instructions for this.

Evolution Client update failures after Evo upgrade: it's McAfee

After the some releases of Evolution (typically but not universally on the Stowe release), a number of customers have experienced client update failures after logging in, where it reports that the update has failed, and it typically leaves the desktop client in an unusable state such that it can't even be uninstalled.

The cause in every case I've researched has been McAfee Antivirus, which is apparently trying to protect you but unhelpfully doesn't log this anywhere, leaving it a mystery. Ugh.

The fix:

1. Reboot the computer; this is necessary to get McAfee to let go of Evolution.exe. Without this step, the file cannot be deleted even as admin, and there's just no getting anywhere until the reboot.
2. Temporarily disable McAfee protection on the desktop
3. Run Evolution again to try to update it; if the software appears to be unavailable, rerun the installer (but while McAfee is still temporarily disabled)
4. Once Evolution is updated, re-enabled McAfee.

We don't know why it doesn't like Evolution or how to disable this behavior: all we have is the workaround.

This is not really related to the Stowe, so it's worth trying with any Evo update failure if McAfee is involved.

Evo Client use inside the service bureau

This post applies only to service bureaus who host their own Evolution servers; it specifically does not apply to those on the HSP.

Evolution provides the Remote Relay service to allow secure remote access for payroll customers on the outside, but Remote Relay should not be used by staff inside the service bureau and on the local network. Instead, the client should be configured to go directly to the Request Broker: this is a common mistake even though it appears to work.

Because the Evo Client installer defaults to the most common case — customers on the outside — all service-bureau staff has to perform a one-time port configuration to change these settings.

1. Launch the Evolution client
2. Use the drop-down box to select Compression=None
3. Click Settings
4. Use the drop-down box to select Port=9500
5. UN-check the "Secure" box
6. Click OK to dismiss the settings box
7. Enter your Evo username and password
8. Click OK to login to Evolution

It's possible that you'll have to go through this twice if Evo has to auto-update itself, but once you're fully logged in, Evo will remember all these settings for the user.

How to check

Rather than visit every user's workstation, IT staff can recognize internal users using Remote Relay from the Evo Mgmt Console. Navigate to Monitoring ⇒ RR Status, and this shows all users connected by Remote Relay.

By looking in the IPAddress column for addresses on the local network (often in the form 10.X.X.X or 192.168.X.X), one can identify local users and ask them to log out of Evolution, then log back in using the proper port settings.

This RR Status page only reflects current users, not those who've been on in the past, so IT staff may wish to visit this page now and then to check for stragglers.

Why does it matter?

It's possible for payroll staff to use Remote Relay successfully — the HSP is based on this — but there are several reasons why it's better for all staff to go directly to the Request Broker.

• Going through Remote Relay adds an additional compression and encryption stage in the process, but without providing any additional security. On modern systems the crypto step is not all that costly, but it's extra work for the Evo systems that shouldn't be necessary.
• Going through Remote Relay adds an additional batch/post step to some Evo screens, and going directly to the Request Broker removes this step. It's faster.
• Occasionally it's necessary to exclude remote customers from the Evo system while allowing payroll staff to remain connected; this doesn't come up often, but when it does, being able to turn off Remote Relay is very helpful.
• Service bureaus are increasingly requiring all SB-type access to Evolution be performed from inside the company offices and disallowing it over Remote Relay. This security measure insures that service bureau credentials cannot be used outside the office.

This is not an "IT" issue

I strongly recommend that this port configuration be done directly by the payroll service staff member because this creates a familiarity that will allow them to help customers going through it.

Of course, remote customers will never be going directly to the Request Broker, but it's common enough to walk customers through changing ports around (such as using port 443/tcp that it benefits the customer when the SB staff actually knows the material rather than reading from a script.

And to the extent that security policies allow, I recommend having payroll staff install the Evo client on their own workstation for the same reason.

Offloading all of these issues to the IT staff means payroll customers get less responsive service to their remote connection issues.

Remote Relay: can I turn off 9901..9903, leaving only 443/tcp?

Evolution's Remote Relay service by default listens on ports 9901..9903/tcp, but for years I've been configuring the services to allow an extra port: 443/tcp, which is typically used for secure web (Evolution's remote relay stream is encrypted, but not HTTPS).

I've written about configuring Remote Relay with 443/tcp in this blog post.

A customer's IT consultant asked if it we could turn off 9901..9903/tcp, requiring all customers to use 443. From a security point of view, this is a really fair question, asking if we can close open ports, but I recommended against it and am noting my rationale here.

This sounds like a good idea, but in practice it's of no benefit:

First, the Evo client installer defaults to 9901/tcp, and you’d be adding a step (changing the port number) to every customer install, one more thing they have to get right even though the default works fine now. Supporting new customer installs is hard enough as it is.

Second, an increasing number of customer firewalls are doing SSL certificate validation on 443/tcp (because it’s mainly used by HTTPS), and since Evo uses baked-in self-signed certs that do not chain down from a root certificate authority, these connections are considered bogus and reset. This prevents the customer from connecting to Remote Relay.

When this happens, moving back to 9901/tcp sometimes gets around that issue, though some firewalls detect SSL on any port and perform the same shenanigans. These customers will require whitelisting in their firewalls.

I suppose you could close 9902 and 9903/tcp, which are almost never used, but that makes some kinds of troubleshooting more difficult because I sometimes have customers change to 9902/tcp so that I can run Wireshark (a network sniffer) on their connections, this altered port making it easy to pick out their traffic.

I understand the idea of trying to close “unneeded” ports, but attack surfaces are about services, not ports, and since all four (443, 9901..9903/tcp) go to the same service — Remote Relay — closing 9901..9903 only reduces the apparent attack surface, not the actual attack surface.

I recommend against this.

As I hear alternate rationale, I'll update this post.

Using port 443/tcp for Remote Relay

Evolution's Remote Relay service — by which customers in the field access the Evolution service bureau's system — listens on ports 9901..9903/tcp, with a default of 9901. All go to the same place, but differ in the amount of compression applied to the stream.

This variable compression is a throwback to the older days of slower internet speeds: you'd probably want more compression to reduce the data sent over on slow dialup connection. These days with monstrous cable speeds the distinction is not really that important any more, but all three are still open as a kind of legacy.

These are open for all Evo installations supporting Remote Relay, but an increasing number of payroll customers are performing egress filtering in their border firewalls, which blocks outbound traffic to unknown ports, and these customers have traditionally had to create manual exceptions in the firewall to allow connections to the Evo servers.

Some customers have a harder time making these exceptions, because perhaps the firewall committee meets only once a week (really!), and since exceptions are often based on IP address rather than hostname, these whitelists have to be updated when the payroll service bureau changes internet providers.

One payroll customer's firewall was centrally managed from corporate headquarters, and the end user said it would likely take months to get this change implemented. Ouch.

Remote Relay supports listening on an additional TCP port, and for years I've been configuring service bureau systems to use 443/tcp, the same port used by secure web (https), because even customers with aggressive egress filtering typically allow outbound https, meaning the connection to the Evo server works too. It's been very helpful.

This requires one-time implementation on the service bureau side in Evolution and in the border firewall:

• In the payroll service bureau's border firewall, look for the rules related to the inbound Remote Relay public IP address (where 9901..9903, and sometimes 9943/tcp are handled), and forward 443/tcp to the same place. But: see caveat below.
• In the Evo Management Console, navigate to Configuration ⇒ RR Ports, then enter 443 in the "Custom" field. I typically select merely "Good" compression, believing there being no good reason to burn local CPU resources to save bytes on a fast connection.

  

  Save the changes.
• Ask an Evo Remote customer test the connection by launching the Evo client, clicking the Settings button, then choosing 443 from the drop-down box.

  

Caveat: this can only be configured by the service bureau if the public IP address listening on 9901..9903/tcp has 443 open in the first place. Those with a single IP address on their internet connection might already be using 443/tcp on the same IP for something else (perhaps webmail or Remote Web Workplace). In this case, it's likely not possible to use this facility.

Once configured, Evo Remote customers having problems connecting on the default ports ought to be invited to change to 443/tcp to see if that works. But once customers are connected to Evo Remote — by whatever port — there's no good reason to change them.