The Evolution middle tier servers all run the Windows operating system, and we have to keep them up to date with periodic security patches delivered via the Windows Update mechanism. With many configuration options, this has been a reliable mechanism for many years, but there's one setting that's dangerous: "Download and apply updates automatically."
Though it's helpful to allow Windows Update to automatically download these updates, you must not allow Windows Update to apply these updates and reboot automatically, even if it's in the off hours.
Evolution services must be started in a certain order: first the Request Broker, then all the Request Processors (as a group), then all the rest (as a group), but after a reboot, services start in their own random order that is not guaranteed to let Evolution start properly.
If the Evolution services are not started in the proper order, they can get in a stuck condition where all services are running, but nobody can login, and the Evo Mgmt Console shows many many realtime tasks that effectively show connection attempts. Evolution is not functional in this state, and the only fix is to stop all services, then start them manually in the proper order.
Since automatic Windows Updates are typically applied in the wee hours, nobody discovers that Evolution is down until people show up first thing in the morning, which has generated frantic 4:30 AM phone calls from East-coast customers. Fun for me!
For Evolution servers, I always recommend the "Download automatically but let me choose when to install them" setting, and for customers where I help manage the servers, I take care of Windows Updates when I can insure that Evolution services are started properly.
Note: this directive applies not only to Windows Updates, but third-party patch management tools (such as the Ninja Patch Manager) that essentially does the same thing. Some customers have outside IT companies that use tools to manage Windows Updates, and these must be configured with the same no-automatic-updates policy.
For customers on the Asure SaaS platform, the Systems Engineering team handles this for you properly, and you need not be concerned about it.
Comments