Cybersecurity

Microsoft Windows patch management driving inefficiency, lost production

Nebulous mechanisms for security patches requiring increased man-hours

By John Rezabek

Have you noticed, “Patches,” the Microsoft Windows security patch dog, still hasn’t been put to sleep? Last week, after applying update patches to one of our operator workstations, it was taking a while to reboot (necessary to complete the patch installation, you know). Since I had availed myself of a brief process outage to install them, the operations team leader was okay for me to go to lunch when it was apparent the machine needed time to restart. Who dares to interrupt a shutdown these days when you’re admonished “Do not power down your PC” whenever they’re applied. Swirl, swirl went the Windows 7 animation along with the “Shutting Down” message. When I got back from lunch, it was still doing it. Swirl, Swirl. As shift change approached more than four hours later, the workstation would still not complete the reboot. What if the process had been running?

The DCS operates with no direct connection to the company LAN/WLAN, or private intranet, let alone the world-wide web of extraordinary hazards: We can’t “ping” google.com or even any random box on the company LAN. Despite this, we are still compelled to install monthly security updates on all the Windows boxes, 64-bit Windows 7, Server 2000-and-whatever along with any Office applications, SQL Server, .NET Framework and so on that may be installed or used in the DCS applications. Applying the patches is a little disruptive, as every machine must typically be rebooted to apply them. This means, for example, one needs to ask the board operator to step aside from whatever they’re doing, lose any trends or graphics they have up, and move elsewhere during the update. When servers are rebooted, gaps appear in trends, and any control modules that may use server-provided data (e.g. lab data passed through the historian’s interface to the business LAN) often malfunction or go into alarm. Even when Patches stays “on the chain,” he’s an unwelcome nuisance.

As shift change approached more than four hours later, the workstation would still not complete the reboot.

In the rack room there’s another trio of Windows servers that simply provide Modbus SCADA and a rudimentary HMI for operator interaction with the ESD system—bypasses, permissives for startup and local panel displays. The servers are from a supplier that specializes in industrial PCs and servers. Applying patches to these machines, with no connection to the Internet or with vetted and approved packages from their supplier, presents some risks. Will the next one stop the Modbus interface or cause OPC to malfunction?

Windows boxes are also employed in the growing number of applications for interacting with our asset management systems for rotating equipment and intelligent devices, and may provide an avenue or gateway for the “things” we dream of joining to some internet somewhere, someday. Gas chromatograph maintenance systems have relied on applications that run on Windows for decades, and in complex analyzers like mass spectrometers and FTIR spectrometers, they are an integral and essential part of the analysis and method development. We have Windows-based interfaces for numerous other engineering interfaces, like for running a “frame” application for FDT-DTM, configuration programs for PLCs or smart relays, and sequence of events (SOE) recorder interfaces. I’ve known some of these to sit idle for a year or more—do you think we’ve been religiously applying patches to them?

Download the Special Report: Cybersecurity in Operational Technology

This proliferation of Windows workstations in every aspect of automation seems unlikely to slow down. Meanwhile, the mechanisms for patch management are often nebulous—where do I get/download the latest security updates? Do I need a separate IT administrator just to look after all these boxes, or do I want to consume a few man-days of my controls professional’s time to diddle around with mundane OS security management? Do I trust a “geek squad” from my DCS supplier or other outside resource? At least in the latter case I would have someone else to hit on the nose with a newspaper besides Patches.

When I came in the next morning, after a few stabs at automated “repair” procedures (that all said they failed), “CTRL-ALT-DEL” eventually appeared. I suppose I should open a ticket with my supplier, but I know that will entail another day or two of data gathering. I think I’ll just wait till next month and hope for the best. Down, Patches—bad dog!