Systems Integration / Emerson Exchange / Cybersecurity

Patch Management Service Streamlines System Maintenance at Eli Lilly

Patching Is Not Everyone's Favorite Job. You're Always Trying to Find the Next Hole and Plug It

By Jim Montague

CG1310 emerson show

One common headache shared by process control engineers and IT managers is how to effectively deal with software patches. And when applications, systems, workstations and the requisite updates multiply, keeping up with all of them can quickly devolve into a sanity-threatening situation.

"Patching is not everyone's favorite job. You're always trying to find the next hole and plug it," said Kurt Russell, consultant engineer for automation at Eli Lilly and Co. "Patching for an entire plant can introduce a whole new level of complexity, and is pretty painful, especially if it has to be done by one person." Russell is responsible for 15 DeltaV distributed control systems at his company's pharmaceutical manufacturing facilities in Indianapolis, Ind.

Russell, together with Bill Beane, SureService and Advanced Services engineer at Emerson, and Shelli Callender, Advanced Services project manager at Emerson, presented "DeltaV Patch Management: An Enterprise Approach" this week at the Emerson Exchange Global Users Exchange in Orlando, Fla.

In part because of the sheer volume of applications and systems, the manual patch management methods employed at Eli Lilly had become unwieldy. "It's a time-consuming task, and so it tended to be reactive, infrequent, and prone to delay or even mistakes," said Russell. "So, we began to experiment with mechanisms to automate patching with some success, but we're especially thankful that Emerson was able to develop its Guardian Software Update Delivery Service (GSUDS) and offer it as a free software tool for patch management."

GSUDS enables Eli Lilly and many other users to download software patches, hot fixes and other updates needed to keep its systems in safe and secure working order. These tasks are performed on the Guardian WSUS Interface (GWI). WSUS is short for Microsoft's Windows Server Update Services.

"We've used GSUDS since its inception, and we struggled early as there was little supporting documentation," said Russell. "So we worked with Emerson to develop better documentation, and invested considerable time into the implementation as it matured. We feel a lot better with the current state of the product, and its popularity is growing at Eli Lilly as a result of my success using the patching service.

"The main benefit is that we don't have to run around and manually do patches anymore. We can really see them all in one place, and easily get the data and reports we require."

Russell explained that Eli Lilly's system updates are managed through an upstream server with Internet access, where the GSUDS application resides, as well as a corporate IT server where Symantec Live Update Adminstrator (LUA) security updates are received. Each DeltaV system, in turn, is handled by a dedicated downstream server machine that hosts Microsoft WSUS, GWI, and LUA applications.

"The downstream servers are all currently small form-factor Dell servers," added Russell. "These servers also support other needs depending on the particular system, including backup and recovery, MiMiC simulation, and virtualization snapshots." Microsoft security patches are automatically approved at the upstream server and synchronized to the downstream server. GWI approves updates and hot fixes specific to DeltaV, and each system administrator approves updates to their own system. Russell stressed that hot fix prerequisites still must be performed. "Otherwise, installation of new patches may fail, and a second attempt will be necessary following a Windows reboot," he said.

"In short, GSUDS gives us a single infrastructure for patching, automatic distribution of updates and other content," Russell said. "It saves us much time and effort compared to manual methods."