Change (Management) is good!

By Steve Kuehn, Managing Editor

Have you downloaded the latest Windows security patch yet? How about Service Pack 2? I think I downloaded a couple of updates when I bought my PC, but then promptly ignored the prompts thereafter. About a year ago I went broadband and it didn’t take long before my computer got a bad case of the ick. Long story short, after a month of trying to fix the thing, it was time to wipe the drive and start again.

I recount this story not for sympathy, but to illustrate the fact that if I thought it was a hassle to keep the software updated on just one computer, it certainly must be tough to manage and maintain all the software associated with a contemporary plant-wide process automation control system.

In Software Engineering, 6th Edition, author Ian Sommerville pins it right down in chapter 27: “Software change is inevitable,” he says and here’s why:

• New requirements emerge when software is used
• The business environment changes
• Errors must be repaired
• New equipment must be accommodated
• The performance or reliability may have to be improved

The reality is that when it comes to automation software, there is hardly a moment when things aren’t in some sort of flux. For that growing breed of process control professional that has one foot in business systems IT and the other in process automation IT, managing the complexity of software administration must be frustratingly hard.

Sommerville notes that it’s difficult for any organization to implement and manage changes to legacy systems. In “Upgrading Legacy Control Systems,” (May, ’04, p30) CONTROL’s Senior Technical Editor Rich Merritt says that “upgrading system software can be one of the least costly ways to improve the operational aspects of a legacy control system.” He explains that an injection of updated software is an economical way to improve process control without having to start from scratch.

Sounds easy. But imagine embarking on a system-wide upgrade on control device software across key processes throughout one plant, then managing its roll-out to other plants as part of an organization-wide program. It boggles my mind, but that’s not saying much. Fortunately, in the parlance of commercial speak, there is a solution! Chaos-managing platforms are available and being implemented to solve many of the complexity issues associated with control and system network software maintenance.

Rockwell Software recognized the enormity of the situation and introduced the Rockwell Software Maintenance Automation Control Center (RSMACC) a centralized, change management platform that overlays an automated organizational template over the process control/network landscape. Rockwell says process engineers can use it to create an audit trail running through the entire plant, guide maintenance and upgrade activity, and to obtain a clear picture of which programs are running, their version, what changes have been made, who made them and when.

Not only do such systems have the potential to reliably manage software change complexity, but the audit trail it provides can help process engineers with regulatory compliance such as 21 CFR Part 11 and Process Safety Management (PSM).

Recently, Brand-Name Orange Juice in Florida (sorry, I can’t mention the name because the guys in the engineering department couldn’t get approval from their Brand-Name Cola-company corporate overlords in time for me to attribute the team’s excellent work directly) implemented the RSMACC system and achieved much of the benefits claimed by the supplier. At Rockwell’s recent Automation Fair, principal engineer “JR” outlined the implementation and explained that security concerns led the reasons why they wanted to install a change management system.

“Simply put, we were unable to secure access to the control system,” he says. According to JR the plant’s union environment, the multiple outside contractors hired by multiple project managers, and the fact that every software modification ordered was an “emergency” change, were among the top issues related to implementing this software “Super Nanny.” JR says process engineers were unable to maintain program versions and provide easy access. “We didn’t know what changes were being made to programs and had no way to track them,” says JR. “Backups of PLC programs were random, if at all. And we did not know what programs were actually running in the controllers.”

The bottom line is that Brand-Name Orange Juice was looking for a way to solve production and quality control issues and drive their associated costs out of the pitcher (pun intended). Space won’t permit more detail, but suffice it to say that the precision minds in charge there recognized that software maintenance issues were a root cause of problems and that existing software chaos was just inviting both costly downtime and heavy security liabilities. And while the solution was ready squeezed, its well-executed implementation was what made it sweet enough for the rest of Brand-Name Cola’s plants to want to get a taste.