This article was printed in CONTROL's April 2009 edition.
By Bob Sperber
Missing project milestones was not an option. Never mind that His Royal Highness The Duke of York, would be visiting at start-up; process expansions and control-room updates were a critical piece of a £390 million “Project Genesis” modernization program to turn Ineos ChlorVinyls' Runcorn, Cheshire, U.K., site into a showpiece for eco-friendly chemical processing. However, with a year to go, process control manager Philip Masding couldn't find a supplier who could model his plant in time to test how his control scheme would handle his process and cope with dynamic process disturbances. It's a lot harder to find experts in chloralkalide process modeling than, for example, more common petrochemical modeling.
“We looked at a number of possible suppliers and their delivery dates were always going to be short of time,” says Masding. Knowing time was short no matter who did the work, he took the work in-house, linking his control system to models he built himself. “We started with the most critical areas, and we would at least train operators on the most critical areas of the plant” using a general-purpose simulator to model, test and ultimately train operators, starting with gas pressure controls.
And while the Duke—Prince Andrew, second son of Queen Elizabeth II and an international trade official—was rightly pleased upon his July 2006 visit, Masding didn't rest on his laurels. He has since “drastically reduced the number of large-scale excursions [or disturbances] that we've had by applying the model to the control scheme,” using updated operating data.
Dynamic models that feed simulated process data to offline test versions of control systems have become an indispensable tool for process control engineers. The term operator training simulator (OTS) that generally applies to them is a bit of a misnomer because these types of systems address ongoing needs, from control system factory acceptance testing (FAT) through start-up and ongoing training and system improvements. While lower levels of fidelity, or detail, characterize most OTS systems, high-fidelity systems are still mainly used for new process designs. As those high-end simulators once relegated to steady-state engineering designs add dynamic modeling, the costs and benefits of such systems are as varied as a plant’s needs.
Testing, Training Benefits
Martin Berutti, business director of Mynah Technologies, says OTS benefits include reduced “operations-induced unscheduled downtime,” faster time to market for control systems, product quality, operating costs and mitigation of risk. He cites savings from $100,000 to $500,000 per day when OTS speeds commissioning and validation; $500,000 to $1 million per production run based on extensive software testing to reduce off-spec product; and $500,000 to $100,000 per hour through better-trained operations staff and reduced automation system errors for cost-avoidance of $50,000 to $1 million per incident.
“The cost of running a full series of tests on-site with live equipment are considerable, versus the cost of running a full series of multiple tests, multiple times, with multiple scenarios in the back of our office,” agrees Andrew Robinson, project engineer with control systems integration firm Avid Solutions in Apex, N.C. The firm works at several pharmaceutical plants where FDA validation is stringent. He cites a recent job where one engineer spent two weeks in a plant starting up a batch-process DCS. Without thorough off-site testing, he says the start-up “would have probably taken two or three times as long and two or maybe three people to get the job done right.”
DCS vendors use OTS on their own staging floors before shipping a system, which lets them test and update the customer’s configuration to “save weeks or months of DCS commissioning time,” says Tobias Scheele, vice president of advanced applications for Invensys Process Systems. This reduces configuration errors in the field. He adds that the value to the plant depends on “the revenue associated with the additional days of early production, which usually dwarfs the cost of the control checkout effort.”
Systems can be updated several times, even daily, on a DCS vendor’s staging floor. Configuration bugs found in the morning will be fixed in the afternoon, and the process can be repeated many times. Once the system is shipped, the plant’s personnel will then run through their own checkout or full FAT cycles, testing the physical, online system as they follow with start-up and commissioning. However, training strategies must start before that because the need is critical—and expensive.
OTS systems address the critical need for training as soon as models and the DCS configuration can be simulated (or “emulated,” which applies to some control simulation).
You Decide on Costs
Software costs can run in tens of thousands of dollars up to several million dollars; Invensys’ largest OTS had more than 80,000 I/O points. And there’s more to a system than process models. The control system needs to offer its own simulation (or emulation) function, to which the customer’s control configuration and graphics must be added.
A full solution also includes training courseware and a training curriculum. Such a system can cost $135,000 at the low end, says Mark O'Rosky, OTS operator training solutions group leader at Emerson Process Management. He adds that Emerson’s entry-level OTS Express occupies a sweet spot for his OTS customers in the $250,000 to $450,000 range, which covers an application “about the size of a hydro-treating unit or a crude unit at a refinery, typically a one-process unit with two operator training seats.”
O’Rosky explains that costs vary because needs vary. Sometimes the models are extensive and the training application requires only a single seat. Other times, the models may need to cover only a few critical vessels, but the training application may require that the instructor manage nine seats and invoke nine different failures at the same time.
Even the organizational chart—the trainer's area of responsibility—can in part determine the scope, or phasing, of OTS modules. “The training curriculum could be composed of 15 critical process operations,” says Pete Henderson, product marketing manager, Honeywell Process Solutions. “Other customers may choose to integrate multiple models into blocks, so that interactions between operation teams can be seen, and the impact to their peers as a consequence of their performance can be managed.”
Working in a recent oil and gas system, Winston Jenks, Cape’s technical director and CEO, says the higher-end Omega system was needed for its more detailed models and a higher-bandwidth interface because “there were so many tags to move back and forth on that system, we needed this sort of high-performance I/O interface.”
More OPC Needed
An entire OTS system can reside in one high-end PC or, more commonly, two or more boxes that are interfaced using proprietary interfaces, or via OPC. At Ineos, for example, Masding chose OPC to send models he built in VisSim, whose U.S. supplier is Visual Solutions, to his DCS (Emerson’s Delta V). In a pilot study, he fed physical 4-20mA outputs from the control system to an analog-to-digital card on the PC running the model, but that was for five loops. It would be “very difficult indeed” to do that on a full system with hundreds of I/O points, he says.
Unfortunately, not all vendors offer enough proprietary interfaces, and OPC isn’t yet ubiquitous. “Our main wish is that every control system vendor would make an off-line control system simulator with open OPC data access,” says Mynah's Berutti.
The lack of OPC may stem from a “mindset” issue, adds Cape’s Jenks, because “people usually think of OPC as a way to read data from the control system, and we’re more often writing data to it.” However, more OPC would offer users much greater flexibility, especially when they want more detailed models, including models for ancillary sub-processes, such as skid-mounted compressors with their own programmable controls. For example, operators may need to know more than the on/off status of a compressor; they may need to know the operation of its staged operation and status of components, such as lubrication-oil pumps, fuel-gas lines, turbines and motors.
Jenks believes the outlook is good for more and better interfaces beyond the DCS as more users demand them. He sees DCS vendors’ integrated safety and emergency shutdown systems as a leading indicator of things to come.
Semper Fidelis?
The characteristic of fidelity—the detail or accuracy of a simulator in depicting the online system—is a somewhat contested criterion. The closest thing to a yardstick for fidelity is the ANSI/ISA 77.2 standard, titled “Fossil Fuel Power Plant Simulators—Functional Requirements.” It doesn’t tell developers or users how to achieve various levels of fidelity, nor is it a program where that can be easily evaluated.
The standard’s Section 6.1 says simulated instrument values in a “steady-state, full power operation” simulation should be within 2% of the reference plant. However, Section 6.2 says transient operations, where dynamic OTS plays, require accuracy within 20%.
Should that 20% raise eyebrows? No, say the experts, including Masding, whose experience gives him “a fair idea when the accuracy of a simulation is good enough.” His trainees will “get a fair-enough idea of how things are going to react” if a simulated tank fills in four instead of five minutes. In tweaking his control scheme, he doesn’t run equipment “within 20% of the bursting point.” Instead, he simulates disturbances, compares simulated responses with the real-time system, and makes incremental improvements when a “slightly different control scheme is better than the one we've got. I'm not expecting to be within 1% or 2%.”
High-fidelity dynamic simulators can achieve single-digit accuracy that may be needed in a highly complex, continuous hydrocarbon process as opposed to a typical batch plant, and that appears to open a market for high-fidelity dynamic simulation. Invensys’ Scheele says these can be used across the board, for plant design, control system testing and operator training.
While noting that highly integrated and complex systems “tend to grow in cost,” Sheele adds that, “The lowest life-cycle cost is usually the development of a high-fidelity model during the design phase of a project that evolves into a high-fidelity operator training simulator.”
Mynah’s Berutti counters that, “The biggest misconception in the industry is that simulation models developed for process design have applicability for operator training or control system testing,” because in part dynamic process design models rarely account for “the range of conditions and real-time performance necessary for operator training or control system testing. The user would be best served by forgetting about this pipe dream and incorporating simulation for control system testing and training as an integral part of the automation project and control system life cycle.”
The only resolution to this issue is found on the documentation of user needs and consultation with experts during the election process. Solutions in this highly complex field are case-by-case. While no hard-and-fast rule can apply to all plants, it’s indisputable that simulation has become a critical component of control system testing and operator training.
3D, VR: The Next Big Thing?
Most process control engineers find that DCS supplier’s simulation/emulation features provide a perfect, realistic human-machine interface (HMI) for control system trainees. After all, the offline DCS training system is identical to the online system running the process, albeit with simulated I/O. It can’t get any “realer” than that. Or can it? Recent developments in 3D modeling and virtual reality are now pushing the boundaries of simulation HMI.
Invensys Process Systems in February announced a new 3D “Immersive Virtual Reality” human-machine interface, complete with headseat, for use by “not only control room operators, but also outside operators,” says Tobias Scheele, Invensys’ vice president of advanced applications. Scheduled for release this year, the system is being used at plants now, including multiple global companies.
While the concept seems bleeding edge, 3D modeling—without the goggles—has proven “extremely beneficial for the control engineer to visualize how a system is operating,” says Brent Stromwall, vice president for control systems integration firm Polytron, Duluth, Ga. He also claims 50% reductions in control system start-up time.
He adds the technology “easily reduces” project start-up costs by 50%. Polytron has used 3D models for complex, full-plant dynamics, as well as more basic challenges, such as flow restrictions in pipes and the interaction of bottles as they move from a wide, collecting conveyor to a narrow, downstream line in a bottling plant. He cites Demo3D from Emulate 3D as a useful tool with underyling physics models he calls “the next big thing.”
In addition to system testing, Stromwall says the same model used for designing lines also “produces tremendous benefits for training by giving operators the visuals they need when operating or interacting with the control system.”
The 3D simulations make learning more “tactile” and “hands-on” when applications are hard to visualize, says George Zhookoff, product manager for E2M, Polytron's system analytics partner. He adds that trainees “ages 10 to 50” can benefit, but “the younger generation tends to have more of a more of a comfort level in the beginning.”
Demographic Time Bomb
A 2005 quote credited to John Payne, BP operations technical training manager for the Western Hemisphere, raises concerns that a ticking “demographic time bomb” is threatening to compromise the ability of plants to find skilled operations personnel.
Citing a paucity of skills among retirees' entry-level replacements, she called for “creative solutions to help new workers gain necessary skills as quickly as possible,” and OTS fits the bill for process control training.
Even with a recession and layoffs afoot, many process industries are scratching to find good recruits. Even in oil-related sectors where capital projects may have been planned four years ago when oil was cheap (as it is again), and layoffs are low to keep up with demand, skilled operators are always at a premium.
