Greg McMillan and Stan Weiner bring their wits and more than 66 years of process control experience to bear on your questions, comments, and problems. Write to them at [email protected].
Stan: The operator is the most underutilized resource in the plant. I think most operators would appreciate a greater understanding of the process and playing a bigger role in improving its performance. This is not to say operators don’t already do a tremendous job in dealing with the inevitable unknowns and problems to keep a plant running nonstop. Downtime in many plants can amount to millions of dollars per day in lost production.
Greg: When we did opportunity assessments, we found the production units at one plant consistently out performed the units at several other plants. The difference was that the operators knew the practical limitations to production better than the technical support engineers and were the initiators of most of the ideas for process control improvement. If the people on the front line who have to resolve problems on a minute-to-minute basis have an understanding of the process relationships, the result can be truly remarkable. The knowledge developed can be put into the automation system. Advanced control is, after all, the embedding of process intelligence.
Stan: The key to unleashing the true capability of a plant is the operator training system (OTS). Most companies realize an OTS is essential for getting the operators to make maximum use of an upgraded instrumentation and control system. The more astute companies realize it offers an ongoing role for exploring and understanding problems and capturing and disseminating knowledge, not only to operations, but also to technical and maintenance support functions. Probably the least recognized opportunity is getting maintenance and operations on the same page. As we said in the March 2010 column, the first question asked when production changes, is what maintenance was done.
Greg: To maximize the performance and benefits of an OTS, we continue our discussion with the president of Mynah, Mart Berutti.
Stan: What are the job functions and skills of people who build and deploy an OTS?
Mart: Operator training systems require process simulations that are dynamic and real-time. Because the purpose of both OTS and testing and system acceptance testing (SAT) is to provide realistic responses at the operator glass, the control system platform is very important in the overall performance of the system. We find that the best developers of these systems are process control engineers that understand the process and process dynamics. If they have advanced control background, they are often very good candidates for developing dynamic simulations. Process design engineers who have an extensive steady-state modeling background sometimes have difficulty understanding the cause and effect and driving forces of a dynamic system. Of course, plant operations involvement is also essential. In many cases, the most experienced operators and operations supervisors can best dictate the use cases and acceptable performance for the dynamic simulator used in the SAT and OTS.
Greg: What type of simulation building environment do your customers find most useful?
Mart: Since we are working with control system engineers more so than process design engineers, we like to use IE1131 programming languages such as function blocks and structured text. This allows the control system designer to make the transition to dynamic simulation developer without learning a completely new configuration environment. The only new paradigm that the user needs to adopt is the use of process equipment objects in the IEC1131 function block environment. In addition, process equipment objects are not connected with wires carrying signal values, but with streams conveying dynamic process information (pressure, flow, temperature, density and composition).
Stan: What are the relative advantages and disadvantages of various methods of communication between the model and the DCS?
Mart: Most offline control systems have an OPC server or Modbus TCP/IP or Ethernet/IP slave interface. The dynamic simulation system needs to have an integrated OPC client or Modbus TCP/IP Master or an Ethernet/IP Scanner. Ideally, the simulator will have all three options, and an I/O service that runs independently of the simulation engine. This allows the user to integrate I/O by tag name and not by the DCS I/O path. Utilities should allow the user to generate the dynamic simulator’s I/O definition and low-level models, such as tiebacks, automatically so that the I/O definition in the dynamic simulation matches the distributed control system automatically
Stan: Do you run your models stand-alone before the configuration is ready, and if so, what control loops do you put in place, and how do you initialize the DCS loops?
Mart: So the OTS simulation can be developed in parallel with the DCS configuration, we run the dynamic model by including the basic control loops in the simulation via IEC1131 control blocks in our library. In order for volumes not to overflow or run dry and for pressures to be in the operating range, the level and pressure loops are immediately put in Auto. Next the temperature loops are put in Auto because these loops are often the key to getting the composition right, in addition to regulating the energy balance. With these loops, the model can be fully explored, tested and documented by a library of operating conditions captured by snapshots. When the configuration is ready, the control can be readily transferred to the actual distributed control system, and a new library of snapshots created for restoring and resuming operating scenarios.
Greg: What do you do for really slow processes, such as distillation columns and bioreactors, to simulate periods of greatest interest?
Mart: We use snapshots of abnormal situations, upsets and interesting points in the batch cycle, start-up and continuous operations, to "restore and resume" and eliminate the need to wait for a model to reach these operating conditions. The virtual plant also has the capability of running about 10 times real time for a control module execution time of one second.
Stan: Greg, what have you done to speed up the simulation of bioreactor batches that normally take one to two weeks?
Greg: I have increased the kinetic factors by a factor of 10 and increased the integration step size for the material, component and energy balances by a factor of 50. Since the effects are multiplicative, the model runs 500 times real-time, and the batch is done in about 20 minutes.
Stan: What adjustments are needed in the control system to deal with these speed-up factors?
Greg: The flow rates manipulated by control loops must be increased by the kinetic speed-up factor. If the major sources of dead time are fixed by agitation rates and automation system dynamics, the main effect of the speed-up in the integration step size is a proportional decrease in the process time constant. If the reset time is set per Lambda tuning equal to the process time constant, there is a proportional decrease in integral time setting. If the reset time is set proportional to the dead time, there is no change in integral time. Since the maximum controller gain is proportional to the process time constant to dead time ratio, the loop will go unstable unless the controller gain is significantly decreased. Since most loops are rarely tuned with more than 1/5 of the maximum controller gain, as a first approximation, the controller gain for an integration speedup of 50 should be decreased by a factor of 10.
Stan: Mart, what type of abnormal situations do you simulate?
Mart: We allow the user to introduce scenarios that include simulated valves that are stuck or that have failed closed or open by simply putting the analog output in manual and setting its output. We can do the same thing with discrete outputs for on-off valves and motors. For transmitters, we put the analog input in manual and set its signal to simulate failure as last value, upscale or downscale. We can also create a huge variety of process disturbances by changing incoming stream flows, temperature, densities and compositions, and by changing model process parameters, such as a heat transfer coefficient and catalyst activity. Introducing sudden changes to atmospheric conditions can also be very effective events for operating training.
Greg: How do you measure operator performance?
Mart: We keep track of the time it takes each operator to solve a similar problem. We get rid of outliers and generate a plot that represents the composite learning curve for the set of operators. We can repeat these tests and generate the composite learning curve for a different type of problem to see if improvements in operator graphics are making a difference. We allow the user to define the expected results for each operator training scenario. Scoring conditions can be weighted and set for each expected results. There is no limit on how the user can define his operator scoring. Conditions can be set based upon operator tasks such as acknowledging alarms or changing the setpoint or valve position on critical loops. Scoring conditions can be set to evaluate the health of the simulated process by evaluating key parameters in the dynamic simulation. The training session results (including scoring) are automatically saved for each student.
Stan: What are the benefits of an excellent OTS?
Mart: The immediate benefit is a decrease in the time for the production unit to come online and start making top-grade product. The biggest benefits materialize over time as better operator performance, better automation system designs and control strategies, and greater operator involvement in continuous improvement provide higher production rates and process efficiencies. Tom, an operations supervisor at a commercial power plant, summarizes it this way, "Every click on the mouse at the operator console has the potential to save or cost the company $100,000. If we can make more clicks positive, instead of negative, the value of our investment in the OTS keeps growing."
Greg: What do users like most about their OTS?
Mart: Tom, was very impressed with the realism and dynamic response that his OTS provides in comparison to the complex interactions of his power plant. This allowed him to use the OTS to teach new operators troubleshooting skills and how to direct and work with the field operators through realistic role playing. The OTS also has allowed him to develop field operators who were capable, but afraid of making mistakes, into capable panel operators. Jack, a control specialist in an agricultural chemicals company, likes the fact the I/O interface is automatically generated whenever there is a change and can be used for development, testing and training from the start of the project for the complete lifecycle of the automation control system. By having an OTS that his plant resources can maintain, the value of the investment continues to grow.
Greg: Normally our "top ten list" is on things that can go wrong (kind of the nature of most humor). This list is on what if things go more than right.
Top Ten Signs of an Excellent Operator Training System (OTS)
(10) Plant production rate is higher than model
(9) Online yield metrics are off-scale high
(8) Operators postpone vacations to get more time on OTS
(7) Operators do an opportunity assessment of process control improvements
(6) Operators are more interested in the process than doughnuts
(5) Operators invite automation engineers on fishing trips to discuss control strategies
(4) Calendars in break room feature control strategy of the month
(3) Operators take the ISA exam to be a Certified Automation Professional (CAP)
(2) Executives hang out in the control room
(1) Executives ask operators to autograph screen prints of online process metrics
