Greg: I had the opportunity to talk with George Buckbee, the General Manager at ExperTune Inc., the supplier of software for loop tuning and analysis. I was always impressed with George's knowledge and his dedication to helping ISA. George worked many years at Proctor & Gamble followed by a career Sanofi-Aventis before ending up at ExperTune. George brings a lot of the user perspective and experience to his job of helping plants improve their loops.
Stan: What are you seeing in the field?
George: Control loop performance is more than just about tuning. There are huge opportunities. About 20% of control loops are in manual. This is an incredible waste of investment. The cost of a loop is about the same as the cost of a car if you consider the entire design and installation cost. Having the loop in manual is like having the car parked in the driveway. You need to get the car out on the road to deliver results. The same is true for a loop. It is frustrating for management to see an investment not leveraged.
Also, we are seeing a big shift in the needs of plants. Retirements, downsizing, and skill losses have changed what is required. In the past, engineers wanted more and more sophisticated technical tools. Today, there is a higher demand for turn-key services to improve plant performance. ExperTune's acquisition by Metso Automation has made it very easy for us to shift into a services mode, and helps us to have more of a global presence.
I have seen many Distributed Control Systems (DCS) replaced and yet the performance of the plant is the same. Many companies spend millions on upgrading control systems and don't use readily available tools to measure if automation system is doing its job.
Control loops in manual can be safety issue such as those making sure the process does not exceed the operating limits of equipment. A level, pressure, or temperature loop in manual can cause a catastrophic equipment failure and hazardous release.
As an industry, we have kind of accepted 20% as the norm. How comfortable would you be if a pilot was only using 20% of an airplane's control systems? What if the air fuel ratio or hydraulic pressure controllers were in manual?
Read Also: Don't Tune These Four Loops
Greg: We put extensive effort into the necessary job of meeting project budgets and schedules often at the price of simply doing what has been done even repeating mistakes neglecting the real goal of improving process control. Since this is fall and football is on my mind, maybe this is like a football team with new uniforms and a new stadium that has the same old plays and doesn't improve the yardage gained or points scored.
Stan: Given that the loop is not in manual due to legitimate reasons such as shutdowns, startups, transitions and sequences of the unit operation, what are some of the reasons loops are being inappropriately put in manual?
George: Great point. A controller in manual is usually the symptom of some other, underlying problem. Operators will not run a loop in auto if the operator sees the plant as worse off. The operator knows the plant suffers but not how much or why. The goal is not just to get loops in auto but rather to find out what was the cause. Loops in auto may have caused oscillations upsetting other loops due to noise, interactions, backlash and stiction, and improper control strategies besides tuning.
Stan: How do you find the culprit?
George: We have a target rich environment. We have to automate the diagnostic analysis, and also to prioritize among them to find the most important root causes. We do a power spectrum analysis and take a closer look at the 3 highest peaks and see what they have in common in terms of frequencies. In the process we clean-up the data and eliminate misleading information. We look at the wave shapes. Sharp corners such as square waves and sawtooth cycles (ramps up and down) are symptomatic of discontinuous responses from hardware issues. The further downstream you go, the more these oscillations are smoothed out and attenuated by tanks. So the sharpest oscillations are in many ways a good clue as to the source of the problem.
Greg: Normally we associate these sharp responses with backlash and stiction in control valves but they can be caused by poor resolution of speed input cards for variable frequency drives, extended at-line analyzer cycle times, poor resolution thermocouple cards in a 1980s vintage DCS, improper wireless settings, data historian update time and compression settings that are too large, and actuator designs meant for on-off valves. Don't get me started.
Stan: We don't want to get Greg started so let's get back to how do you gain additional knowledge?
George: With so much data history available, we can use engineering rules to find more opportunities. We look for naturally occurring bump tests and automatically develop tuning. We notify operations only of the there is a big change in the tuning settings. Some users have no experience so you need protections against common mistakes like using data from a bump test during a load upset.
Greg: If the settings used are slower (much lower gain or larger reset time) than identified, the problem could be an unidentified nonlinearity or someone messing with the tuning. At any rate, the slower tuning settings can be readily translated into an increase in peak and integrated errors. If the settings used are faster than identified, it could be due to some degradation of catalyst, unit operations, and sensors. Fouling of heat transfer surfaces and column trays can considerably increase process lags in series creating a large amount of extra dead time. The 86% response time of a pH electrode can go from 6 seconds to 6 minutes due coating or aging of the glass.
Stan: Are there some easy pickings?
George: There is a whole bunch of low hanging fruit. We find instruments that are completely dead. Some were never put back in service after maintenance. The faked number and red tag was never removed. The operator loves the faked number because it is rock solid often close to exactly what he wants, which was purposely done to keep the operator happy during maintenance. A simple check to see if the measurement ever moves will find these "dead" instruments.
Greg: Sometimes noise is a clue to the problem. Terry Tolliver, a longtime friend and Fellow Hall of Famer, found out the poor level control upsetting the triple effect evaporator he was working on was due to a an unsecured level capillary system of a recent differential pressure level transmitter dangling and blowing in the wind.
Stan: What can you say to put a damper on this before we get Greg all worked up? He does love stories about dampers about as much as on-off valves as final control elements.
George: In one plant, we wanted to do bump tests on a hot air damper. When we asked to move the damper, the plant said no because the damper was wide open, and the loss in efficiency would be too much, because the damper was a source of free energy. Later we noticed the temperature increased when the damper closed. It turns out the damper was configured increase to open and the DCS was setup to be increase to close. The plant was running with the damper fully closed. Correcting the valve action in the DCS resulted in millions of dollars in savings and increased production.
Greg: Since we are running out of space and time, let's take the big step forward and say we have made sure the automation system is not the limitation, how do we make the big decision on how fast to tune a loop given there is always tradeoff between robustness and performance. The tuning settings for minimum peak and integrated error are nice to know but due to the inevitable operating point and run time nonlinearities and unknown, we have to make the tuning settings slower. The question in my mind is how do you know how much slower? You need to recognize the goal. For example, the purpose of a level loop on a surge tank is to maximize the absorption of variability so flow changes coming into the tank are minimized in terms of manipulated flow changes out of the tank that is inevitably feeding downstream operations. For liquid column and vessel and temperature control, the goal is to minimize the integrated error from setpoint and the peak error particularly if undesirable side reactions can be triggered or exothermic reactions are occurring. For pressure control loops operating near the relief or shutdown point, minimizing peak errors are critical. How do you deal with all the different goals?
George: We give the user the ability to choose tuning methods and the ability to change "safety factors". Engineers love to tweak things. For more complex situations, we provide guidance for such as tuning for coordination of loops. Here we want to make sure the shape of the response not just the timing of the response is identical particularly for blending and maintaining the stoichiometric balance of reactants in ratio flow control systems. Having been in this business for 25 years, we keep finding more heuristic rules on what data and techniques to use are based on purpose and diagnostics. You need to keep adapting what you have. We are always trying to develop new heuristic rules. We provide a continually evolving tool based on plant experience gained.
Stan: Since you have worked for large consumer care and pharmaceutical companies, how do the manufacturing processes and automation system challenges differ than for chemical companies?
George: The spectrum of consumer products is broad and diverse. Quality attributes of what the consumer cares about are difficult to analyze or quantify. You end up controlling secondary aspects.
Greg: For foods and beverages there may be a taste test. For beer, the brew master rules as to changes in the operating conditions. When we interviewed Joe Ruder who is the principal process control engineer for a large pet care company, the consumer or tester could not speak so you have to go by other clues such as the enthusiasm of the pet eating the meal. A similar problem existed as how to control quality attributes that are not measureable. I have worked with some pharmaceutical companies and found that to be a whole different world.
George: The standards for quality and an overwhelming amount of government standards require incredible attention to detail and documentation. For example, the FDA wants to know where the steel comes from in a control valve and proof that the composition is exactly as specified. It is difficult to change anything once it is written down. Procedures in the 1950s to measure temperature with a mercury bulb thermometer are still followed, even though such thermometers are difficult to find and are not as accurate as a resistance temperature detector (RTD). Manufacturers do not even want to ask the FDA to use an RTD because the FDA may then want to review the whole procedure. The Process Analysis Technology (PAT) initiative that is meant to encourage innovation is considered for new but generally not for existing processes.
Greg: It seems that doing anything for existing processes is like opening up a can of worms similar to the problem we try to avoid with the Environmental Protection Agency (EPA). A release no matter how minor may open up all of your permits to EPA scrutiny and possibly tougher regulations depending on the individuals. I remember a 1 second excursion in the pH of a stream entering a surface impoundment below 2 pH or above 12 pH would be recordable violation classifying a pound as hazardous waste even if the pond volume was 1 million gallons and the stream flow was 1 gpm resulting in an immeasurable quantity of acid or base in pond. Don't get me started.
"Top Ten Signs Loops Need to be Tuned"
(10) Console numbers are red
(9) Operator faces are red
(8) Trend charts are off-scale
(7) Operator exclamations are off-scale
(6) Surge tanks are full and downstream units are starving
(5) Food in the kitchen is getting old and operators are starving
(4) Process streams are flaring
(3) Operator tempers are flaring
(2) Loops are retiring to manual
(1) Operators are retiring to fish