"The operator will be in a state of panic immediately after an earthquake," says Wilkins. "Don't expect complex decisions." He adds that the same holds true for an abnormal situation in the process industries.
David Strobhar, director of the Center for Operator Performance notes, "The actual state of the system needs to be transparent for matching with the state the operator believes [emphasis added – ed.] the system to be in. Mismatches of this type have resulted in significant problems in aviation." In other words, operators need to be able to compare what's really happening in the system to what they think is happening.
Yet as an industry we have been working on alarms and alarm management techniques and technologies for decades, and we have imposed extensive shutdown and safety systems on plants. Clearly these are becoming understood to be band-aids on the real problem.
Wilkins points this out. "In times of abnormal operations, systems are configured to produce lots of data—humans are not configured to handle or interpret it."
Strohbar says, "Operators frequently take manual control due to the 'slowness' of the control system in an upset." Yet, as the data from Texas City and other accidents show, this often makes the situation worse.
The alarm handling can also make the situation worse. In the Deepwater Horizon accident, an alarm cascade so confused the operators on duty that they were paralyzed with indecision beyond the point of no return for the accident.
Abnormal Situations and the Continuous Process Control System
The start-stop nature of batch processing made it necessary for the ISA-88 standard to specify ways to include accurate descriptions of process states in the programming. Most continuous process systems are not designed with recipes and states in mind. This means that when abnormal conditions hit, the operators do not automatically know what to do or how to get the control system to a safe state and gracefully shut down the systems.
SIS or ESD systems are generally designed not to shut down the plant gracefully, but immediately. This can have far-reaching consequences, such as needing many weeks to recover from a single spurious trip. So alarms are often ignored, and sometimes the safety systems are bypassed.
Dr. John Lambshead, professor emeritus of evolutionary biology at London's Museum of Natural History, calls this phenomenon the "leopard in the grass" issue. "This is hardwired primate behavior, not just humans," he says. "The grass rustles, the leopard leaps, and all the primates that are left run. But after the grass rustles a few times with no leopard, things go back to normal until the leopard shows up to lunch again."
Wilkins points out, as in the cases of Texaco Milford Haven and BP Texas City, the inexperience of the operators in handling alarm floods contributed to the disaster, and sometimes it is averted only by sheer good luck. In the case of Qantas Fight 32, when in the midst of a flight from Singapore to Sydney in 2010 one of its engines self-destructed, the pilots were inundated with over 50 alarm messages, but could only display eight to ten messages at a time on their screens. By great good luck, the plane was carrying three experienced captains as well as first and second officers. It still took nearly an hour to prioritize and work through each of the messages—necessary steps to determine the status of the plane. When you add in the age of control systems and their process plants (typically plants in North America and Western Europe are 40 or so years old) and the as-builts that have not been updated since the plant was first brought online, the conditions plant operators face in similar situations are way worse than the one on Flight 32. As they are paging through the procedure manuals, trying to read all the sticky notes, they waste valuable time and do things that make the situation more dangerous, rather than less.
The Abnormal Situation Management Consortium, led by Honeywell and Strohbar's Center for Operator Performance, have been applying lessons from aerospace and military studies of operator effectiveness to the practices and procedures in the process industries.
We know that alarms need to be few and accurate and relevant. Chris Morse of Honeywell says, "Only the relevant alarms should be active at any state, which should significantly reduce the number of irrelevant alarms and let the true alarm conditions be discernable." He continues, "Often control strategies have been optimized to the most common operating state of processes and plans. The less common states are typically left up to manual intervention."
Training the New Workforce
Now add to the mix that the operators and technicians who have grown up with most plants are retiring, and new employees without the institutional knowledge of the more-experienced workers are replacing them.
Habibi says, "There is a significant shortage of experienced automation professionals at work today. Capturing, contextualizing and pushing plant-critical knowledge is key to sustainable safety and profitability of every plant."
Dave Emerson notes, "As older workers retire, there is a stress on engineering organizations. Using a standardized approach for automation engineering helps by letting younger engineers spin up faster, and allows them greater re-use of engineers' past products and techniques."
Wilkins warns, "These younger employees don't seem to want to put in the time to learn and become proficient."
It will take new methods of training to teach them the situational awareness their predecessors got over time. Wilkins continues, "Using different operating states for process equipment makes it easier to explain how it 'looks' in those states, how different alarms may or may not apply, and what moving from one state to another means in terms of different control actions."
In the 1960s, some companies, particularly the Dow Chemical Co., pioneered the concept of state-based control strategies. Dow's Levi Leathers drove the company philosophy toward the development of safe states in the control system design, so that operators' difficulties in making correct decisions in the middle of a crisis were minimized. This philosophy underpinned the creation of ISA-88.
This concept has now come full circle and is being standardized by the ISA-106 Procedure-Controlled Automation standard. Shingara describes it. "State-based control facilitates a consistent and robust control strategy that
ensures that plant equipment operates efficiently and safely under all operating conditions. When standards are developed and systems are deployed consistently across an entire enterprise, operators have a clear understanding of how the equipment operates, are able to cross-train easily, and corporations reap the benefits for decades."
Shingara paints the future for procedure-controlled or state-based automation. "Fully automated, state-based control strategies are routinely used to schedule production allocate equipment, control plant devices, manage analog loops, dynamically adjust alarm parameters and detect abnormal situations. Using this strategy, operators focus more on running the entire plant and less on running individual pieces of equipment [or processes – ed.]. They have a better understanding of overall plant operation and are able to analyze KPIs related to efficiency, quality and profitability. Consequently, operators focus more on business-related KPIs, feel like they play a key and more important role in the company's success and are more easily trained."