By Walt Boyes
Process plants, chemical plants, refineries, offshore platforms, water and wastewater utilities and power generation and distribution systems that form a large part of the global critical infrastructure are made up of complex systems which are, in turn, made up of complex systems made up of simpler systems down to the level of the sensor, controller, and final control element. Traditionally, safety systems have been designed in a vacuum and poorly integrated with the plant process and the basic process control system. So, too have cyber and physical security been designed and implemented in a vacuum. The failure to account properly for the interaction of these complex systems in safety system, control system and alarm management system design was specifically named by the Baker Report as one of the factors in the BP Texas City disaster in 2005, and can clearly be seen in other incidents. Human factors engineering has been often disregarded, and operator training has never focused on dealing with the complicated interactions of these complex systems. The author presents a case for integrating safety systems, security systems, human factors, alarm management, and operator training in a unitized system to deal with the problem of complex systems.
We have been working on making our process plants safer for nearly 50 years now, and we've made some progress. But as we have seen, with the incidents at BP in Texas City, and others since, we are far from making our plants as safe as they can be, and as safe as they should be.
And in 2008, we had another incident, this time at a Bayer facility in West Virginia.
Many dedicated professionals have spent years working on standards and implementations to make our plants safer. But there are a large number of issues– vectors, really, that determine whether we have a safer plant or not.
There is a highly complex interaction between a large number of those vectors. Safety, security, alarm management, operations, training, and of course, your company goals all interact, and, like any complex system, simply changing one vector makes more changes than can often be visualized or calculated in advance.
No one expected the operators to have difficulty seeing both the inlet and the outlet flows to the isomerization process and the raffinate splitter tower at BP Texas City. No one expected ALL the level measurement devices on the tower to fail at the same time. No one expected the safety system to fail. No one expected that the operators would consistently make wrong decision after wrong decision as they tried to recover from the impending disaster. No one expected the diesel pickup truck to be running in the same area as the cloud of hydrocarbon vapor.
Yet all of these things happened. And people died. There have been many more accidents in the three years since the BP disaster, and there will be many more. And many more people will die.
We need to start thinking about safety, security, alarm management, operations and training as an integrated whole, and we need to have our companies agree that the safe way is the most profitable way. We have not done this yet, and until we do, people will continue to die.
Our first attempts to build safety systems took the form of dedicated systems that were stand-alone and completely separate from the basic process control system. This was done to ensure that these multiply redundant systems shared no points of failure with the control system itself.
This was both good and bad. The good news was that the safety system could only be used for one thing. It was strictly to shut down the plant if something abnormal occurred. The bad news was that it encouraged safety practitioners to develop a curious tunnel vision, so that the interactions of the safety system to the rest of the plant were often not investigated.
So, a few years ago, we began integrating safety systems and control systems…with many engineers still unwilling to do that to this day. What we learned immediately was that there were those interactions, and that a Safety Instrumented System cannot be built in a vacuum. It must be part of an overall proactive operations strategy that includes safety, security, plant operations and maintenance. Probably the best example of what I am talking about is what Dow Chemical's Levi Leathers called for, almost fifty years ago, an operating discipline in which safe operation is the most important engineering rule of the company.
Safety systems are part of the control systems in the plant, and safety systems must be considered in any cyber security strategy we implement. Even a traditional standalone SIS system could be penetrated and damaged if connected in any way to a control system or to the plant information network. At the 2008 ACS Cyber Security Conference, Bryan Singer, co-Chair of the ISA99 cyber security standard committee, and Dr. Nate Kube, a principal of Wurldtech Security Technologies demonstrated a hack of an integrated safety system (one that has received TUV approval and is being sold today). In less than 25 seconds, they were able to cause the system to force it to fail unsafely. Other hacks have been demonstrated to work against traditional stand-alone systems, too.
This illustrates the absolute fact that safety and security interrelate. And so do perimeter security, fire and gas safety controls, and personnel locating technologies. In this age of integrated systems, nothing stands truly alone.
In the 1960s and 1970s, operators were able to get an instant grasp of the operating condition of the plant or the part of the plant they were responsible for by looking at the panel wall. We gave up that viewpoint by migrating to small screens where only a part of the process could be seen. Now we are moving back to screen walls and working on visibility issues. But for years, we've had real problems with giving operators more ability to see the eagle's eye view of their processes and we wonder why their tunnel vision leads to accidents that certainly should have been prevented.