By Ian Nimmo, President, User Centered Design Services.
What is a supervisory system?
The early 1970s were the commencement period for computer control. Memory and disk space was very limited, and the interface to the computer was very basic. The initial control was what we describe today as Supervisory control. The computer had limited output capability, but could influence instrumentation systems by changing the setpoint to PID electronic controllers.
During the startup phase of a continuous polymer process the characteristics of the polymer change. Based on plant throughput different process control setpoints require adjustment. In the old days the operator used to tweak the controller setpoint but would often over compensate and introduce new disturbances in the process. The early computers only changed the setpoints to these controllers but within a couple of years the mathematical model of the PID controller was developed within the computer program. This allowed not only automatic changes to the setpoint but changes to the tuning constants of the controllers also. This was very useful for non-linear processes.
However, the User Interface was still very crude and plant supervisors used to interface to the computer by identifying the address a variable and entering it through a set of 16 piano style keys, then entering new data points using the same keys. The process was very dangerous, especially when we consider that the data was not entered in normal base 10 numbering but base 8 octal or binary formats.
Soon more powerful monolithic control computers were introduced but were limited by small amounts of memory and often no disk storage capability. The Supervisory Control Computer System held the main program and instructions for the target control systems. The Supervisory system downloaded the latest software version to the control system and the supervisor initiated a booting or startup sequence. During the plant control the target system fed information about the process back to the supervisory computer system, including batch records, historical plant data and any abnormal condition reports.
The control system user interface improved slightly allowing operators to change the control computer setpoints and alarm values directly at the control computer system. This, however, required regular updates back to the Supervisory system to allow easy alignment should the computer shutdown and require reloading of the basic program. This meant that the Supervisory Control Computers required several versions of the plant software, the basic or virgin image, a sucked back copy which held the latest setpoints and other data parameters, and often development versions of the software which was either under test or being developed by a programmer.
The Supervisory system became a very powerful system especially on batch processes and as Digital Corporations VAX computers and commercial VMS operating system became available and cost effective, the use of off-the-shelf database products improved the historical data capture and replay capabilities.
As the Supervisory systems continued to evolve, Honeywell introduced a new more powerful Distributed Process Control Computer System (DCS) that could work independently of the Supervisory systems. They had good human interface, historical data capture and replay capabilities and powerful alarm management systems. The Supervisory Systems became a place to store large amounts of historical data and for the optimization or mathematical modeling environment.
Today, the DCS works independent of any supervisory system but has relinquished some responsibility for advanced control to more powerful supervisory systems, however, even this is changing due to the power of the new open system technologies being used for control. It is becoming difficult to tell were the control system starts and ends and if a separate supervisory system as we know it is required anymore.
The ICI Robot story
The development of more capable computers allowed mechanical systems, such as robots, to be used for many highly labor intensive activities that only needed muscle and little in the way of human skills. However, there are some lessons for us as we consider the introduction of this new technology. The robot was identified for repetitive tasks. Also people and robots never should work together. The robots were put into large safety cages and isolated from all contact with people.
I remember my first robotics application within Imperial Chemical Industries (ICI) in the UK. The company were keen to identify and use time saving and cost effective automation. A group of engineers with ideas for applications of this technology was formed and several trial applications were investigated. It became obvious that there were many reasons why people should not trust this technology. The British Government had identified high risks to people from the robots as they were very robust and could seriously hurt a person or even kill them should people get in the way of their movements. The answer was simple. Isolate the robots from people by placing them in high security prisons. Should a person need to enter the room where the robot was working, elaborate isolation equipment was employed to secure the robots. This made them expensive and no longer cost effective.
My first robot was no exception to this rule, I had the great idea to use my robot as a shrink wrapping system for 300Kg bales of staple fiber. The existing bales of what looked like very large cotton pillows were wrapped in polythene which needed to be shrunk by heating the polythene at very high temperatures. The existing system consisted of a large open electric oven with a conveyor through its middle. The conveyor carried the bale from the wrapper to the oven, to labeling and the warehouse for transportation to customer sites. The electric oven was very inefficient and was costing millions of dollars in energy usage, the wrapping was poor due to the large side ears that resulted because of the wrapping and shrinking process.
The idea was to train a robot to hold a hot air gun and effectively shrink the bale by blowing hot air in a motion similar to spraying a car. I had witnessed robots performing this activity very effectively in the automobile industry. The first step was to develop a gun suitable for this activity and a local company soon came up with an effective prototype which we could use with the robot. By chance another part of the company had been trying to use robots in the manufacturing of explosives and after an unfortunate accident the explosive was re-classified and no electrical equipment was allowed during the manufacture of the product. This allowed me to acquire the perfect robot for my application and one that had already been paid for by my company. With little negotiating the robot was transferred from Scotland to England and was on its way to save the company a lot of money and produce a better package for our customers. What could go wrong?
The robot arrived in bits but with the aid of some untrained personnel and in true engineering tradition we figured out how to re-assemble it. The basic computer was connected and after a little experimentation programming the robot in assembly language was a breeze and the robot responded to my every request. The sight of such new and exciting technology set the political wheels in motion. The craftsmen who would have to maintain it saw it as a large threat, the operators who were to interface with it could not make their minds up and senior management needed to be convinced. So my boss decided that we should gather all the interested parties together and show a demo of the capabilities of this new technology.
What can you do in 48 hours with no real equipment and no time to do a correct installation? We came up with the idea if we could put a box in front of the robot and write the name of a the senior executive who would be present we would have demonstrated the flexibility and precision of the equipment. Again programming was easy and soon we had a working demo, now we had confidence that we could win everyone over to this new idea and effectively use this technology. At the end of the demo I would give a short presentation on the real application and the benefits and savings.
On the day of the demo one of the mechanical craftsmen noticed a small hydraulic oil leak and without questioning he recognized that this required his skills to just go ahead and fix it as he would do with many other pieces of plant equipment. However, the very small leak was not going to be fixed by a couple of turns of his wrench so he took the pipe off and went to find new olives to seal the connection. He was not aware that the pipe was special, was designed for very high oil pressure and that the pipe must be installed the correct way round. A simple oversight? As the audience assembled after a few brief words the demo was started and to my horror the robot went crazy and instead of writing the executives name on the carton it used the felt pen as a weapon and destroyed the carton with several lethal blows, after this it with one simple blow knocked the 3 foot square carton towards the amazed audience then drove the pen into the ground as if it had a bayonet into a dummy until all resemblance of the pen was totally gone. By which time I was able to make a safe path to the emergency button on the console. Without any discussion I was told to get rid of it and never discuss the word robot to anyone ever again.
Configuration management as root cause
This was a large set back for technology and this Engineer learnt never to do anything by half measures. Every piece of equipment introduced should have the protection of competent people working on it, professional installation and clear operating procedures and management of change control. If we had the system identified as a plant asset and it had been put on the maintenance system the craftsman would not have worked on the robot without a work-order and a permit. The work would have been reviewed and the correct replacement parts would have been identified from the manufacturers recommendations. The incident was one failure but as we looked back over the history of what went on the craftsman was also in a dangerous situation during his maintenance activity as he was not aware of the high pressures, the system had not be isolated correctly and could have seriously injured him. The only isolation was that the computer program was not running and traditional electrical and hydraulic isolation which was not done would probably not have been sufficient based on the capability of this beast.
Why did this one event temporarily stop progress?
This one incident had a wide impact across the whole workforce. Peoples fears and beliefs were biased on the one incident and every robot became a bad thing despite success in other parts of the company. People wanted to believe that the technology was unnecessary and dangerous and the old way of doing things was the best. After all it had been that way for hundreds of years why change?
Lesson: Attention to Cultural issues
The demo was the wrong thing to do, I thought people wanted to see how impressive the technology was but they really didnt care how good it was, or how flexible and easy to program it was. They were only interested in the impact on jobs, the change in responsibilities and training required to be competent on this equipment, would the existing people be capable of supporting and maintaining the equipment. I truly believe that this was the right solution, it could have saved large amounts of money and been very efficient and produced a better product.
The ICI Robot AGV Story
Well after the success of the first robot and many job security threats after such an expensive failure as the money for the robot was transferred onto my divisions books and as far as I know the robot is under a wraps somewhere in the company and was never used. Well the same plant was having difficulties transferring the same bales of fiber from the baling machines to the wrapper and to two different storage locations. The existing transportation was via computerized hydraulic hoist. The hydraulics were shot and the leaking oil contaminated the fiber but more importantly the hoist dropped the bales on the floor and a physical operator would have to drive the hoist in manual control and pick the bales of the floor and transport them to the wrapper, unfortunately the unique identification of the bale had been lost and the automatic labeling system would get out of step and put the wrong label on the wrong bale, which often would cause a major customer relations problem and contamination of whole batches of clothes which would result in customer compensation for hundreds of thousands of dollars in compensation.
Well conveyors didnt work very well with this product as the small tuff fibers would get into bearings and seize the conveyor in a short time, a new hoist would have the inherent problems with hydraulic systems and would not be very reliable from past experience. So what would be the best solution? Well obviously a robot truck Automatic Guide Vehicles are used in many warehouse applications andf are designed for carrying problem loads. The advantage of the AGV would be that it could also maintain the orientation of the bale out of the box of the baler gaurantying perfect repeatable wrapping everytime and perfect orientation for autoamtic labeling evertime.
So no problem all we had to do was write a capital expenditure request get some money and do it, well there was one problem I was not alowed to use the word robot wiithin 20 miles of my management team!
What did we do differently?
Well this time before I mentioned the R word I meet with the Unions and the operations folks and identified and confirmed the existing problems and issues. We all agreed that we could not accept the current way of working and we needed a better solution. For an almost fully automated plant the last place the operators wanted to interact with the process was the bale handling and labeling area, they had bad memories of John Bull label printing kits and complaints of boredom and human error.
During the discussions we reviewed all the options and reviewed conveyor transportation issues but agreed we would investigate and get quotations for the best available conveyors. After the initial investigation we arranged for sample conveyors to be tested and as in the past the loose fiber founds its way into the works and they didnt work anymore. The union and the operators concerned that we were not keeping up to date with progress demanded that we find other alternatives surely technology has progressed so that we could move a 300kg bale of staple fiber a couple of hundred yards and maintain orientation for wrapping and labeling? So my management demanded we come up with other alternatives.
Well this was the opportunity I had been waiting for to introduce the R word but again recognizing there were cultural issues still outstanding from the first R failure. So this time I worked with the manufacturers to identify all these issues and to demonstrate what other industries were doing about them. So with the aid of a couple of films the team was assembled to review what other folks do. We showed a frozen food warehouse with extreme cold areas were people did not want to work and how AGVs are being used, how they interface and physically participate with people. We showed how full size automatic forklifts were handling difficult products and how reliable and accurate they were. How they could maintain orientation because they followed guided wires burried in the floor. We also presented an estimate of the proposed system which was significantly less than the proposed conveyor system. We also presented how we could maintain the trucks with the engineer visiting the site and working with the existing maintenance group.
It was the Union that recommended that management make a case to justify this work and that they would support the successful implementation of this system as it was the only sound alternative to a major problem that was now a bottleneck to the whole business. Well what could management say, the whole work force was united behind the AGV solution.
The system went in fully computer controlled and worked from day one without any problems. We identified the potential threat of non-plant operated forklift drivers seeing these as a threat and crashing into them so we isolated all non plant personnel by a barrier. The bales were transferred to the manual fork lift drivers through a storage rack which divided the handover area between AGVs and forklifts. It was interesting that the operators took pride in the AGVs from the early days and demonstrated their acceptance by naming them and creating faces on the glass shields of the fork. Each truck had its own personality which the workers identified by watching the way they worked. The final acceptance was expressed in the form of a cartoon were the AGVs were depicted as a team members taking a break and playing dominos, and the plant manager pulling his hair out as he had recently had a campaign to keep people on the job and out of the mess room.
What did this do for progress
There is nothing like a successful project to assure employment and give you the flexibility to introduce more of the same. The robot truck system was fully controlled by a supervisory system which tracked bales and handled grade changes and handover to the warehouse for putting into campaigns for customers delivery. The supervisory control computer was an important part of the system but most people probably did even realize that it existed. The only time people interfaced to the system was so request information about the product and where it was in the plant or to call a truck to do something unusual such as store and retrieve bales with a unique identification during wrapper maintenance or other outages.
What culture lessons were learnt
Most people are not interested in technology only how it impacts their way of life, they are more interested in what they will see and how they deal with problems and failures. Modern industrial plants are using technology in every operation and people are stretched and often not considered partners or co-workers. The AGV system demonstrated how people and equipment could work in harmony and be one team. Preparation for change is important and winning sponsors is extremely important. People do not like to face the unknown they like to see well managed and professionally implemented projects. Having a supervisory system was not important it was the functionality and how people identified and worked with it was all that mattered to them.
The Formentor story
Technical success but maintenance failure
Formentor is a research program being implemented in Europe under an Eureka (project #19) by CAP GEMINI SOGETI, the JOINT RESEARCH CENTER of the Commission of the European Communities and AEROSPATIALE PROTECTION SYSTEMS. The goal of this team was to develop a risk management solution in the form of an intelligent watchdog. Formentor is a decision support system that provides the process operator in charge of process supervision with:
- situation assessment,
- diagnosis support,
- reactive planning,
- prediction capabilities
Formentor was intended as an added-value system to the existing control systems, to provide a global and permanent overview of the process state, helping to react to deteriorated or disturbed operation, allowing the anticipation of severe process states by detection of precursor signs and malfunctions, measuring decisions impact on the process behavior, limiting the shutdowns and improving quality of the products.
Lesson: Nonmonolithic solutions
The Formentor team have completed two industrial prototypes one in BP Grangemouth in Scotland, and the other at Total in LeHave, France. Both of these applications have been abandoned by the industrial sponsors due to the high cost and required knowledge skill set to maintain the application. As small changes in the plant equipment or the process operation occurred over time specialist computer experts were required to change the models within Formentor.
Also the display for Formentor was not incorporated within the control system platform but on a separate independent system. This meant that when an operator had a disturbance they would have to move away from their main view of the process and their only means of real control the DCS control system. Operators will not do this regardless of the extra information that may be available in a box somewhere else in the control room. For a diagnostic or decision support tool to work it must not add to the confusion but must seamlessly work with the existing control system. The Abnormal Situation Management (ASM) team have discovered that there are distinct modes of operation one being normal operation and the another being abnormal operation and the interface for the two modes may be completely different. What is required is context sensitive views based on what is happening and what is being achieved. Formantor tried to maintain both views which could be confusing to the operator and may show conflicts that do not exist.
Designing this type of solution independently for each plant application is not cost effective and may not be necessary as we consider the history of plants and processes and the number of similar and repeatable faults.
Segmentation of problem space
The ASM team have confirmed that the type of problems presented by the petrochemical industry are not unique but any single fault needs to be resolved from thousands of variables with hundreds of possible scenarios relative to a single failure. The trick is isolation and elimination in a lot of cases. This type of workspace does not lend itself to a single solution and we have discovered that multiple assessment and evaluation/comparison techniques are required. The production of tools to eliminate or identify sensor faults is only one step in the process. Early identification of the initial occurrence of the fault is important and unfortunately the fault may be masked by control system compensation before the fast and often catastrophic consequences occur.
The User Interface domain currently supports reactive response to problems, however, this is not to successful due to the speed of response required and the elimination of nuisance and conflicting data. The introduction of diagnostic information needs to managed and the lessons learnt from poor alarm management practices needs to be applied. Formentor had a interesting User interface that visualized risk to existing safety, quality and production goals but the lack of context with the existing control strategies and information limited its use to the plant operator.
Multiple applications for each problem
Formentor has died within BP and Total due to support and integration issues and AEGIS must learn the lessons that have led to its down fall. Poor and inconsistent Human Interface has resulted in many failed advanced technology problems in the past. Solving a single unrepeatable problem is not a cost effective business proposition. The idea of identifying common problems and having common repeatable tools is by far a way of making progress in this area. By eliminating all the recurring faults will allow the man and machine to work together to resolve the more complex and unidentified problems.
The Advanced Air Traffic Control system story
Billions spent solving hard technical integration problems, but human factors issues intractable; system abandoned.
Lesson: When designing supervisory system, design for the USER
The AEGIS Story
AEGIS or the Abnormal Event Guidance Information System1 was not designed in isolation by creative thoughts of individual scientists. The ASM program spent over a year identifying the problem, creating Functional requirements and investigating available technologies. From the collective sources of information a basic design was implemented, tested and refined until a model of the solution was created. This model we call AEGIS and is being tested against known industrial problems.
OK, paid attention to cultural issues, nonmonolithic approach, and it is user centered... Success not known, but issues include:
- integration with existing technologies
- emergent effects of connecting previously isolated functions
- Paradigm shift-impact on role of DCS in operations
- Impact on the culture of operations
- Field now in the loop again
- accommodation of various styles in member companies
- Need to prepare the way for the paradigm shift--just introducing it will fail.