Stan: As instrument engineers for Monsanto, Greg and I got to check out and start up all of our projects. In this follow-through that was part of our job description, we learned what worked and didn’t work. Before smart instruments, we had to be smarter in order to calibrate and check out the instruments. Pneumatic controllers, transmitters and positioners were a special trip because they were frequently out of alignment and calibration just from being handled and installed.
Greg: I also spent most of the first seven years of my 46-year career at plants in electrical and instrument (E&I) construction learning how to install motor control centers and automation systems. My first two years were spent as an E&I field construction engineer at a specialty chemicals plant starting up four production units. I arrived just in time for the commissioning of the first unit. Unfortunately, plant E&I would not get involved at all until the unit had completely demonstrated capacity and quality requirements. Just to make it more interesting, construction did not have any E&I people before I showed up and didn’t know instruments should be handled with more care than pipe fittings. The spring-return piston actuators rusted from sitting in water did not work in the batch process where nearly all of the streams were sequenced with these actuators. The project even used the same on-off valves for some of the control valves. The positioners had a crazy pulley system. Poor flow characteristics, excessive backlash and stick-slip of the valves added to the bizarre experience. All of the instruments were out of calibration. The process was new, difficult and not completely known. The combination of instruments not working and extensive process problems made for a traumatic start-up. I learned quickly how bad a start-up can be. I can really relate to this next topic and the extensive experience of Tim Green, operations manager for field services at Maverick Technologies, where he has been responsible for over 50 start-up/commissioning projects and more than 100 construction/commissioning managers, technicians and electricians.
Stan: How did you get started, Tim?
Tim: Coming out of the Navy as an electrician mate, my first job three decades ago was as an E&I technician at a chemical plant. My interest in doing safer and more efficient start-ups dramatically increased after a disastrous start-up with numerous fatalities. We had to start up a large project that was going from pneumatic instruments and analog controllers to electronic instruments and a distributed control system (DCS). The start-up was going reasonably well with the exception of communication and coordination between engineering and operations, which was less than desirable. Operations’ focus was on operating costs and production delays. The engineering focus was on project costs and schedule delays. The lack of communication ultimately increased the duration of the start-up, which placed additional pressure on the plant to push through the remaining start-up activities as quickly as possible. Operations chose to start preparations to run the live process while some remaining construction and commissioning activities were still underway. Even with good lines of communication, this would have been risky and ill- advised. In this start-up, the results were catastrophic. A reactor went exothermic and blew apart. We lost many lives that day and many more people were severely burned. The entire specialty chemicals unit was destroyed.
Stan: What is the major lesson we can all learn besides the need for engineering and operations to have a safe start-up as the most important goal?
Tim: Seventy percent of industrial accidents occur during non-routine operations. For most plants, start-up and commissioning is extremely non-routine. For most engineering companies and integrators, operating plant equipment is extremely non-routine. Safe and effective start-ups require a methodology and extensive coordination and communication to eliminate the non-routine aspects of a start-up. Since there are activities with vastly different expertise and focus, the people who need the knowledge to make every activity routine must readily get it from the people who have the knowledge.
Greg: A critical part of making the operation during start-up as routine as possible is the extensive use of process simulation for testing control systems and training the operators. At Monsanto, after a protracted start-up of a new process in the 1980s, a company standard was developed that required all DCS projects to have extensive testing and training using simulations connected to the actual DCS system, so that the actual control system configuration and operator graphics were used. High-fidelity simulations were done for parts of the process or control system that were new or complex. Virtual I/O was developed that allowed the test of channel assignments, but eliminated the need for wires during configuration testing. In the past decade, the virtual plant was developed so the actual configuration files, including operator graphics and simulation software, could run on a personal computer. Now, the entire automation system and process was accessible, enabling extensive automation system and simulation development and prototyping that could provide added emphasis on "what-if" scenarios, abnormal operations and the behavior of new processes and new control strategies. What did you do to make the start-up/commissioning activities routine?
Tim: We have a three-step instrumentation and controls (I&C) commissioning process with traffic colors indicating the relative immediate risk level. The colors do not take away from the fact that all steps must be done right for a safe and effective start-up. Our process is the product of a lot of people who have done a lot of start-ups and know what can go wrong. The process allows I&C states of construction, checkout and commissioning to proceed in parallel, saving a lot time using a piecewise "heel to toes" approach as areas progress.
The first step is preparation and planning with risk traffic color "green." People tend to slip up here by waiting until the start of their field activities and then trying to do as much as fast as possible. Having people from the start-up team help do the preparation and planning is key. Early involvement is huge. We start working with operations on a dynamic test plan. We get the I&C team selected, data base and checkout sheets prepared, and schedule, start-up procedures and responsibilities documented. P&IDs on the walls are highlighted in green as to construction completeness.
The second step is pre-static inspection (PSI) with a risk traffic color "yellow." Here we go from P&IDs to loop sheets, single-line drawings and motor elementary diagrams being highlighted in yellow and checkout sheets being completed. There are a lot of check boxes to force thoroughness with a minimum of wasted effort. The boxes are organized and worded to prevent blowing off the effort by just successively checking boxes.
The third step is pre-dynamic testing (PDT) with a risk traffic color "red." Systems are energized and design documents are redlined for power, sensor signal simulation, single- point calibration checks and stroking of valves. Motor thermal protection is verified against nameplate data. Rotation is verified with a rotation meter. Signals are checked to appear properly on control room and maintenance system screens. We make sure radio communication is clear and concise. The technician in the field states the tag and location, but not the value of the signal. The control room must repeat the tag and location and say what value is seen on the screen.
The status of final dynamic testing via water runs is the responsibility of operations and is highlighted in blue.
By having the steps defined, there is less chance of skipping something important. The team and the extent of "hands-on" expertise grow as the effort moves form one step to another. For example, as the phases move from green to yellow to red, the number of techs might increase from two to six to 16, respectively.
Stan: What do you do to avoid being overwhelmed and bogged down?
Tim: We divide and conquer. For example, if we have 1,000 loops to check out, our goal may be 250 loops a day. We divide up into eight crews. Thirty loops per day are expected, but if a loop cannot be readily fixed, the loop is given off to a SWAT team so the crew can continue on. The SWAT team has particular in-depth troubleshooting expertise and is in a better position to see common errors. For wide-scale problems, the correction is handed off to construction. In some cases, the problem needs the attention of the mechanical/piping contractor.
Greg: What is done prior to your involvement with the automation system?
Tim: The software acceptance test has been completed. Generally, we are starting up someone else’s configuration.
Stan: What is happening to the workforce?
Tim: We are seeing an aging workforce. We have an ever smaller pool to draw from, since we want people with plant experience. The type of people we use started when instruments had to be understood and calibrated. Automation systems today have evolved to a point where human disengagement has become an increasing concern in terms of preventing an unforeseen problem and quickly fixing difficult application problems.
Greg: The effective use of an automated management system (AMS) is more important than ever. Users need to be able to take advantage of an ever-increasing degree of intelligence from the embedded diagnostics and auxiliary signals of today’s transmitters and valves. Fortunately, the use of such tools can be addressed in training courses. More difficult to fix is the continuing loss of knowledge realized the hard way by things not going as expected. The essential knowledge, in this profession more than any other, is gained from working on industrial applications. The people doing the work don’t tend to have the time or encouragement or skills to publish what has been learned. Most of my effort at this point is documenting as much as possible of what I know.
Tim: We are on the same sheet of music. The main remaining goal in my career is to share what I have learned. I have had many teachers, seen a lot of what can go wrong, and had many "aha moments." This technical profession has been such a big part of my life, I hate to see what I have gained be lost and go away.
Greg: With some help from Tim, we finish with the "Top 10 Signs You Have a Fictitious Start-Up Date."
"Top 10 Signs You Have a Fictitious Start-Up Date"
10. Research is still researching the process, and design is still designing the process.
9. The electrical contractor is doing the loop simulation.
8. Project plan is to use maintenance staff to do the loop simulation when not busy doing plant maintenance.
7. Project plans to use a staffing company to get some technicians to handle I&C commissioning because it is cheaper than using a company with an instrumentation team.
6. One-stop flowmeter shopping at orifice supplier.
5. One-stop control valve shopping at piping supplier.
4. Piping designer thinks the flowmeter orifices are restriction orifices.
3. Engineering, construction, commissioning team and operations are strangers.
2. Engineering does not have worn-out hard hats.
1. A year-end completion date.