When Savannah River Tritium Enterprise (SRTE) built its new tritium gas extraction facility in the years around 2004, project and controls engineer Laura Sheets reports she and her colleagues knew they needed to automate more of their meticulously documented and compliant processes and procedures.
"Disciplined operations is a way of life at my site, including performing actions in a stepwise manner per the instructions specified in paper procedures, a practice that originated in the nuclear U.S. Navy," says Sheets, who is lead engineer for the DeltaV DCS on the project side of the tritium process control groups at SRTE, which is part of the U.S. Dept. of Energy's (DoE) Savannah River Site. "The operations team is considered the owner of the plant, and the operators are responsible for its safe and effective operation, and the control system is viewed as a tool that aids the operators. To decrease the potential for errors in procedure compliance and increase efficiency, we decided to utilize automated procedures."
This extraction facility and three other facilities are used to execute the Savannah River Site’s tritium missions. Tritium, a radioactive isotope of hydrogen, is a crucial component of U.S. nuclear weapons. The supply of tritium in the U.S. nuclear stockpile must be continually replenished because it radioactively decays 5.5% annually to helium-3. SRTE is responsible for providing the nation’s supply of tritium, which it accomplishes in two ways: recycling tritium from warheads and extracting it from target rods. These recycled and extracted gases are purified to produce usable tritium, which is loaded into stainless-steel reservoirs that are shipped to the U.S. Dept. of Defense (DoD) for installation in nuclear weapons.
The Tritium Extraction Facility (TEF) is the newest of SRTE’s operating facilities. Its first successful tritium extraction was in 2007, but it had to solve several procedural operations challenges first (Figure 1).
New jobs, standards for automation
Because consistency isn't a famous human characteristic, all kinds of endeavors have relied on passing down useful methods and best practices, orally at first, then in written and printed formats, and now in digital forms. Procedural operations in process applications is one of these, especially as they transition from text and manual to digital and automated. To help automated procedural operations go mainstream in the process industries, some control engineers and other process professionals have been organizing standards efforts and developing technological solutions to improve and expand automation of procedural operations in their applications and industries. Chief among these is the ISA106 committee and its standards effort, "Procedure Automation for Continuous Process Operations," which grew out of the principles and modular organization if the ANSI/ISA-88 batch control standard. Nicknamed S88, it breaks manufacturing production applications into specific tasks, phases, transitions and other states of being, and organizes them into easily understood flowcharts, which have been extremely popular with users in many industries for managing their operations.
"Procedural automation and the upcoming ISA106 standard are allowing industry to take more of the complexity out of operators' jobs, and put it into programmable systems," says August Tassin, plant support services business development lead at John H. Carter and ControlWorx LLC, and an ISA106 Standards Committee member. "S88 and now ISA106 include great methods for process automation, where a higher degree of process knowledge is embedded into the operating code. In the absence of these methods, as we push more knowledge into control systems, the level of complexity can go up exponentially. This creates a problem for those who use and maintain those control programs. This is why some previous efforts to automate procedures may have seemed unsustainable.
"Traditionally, with single-loop control each transmitter provides a data point through an HMI, but the responsibility for connecting the dots to create a picture of the process state resides with the operators, their training and their procedures. This places enormous responsibility on operators to put all the data points together, decide what it means relative to the process state, and then act to ensure process optimization and safety. Operators spent years developing the mental models to reliably interpret and respond to various operating conditions. If you try to implement procedure automation at a single-loop control model level, it becomes very complex because there are too many procedures at that level. If you want to do more with procedural automation, you have to modularize, and create intuitive relationships between the physical model and the automation system. Otherwise it can become unsustainable.”
Consequently, while it can seem like almost all process control applications are already automated, dozens if not hundreds of tasks, operations, equipment and situations in most process control applications and facilities are still manual or only partially automated. These typically include startups, shutdowns, abnormal situations, equipment status events and many others.
So, despite the care and historical expertise of their operators and engineers, these operations remain subject to human inconsistency and error—and the inefficiency, reduced quality and potentially unsafe conditions they create. Traditional remedies were based on carefully documenting ideal sequences and proper procedures on paper and in binders and manuals, which were filed and reused, but often weren't sufficiently updated or were sometimes lost—though some are now making their way onto more-accessible digitized and computerized formats. Many hard-won best practices are also fading as more veteran operators and engineers retire, and ironically as improved plant operations increase time periods between startups, shutdowns and exceptional events, which are difficult but still instill useful experience and better practices.
"Process managers want to eliminate repetitive tasks and errors so their operators can focus on optimizing processes and improving value-added quality, and modular procedural automation (MPA) can help reduce those errors and increase productivity," says Eric Heavin, advanced solutions consultant at Yokogawa Corp. of America. "Using MPA is like having your best operator running every shift."
Reasoning nuclear needs
Nowhere are procedural operations more crucial than in the nuclear power and materials industries, which are paragons of documentation and proper procedures, but still often struggle to modernize.
Located on 29 acres in the northwest part of DoE’s 310-square-mile Savannah River Site, SRTE is operated by Savannah River Nuclear Solutions for the National Nuclear Security Administration. Its newest facility pulls the gas from tritium-producing, burnable absorber rods (TPBARs) that had been irradiated in commercial, light-water nuclear reactors operated by the Tennessee Valley Authority (TVA).
"We have to move and process a lot of gas to extract tritium, which means opening and closing many valves, either manually or automatically via PLCs and DCSs. This is where procedural operations comes in," explains Sheets. "This site has a long history of using paper procedures, which operations strictly adhered to and performed as written. More than elsewhere, we're very focused on procedure compliance. It's a huge deal if they aren't followed. However, as technology advanced in recent years, we recognized that more automated controls were needed to make operations’ jobs easier, reduce mistakes, avoid skipping steps, and produce better product."
TEF's three main structures include its remote handling building (RHB) for unloading TPBARS and extracting tritium gas, tritium processing building (TPB) for preliminary purification, and tritium support building for housing management, maintenance and support staff. Once extracted, the gas is piped to the tritium loading facility at SRS for further purification before shipment. To process and transport tritium gas safely, TEF's classified processes use a variety of sensors, valves, PLCs, DCSs and other components, and personnel had been weighing how much manual control versus automation is needed. TEF has been using a D/3 DCS from NovaTech and its Paperless Procedures software since its construction, but lately it's migrating some systems to a DeltaV DCS from Emerson Process Management, and is presently considering its Syncade software. In general, automating procedural operations means taking formerly paper-based instructions, and digitizing them into software that accompanies and can interact with their regular process controls.
"The procedures at our worksite are for starting up, operating and shutting down the main production systems in a facility that was built as a greenfield project," says Sheets. "For us, procedural automation was a blending of two automation styles. One style focuses on equipment groups with defined operational states. The simplest are online, shutdown and maintenance, where the basic process control system (BPCS) performs procedural actions with limited operator intervention. The second automation style involves operators authorizing each procedural action step-by-step, and triggering an operational state change in the automated procedure for each equipment group, such as pumps, tanks, beds, etc. The actions are performed directly by the BPCS or manually by an operator.
"Because of our two automation styles, when we adopted procedural operations, we had to decide which parts would be performed by the equipment groups and which would be done step-by-step. We wanted the operators to always be engaged, but did not want them to become bored by performing repetitive tasks, such as the valve alignment associated with starting up a pump. In the past, everything was on paper and we still have dozens of procedures for every process we do, such as turning on furnaces or pumps or transferring gas, but the instructions on paper allowed decisions to be left up to operators' experience and used ambiguous language. For instance, a procedure might call for a transition to the next step after 'fluctuations are acceptable.' So, implementing procedural automation also meant interviewing our engineers and operators, finding out what the acceptable parameters are supposed to be, and rewriting the procedures more precisely to specific criteria that we could program into our computers, which improved product consistency and quality."
Reassurance aids acceptance
To convince operations management to use their cutting edge procedural operations tools, Sheets says the Process Control Group explained the Paperless Procedures software in D/3's controls would essentially "automate paper." This means their automated procedures would look like the text-based, "if this happens, then do this" procedures they had before.
"The control system's automated procedures determine what needs to be done, and informs users what to do just like the written procedures. However, now we also have a current-step function that's highlighted, so operators don't lose their place, and can click to execute when they're ready," says Sheets. "Since then, operations has liked their automated procedures so much that they're adding it to Tritium’s H Area New Manufacturing (HANM) facility. The other reasons they've grown to like procedural automation is that we have a comfortable format that lets them easily understand what procedure they're on, review all steps, and check ahead. It also guides them through each step, so it's impossible to make place-keeping mistakes, which removes another burden.
"The operators know the control system can help, but they can also stop a procedure or process, decide not to execute, and take over if needed. Our operators are still in charge, but they're aided by automated procedures, which is less stressful. Procedural automation was a new concept when first installed, but now operations considers it indispensable. More recently, we're also adding automated procedures to our older facilities, too."
Chris Kourliouros, product marketing director at NovaTech, adds that, "ISA106 is an effort to develop a consistent approach to continuous and batch applications incorporating automated and manual procedures. Operators have always followed procedures, but procedural automation interacts more closely with process controls, fosters consistent procedure execution, and documents everything. We developed Paperless Procedures with users in heavily regulated industries because their challenge is solving procedural problems and making solutions readily available. Procedures preserve and transfer knowledge, so operators can pull up the latest procedure, and execute it while creating the necessary detailed records.
"Many operators still have insufficient details about the steps they need to take and can't answer, 'Where am I and where do I need to be in my process?' Engineers need tools to balance automated and manual process controls. Software such as our Paperless Procedures brings these capabilities together, so engineers can define their requirements, procedure writers can configure the procedures, and programmers can integrate automation with the procedures, resulting in procedures that effectively guide the operator through the necessary steps."
Seeking a step-by-step standard
Besides its foundation in the S88 batch standard, ISA106 also has roots in Dow Chemical Co.'s long effort from the 1960s through the 1980s to develop its Manufacturing Operating Discipline (MOD) process automation system and proprietary MOD series computers, which also automated startups, shutdowns, abnormal situations and other functions. This culminated in its MOD 5 computer in 1980, which was implemented in about 90% of Dow chemical plants worldwide. Many of MOD's procedural operations principles also began to be used more widely when Dow partnered with ABB to enhance its 800xA process control platform, which Dow adopted in more recent years.
"I was the principal designer for Dow and Aramco's joint venture, Sadara, which is the $25-billion petrochemical project that started to make product in 2015. However, when Aramco and Dow sought to document procedural operations, we experienced some problems," says Yahya Nazer, CTO for Control Designer, newly retired Dow Manufacturing and Engineering Fellow and senior technology leader, and co-chair of the ISA106 Standards Committee. "We realized we and our clients needed to know more about procedural automation and state-based control, and so we were asked to engage and educate manufacturers and everyone else about them. That's what really started the ISA106 standards effort three or four years ago."
Nazer adds that S88's modular principles provide essential guidance and similar definition, but because it's designed for pharmaceutical, food and other batch applications, ISA106 needed to establish a new vocabulary and common terminology for automating procedures in continuous processes. Nazer adds, "We did our own procedural operations with MOD 5, but it wasn't as complex," adds Nazer. "S88 has one guy doing each task, and defines many different tasks, people and phases. ISA106 is very straightforward, focuses on one unit, modularizes successive steps within it, and reuses as many as needed. The goal of ISA106 is to simplify and define automating procedural operations for continuous plants."
In their article, "ISA106 concepts of procedural automation," Yokogawa's Maurice Wilkins and David Tennant explain that, "ISA106 uses three key models: physical, procedure requirements and procedure implementation. The models organize various elements of the process to simplify analysis and create steps. Each begins at the highest level and drills down from the enterprise to an individual field device (Figure 2). Moving down the list, each level grows in number, so the lowest level has the largest number of individual elements."
They add that the ISA106 physical model organizes physical equipment according to a hierarchy from the enterprise level down to individual devices, and shows that any level can have procedures, from starting a whole production site to using a single analyzer. The procedure requirements model defines a procedure by describing what must be done to accomplish it, establishes the functional requirement for the automated procedure, and ties these requirements to objects in the physical model. Finally, the procedure implementation model outlines the actual equipment and procedural needs in detail, which allows construction of the procedural software program, function blocks and sequential function chart. Procedures are carried out by implementation modules, which consist of ordered tasks and sub-tasks that perform step-by-step actions. Each module has three execution work items: command, perform and verify (CPV). Command triggers an individual action, perform completes the action, and verify checks if the action was performed or failed. CPV items can be done by a computer or a person (Figure 3).
Nazer reports that the ISA106 Standards Committee's 26 voting and close to 200 total members first spent two years developing the definitions and models that ISA106 will use, and unanimously approved and published their first technical report in 2013. The group has three face-to-face meeting per year, and has monthly call-in meetings. They're presently working on their second technical report, which will cover work processes, and are scheduled to finalize it in June 2016. They're also planning to have the ISA106 standard completed and published by the end of 2017.
"The main question is: do users want their operators to run at the equipment level or at the unit level? At the manual level, operators must manually turn devices on and off, decide when to do it, and be responsible for everything. At the unit level, operators tell the computer to carry out tasks below a certain level," says Nazer. "From cold to up and running, a typical process application may have 180-190 tasks to perform. Each one can be done manually, or procedural automation can help reduce them to about 15 tasks, which is more than 10 times fewer."
Even though most process applications are continuous, increasingly capable and sophisticated controls are enabling many to run years longer without complete shutdowns, startups and unusual events. Unfortunately, staff changes in the interim mean less expertise on the plant floor. "If you're only shutting down and starting up every eight years, procedural automation is the proper way because it preserves the know-how of the best operators for handling those unusual events and abnormal situations. Now, with procedural automation, pressuring up and setting alarms become tasks the computer can do," says Nazer. "The Texas City overfill, spill and explosion in 2005 happened because its system was in manual and forgotten. Procedural automation would have set and controlled its distillation column within safe levels, and enabled safe alarming."
Alex Johnson, next-generation systems architect at Schneider Electric, adds, "Thirty-five years ago, plants turned around every three years, so a person with 20 years experience would complete about six turnarounds. Now, turnarounds are pushed to five to seven years, and that 20-year veteran does only two or three, and they were a long time ago. The tribal knowledge of how to start up and shut down a continuous plant has been lost, so now turnarounds take longer and cost more—not just because of increased project costs, but also due to larger lost production and disaster-recovery costs when things go bad. On top of that, groups like the Abnormal Situation Management (ASM) Consortium have established that operator errors contribute 40% of all plant upsets. That’s an expensive and dangerous number. If we can capture the best practices in procedural automation, we’ll have a chance to improve both areas.
"In the past, you saw some information capture of the procedural parts of a recipe in batch plants. More recently, some people have begun to realize—thanks to Dow—that operator support can be applied to any operation and any plant type. Startup/shutdown are the most commonly mentioned activities, but alarm handing is another. Our workflow software can guide operators to take the right actions, so we don’t have to rely on their memories, especially in tense situations. The more we can automate, the more we can engage the operator in higher-level thought about what's being done. Remember, automation doesn’t mean the operator is disengaged. Instead, it ensures that the operator takes the best path. Of course, a good implementation would allow the operator to adapt to the unexpected, such as adapting and revising a procedure according to the appropriate management-of-change process."
Procedural powers distributed
Probably the most useful application of procedural automation is when it's more closely integrated with DCSs to expand their jurisdiction and improve their performance. For instance, ExxonMobil’s Downstream Central Engineering Office supports global deployments of automated procedures using procedural operations—or Proc Ops—across multiple business units. Consequently, its refineries and chemical plants employ automated procedures to deliver consistent procedure execution with reduced console operator workload, and help improve business performance by reducing transition losses and increasing the amount of time units spend at optimum feed rate.
"Proc Ops is an integrated set of functions embedded in Honeywell Process Solutions' Experion PKS control system to facilitate execution of operating procedures, and Experion PKS Proc Ops can work in tandem with existing TDC 3000 hardware and software," says Rose Thomas, senior engineering associate in the downstream office at ExxonMobil Research and Engineering Co. "Proc Ops helps our operating teams execute procedures consistently and efficiently, allows manual and automated steps to be combined into semi-automatic steps for console operators, and facilitates development of effective operator interfaces to start, monitor and end procedure execution. It also reduces application development and sustainment efforts because its modularity facilitates development of structured automated procedures, such as sequential control modules (SCMs) and recipe control modules (RCMs), and its graphical block models are easier than programming with TDC 3000 CL code."
To decide which process applications and procedures to automate, Thomas says potential users must weigh several factors, including frequency of execution (from years to shifts); degree of console interaction (from in the field to the console itself); number of steps from few to many; parallel activities from little to much; consequence of error from small to great; and fidelity to procedure documents (from using them as guidelines to requiring specific details).
"Experienced operators have the knowledge they need in their heads, so existing procedures become more like guidelines," explains Thomas. "When we talk to our sites, we can improve our procedures and quality by getting to their level of detail, and we can retain that veteran knowledge in our procedures."
While incorrect execution of procedures can potentially cause operational incidents and inefficiencies, Thomas reported that automated procedures facilitate:
- Procedure execution consistency, by addressing procedure execution differences among operators and/or shifts, and by standardizing responses to abnormal situations, which often aren't included in written procedure documentation.
- Retention of procedure execution know-how, by establishing processes for effective knowledge capture, retention and transfer of best procedure execution based on best operational and procedural practices.
- Procedure improvement, by establishing the best, validated site for comparison to future executions, and by leveraging a proven, assured, structured platform to manage and implement procedural improvements.
Thomas adds that ExxonMobil's refineries and chemical plants gain several primary benefits from adopting automated Proc Ops, including better margins in grade transitions and feed-rate changes, reduced workloads for console operators and other staff, improved ability to address abnormal situations during execution, and reduced variability in procedure execution performance.
"Before automated procedure implementation, our sites had margin loss due to product downgrades during grade transitions, limited repeatability of each grade transition, and units away from optimum feed rate longer than necessary," adds Thomas. "After adding automated procedures, they reduced automatic feed rates, effectively managed parallel and time-sensitive process actions, and increased service factors for multivariable advanced control applications."
On the workload side, ExxonMobil's console operators are typically responsible for about seven units, such as reactors or distillation applications, and handle about two or three grades transitions per week on each unit. "Before automating procedures, executing grade transitions was a large part of the operators' workload, it was difficult to convey current procedure status during shift handover, and there were many process alarms and operator changes to maintain levels," adds Thomas. "After automating procedures, operator workload related to grade transition execution was reduced, and they could redirect their focus on higher-value tasks. They also improved shift handover with an interactive operator interface that explicitly displays current procedure status. And, they reduced process alarms and operator changes to maintain levels."
Likewise, before automating procedures, ExxonMobil's operators had to manually detect abnormal conditions during execution and make corrective responses. However, their responses to abnormal situations often weren't fully documented in their procedures, and so they had to rely on operators' training and experience, which reduced the likelihood of timely corrective response in stressful or high-workload situations. "After automated procedures were adopted, operators could automatically detect abnormal conditions and automatically interrupt procedure steps," says Thomas. "They can also automatically implement corrective actions when possible, or hand-off to other operators, and confirm their corrective actions are completed, allowing them to proceed.
Automated Proc Ops even helped ExxonMobil handle abnormal situations during automatic feed-rate reductions. "The main focus used to be handling rate reduction activities in response to alarm floods," adds Thomas. "Automated procedures let operators be available to other units in the scope of their consoles, and focus on handling rate reduction impacts in other units. Automated procedures mean units can continue running and reduced feed rates, and gives us more ramp options for feed-rate reductions."
Finally, before automated procedures, each ExxonMobil site wrote and updated many procedures independently. There was limited sharing of best practices and application designs, so procedure execution performance varied among sites. "Automated procedures have reduced variability in our procedure execution performance, and we now have common, high-level application design, which facilitates application sustainment and capturing long-term benefits across multiple sites," concludes Thomas.
DIY procedures? Talk them out
To undertake a procedural automation project, SRTE's Sheets recommends involving all operators, engineers and any other related staffers from the beginning. "You want people to know about this," says Sheets. "Operations has to be aware about what's going on in their processes, but you also don't want them to be bogged down in monotonous details. You their participation, so they'll learn how much procedural automation can help make their processes better, faster and safer."
Heavin adds that Yokogawa's MPA staff consults with users on transitioning to modular procedural automation by interviewing operators, identifying different behaviors, forming best practices, and capturing knowledge that can be added to their procedures. Users can also implement their procedures with Yokogawa's Exapilot MPA software, which works with Yokogawa's Centum VP and other non-Yokogawa control systems, and has a library of functions that can be modified to quickly build and test procedures based on what an individual plant's operators require. Users can also employ Yokogawa's Sebol sequencing software to enable procedures on its controllers, as well as its competency tracking and documentation software for replacing manual, binder-based tasks and competencies, while still focusing on metrics.
"Standards like ISA88 and ISA106 govern how procedures are developed and structured, so they give users more confidence in transitioning to them because they're easier to understand and maintain," says Heavin. "One guy may write an ISA106-based MPA program, but after he leaves, the others won't have to start from scratch."
Tom Troy, director of the operations and asset management portfolio at Schneider Electric, adds that procedural automation can also integrate with solutions that coordinate people and their workflows. For instance, Schneider Electric's Wonderware Skelta Business Process Management (BPM) software combines batch and embedded workflow functions with its Intellitrak mobile data collection software. "As a result, when users are following procedures and doing data collection rounds, they can use Wonderware Skelta BPM to define what to track, kick off added procedures, guide operators to perform the right actions, and add push notifications to coordinate their staff," says Troy.
When a customer recently asked John H. Carter and ControlWorx to implement procedural automation at their facility, Tassin reports they had to gather process knowledge early in the project execution schedule. "The conversations to create functional design specification were more involved than in the past. Creating higher-value automation work product by embedding more process knowledge takes time and effort, but it results in better control of the process," he says. "You have to talk to process-savvy people about how each control element is supposed to work, and how they work together. In the past, when you had a P&ID, the line between a transmitter and valve, for example, would describe fully their logical control relationship. This is the device layer in the ISA106 physical model. For procedural automation, the P&IDs alone don't have enough information; you also need procedures. Going beyond the device layer, relational logic lets users include more context about what's being done with the process, including as how adjacent parts are operating together. This means identifying devices, which are grouped into equipment models, and then creating collections of these equipment models into unit models, where the procedural control sequences can better reflect the operational state of the physical process. The hierarchy of the physical model helps to define a structure for the procedural logic.
"The goal of procedural automation is to get more from process automation by putting more process knowledge into it, and do it in a way that's sustainable. We can't do that by putting procedures and data into a monster, spaghetti-code program that no one can read."
Future hurdles and advantages
Control Designer's Nazer adds, "We've shown that if you determine the state of your process and write code around it, then procedural automation has a track record of improving process operations. It can increase production by 3-7% on average for all processes when procedural automation is implemented, which can make a huge difference for process plants over the long term. Unfortunately, many users only think of controls as 2% of their refinery or facility's initial costs, and don't pay attention to it in the long term cost of ownership. Despite the benefits of procedural automaton, the process industry still isn't picking up on it. Traditional, one-wire-in/one-wire-out, loop-based, manual control is still the norm, and a lot of available computing power isn't used."
In spite of these hurdles, Schneider Electric's Johnson expects procedural automation to get significant investment in the future. "We have to address operator error rates and the best practices around startup/shutdown and alarm management," says Johnson. "The control world is changing dramatically. Instead of being control-block-centric or control-loop-centric, more focus is being put on automating major pieces of plant equipment in a fashion that includes understanding state, necessary control algorithms, alarm management based on state, and other activities like workflow, which center on the equipment. We envision working with OEMs to create autonomous industrial assets in the form of avatars, agents and objects.
"You can even begin to think of plant assets with these characteristics as rough equivalents to autonomous cars. We then see uniting these autonomous industrial assets to construct a plant hierarchy of automation that matches the physical plant structure. We expect control mechanisms of the future will be based on these autonomous industrial assets that have embedded vendor intellectual property (IP) to create more asset value. It will be something analogous to how DTMs allow instrument vendors to differentiate their equipment, and it will have a significant impact on all aspects of control."