Controls and process applications can be restored

June 11, 2013
Restoration possible: Many distributed control systems (DCSs) are rapidly aging and risk breaking down. Luckily, there are many tools and innovative methods for supporting and breathing new life into controls and process applications
About the Author
Jim Montague is the Executive Editor at Control, Control Design and Industrial Networking magazines. Jim has spent the last 13 years as an editor and brings a wealth of automation and controls knowledge to the position. For the past eight years, Jim worked at Reed Business Information as News Editor for Control Engineering magazine. Jim has a BA in English from Carleton College in Northfield, Minnesota, and lives in Skokie, Illinois.

Check Out Montague's Google+ profile.

Youth may be wasted on the young, but there's no reason you and your control systems and process applications can't regain and maintain some youthful strength and couple it with your veteran experience and know-how.

Just as getting up, moving around, walking, biking and sensible weightlifting can get people's blood moving, release endorphins and fuel a sense of well-being, there's a boatload of innovative tools and methods that can do the same for supporting, migrating and reviving distributed control systems and their overall applications—and help them overcome the problems of obsolete equipment and persistent downtime, and reach new levels of efficiency, safety and sustainability. Of course, most process applications go years or decades without replacing many controls and other equipment, but there comes a time when maintenance and repairs must give way to technical advances and more capable components and systems.

For instance, Valero Energy Corp.'s refinery in Benicia, Calif., recently completed a major upgrade and rearrangement of its control systems, control room and networking infrastructure with help from Honeywell Process Solutions (HPS). The refinery was built by Exxon Mobil in the late 1960s and originally used an IBM 1800 system for process control, but later replaced these mainframe computers with Honeywell's TDC 2000 and TDC 3000 systems with extended controllers on Data Hiways for base regulatory control and application modules for supervisory control.

Beginning with 44 nodes, the system grew to more than 60 nodes over the next 20 years. The refinery added a local control network (LCN) as its node count grew, but later replaced it with total plant network (TPN) bridges to maintain the performance of this large system. High-performance process managers (HPMs) were added in 2004. And Valero began planning in 2006 to remove its TPN bridges by 2010. The company also wanted to improve its native Windows (NW) interfaces and reduce the possible scope of failure by dividing its system into manageable clusters.

"TPN Bridge support was going away, and, though our big LCN was working OK, a refinery-wide reliability study in 2003 reported that using it was out of step with the industry. The study pointed out that multiple smaller LCNs were more common in the refining industry," says Denise Plaskett, Valero's principal applications engineer for control systems. "So the best solution for us was to remove the TPN Bridge and add Honeywell's Experion PKS on top, allowing distributed system architecture (DSA) communications between clusters to keep our system whole." Plaskett adds that some of the main challenges to renovating with Experion included:

  • Limited space in the refinery's centralized control room;
  • Seven-week shift cycles that made it hard to get operators in to help with design and training plans;
  • A traditionally do-it-yourself Control Systems group that needed external support for this renovation, even as it continued to support day-to-day refinery operations; and
  • Timing issues that required all integration and cutover to the new system be done on-line, while working around unit turnarounds over a multi-year time period. 

Reasoning the Need

Of course, Valero's Benicia plant is just one of many on the quest to renovate its controls, and though each has many unique characteristics, they also share many common challenges and can use many of the same solutions.

Read Also: Blending Process and Discrete Applications

In fact, a new automation survey by system integrator Maverick Technologies LLC in Columbia, Ill., included responses from 175 technical support, asset maintenance and project planning and execution personnel about their DCS applications and efforts (Figure 1).

Living with Legacy, Seeking to Migrate

Figure 1: More than a third of respondents to Maverick Technologies' recent automation survey report operating with legacy automation systems, while just about one third report they're planning to migrate legacy automaton systems in the next two years.

More than half of the respondents (53.5%) report they're operating with new or updated automation systems, but more than a third each are running legacy automaton (35.7%) or integrating control data into their enterprise (33.5%).

They add the biggest constraints on automation in their plants include budget (28%); capital expenditure priorities (26.5%); resources (19.5%); integration of platforms (9.7%); maintaining legacy systems (8.6%); and planning (7%).

To improve their applications, close to half of the respondents plan to expand with new automation systems (47%) within the next 24 months, while about a third each plan to add remote support for data and controls (37.8%); integrate controls with their enterprise/business systems (33.5%); and/or migrate their legacy automation systems (33%).

For more detailed results of Maverick's survey, visit

Plan is Key, Schedule is King

Before securing the most suitable hardware and software for upgrading a DCS, the most important task is evaluating what's truly needed by the application and the overall business, drafting a thorough plan meet those requirements and following a precise, but flexible schedule.

For instance, Indonesia-based PT. Pertamina recently migrated the DCS that runs its paraxylene plant, which is part of Unit IV at its Cilacap refinery. It worked with Azbil Corp. and local affiliate Azbil Berca Indonesia (ABID), and together they planned a combined cold changeover (CCO) and hot changeover (HCO), minimized switchover time by retaining all sensors, valves and terminal panels in good working condition, and migrated only the plant's DCS to Azbil's Harmonas-DEO monitoring and control system.

Read Also: Addressing Control Applications Using Wireless Devices

"The DCS was replaced while part of the plant was shut down for planned maintenance, so the timing would change depending on how the maintenance work progressed," says Erfan Gafar, deputy section head of Pertamina's project engineering department. "Despite repeated schedule changes, switchover was completed as originally planned without delays."

Likewise, the Langkawi cement plant in Kedah, Malaysia, has two production lines producing 3.3 million tons of clinker per year. The facility is one of three operated by Lafarge Malayan Cement (LMC), and its Line 2 was run by an aging Polysius DCS for which spare parts were no longer easily available.

Smarter Cement

Figure 2: Operators at Lafarge's Langkawi cement plant use their new 800xA DCS from ABB to adjust their process to run at optimal levels.

So beginning in 2007, Lafarge planned and undertook a four-phase, five-year revamping of the line's controls with help from ABB Malaysia and its associated Operation Center in India (INOPC). The third phase shifted key process areas of Line 2 to ABB's 800xA control system in June 2011, and the final phase was completed in June 2012 (Figure 2). This new DCS included with 11 AC 800M controllers for 14,000 I/O points, integration of existing 5 AC 800M controllers for 6000 I/O points, and seven operator workstations.

To upgrade the plant's safety capabilities at the same time, Lafarge also planned to renovate the line's electrical systems and installed visible cut-off, switch-power isolation and control units on more than 300 motors and local control units for 13 high-tension (HT) motors, and added about 100 pulse-controlled speed sensors for critical drives. This allowed 800xA to provide safety instrumented systems (SISs) that comply with IEC 61508 and IEC 61511.

"Our operators now have full control over energy and raw material consumption with 800xA and can keep vital processes running without interruption," says Mohamad Fadzil Ramli, Lafarge's methods manager. "For instance, with a fully integrated control system, the average temperatures in the kiln can be adjusted for minimum energy requirements. When an operator combines process knowledge with 800xA, energy consumption can be reduced by 10%."

Divide and Conquer

Because migrating an old DCS is such a complex project, the best way to carry out its upgrade and replacement plan is to break it up into a logical series of workable chunks.

For example, Golden Triangle Energy Cooperative (GTEC) in Craig, Mo., makes about 20 million gallons of ethanol and 160,000 tons of wet livestock feed per year. It had been controlled by a DCS installed when the plant was built in 2001, but just a few years later its manufacturer reported it was obsolete and would no longer be supported.

"The supplier would still repair some rundown parts, but the refurbishment process could take up to three weeks," explains Roger Hill, GTEC's general manager. "One day of downtime can cost us $35,000 in lost profit, and unexpected shutdowns can cause unsafe conditions for our employees. We started shutting down operations at every threat of a thunderstorm just to avoid the risk of an outage. We were even browsing online auction sites to find backups for our critical components."

To solve these problems and replace the old DCS, GTEC sought help from system integrator Bachelor Controls Inc. in Sabetha, Kan., and together they implemented PlantPAx process automation system and ControlLogix PLCs from Rockwell Automation, and minimized costs by retaining all existing field wiring and devices and reverse engineering control logic and strategies from the old DCS.

Read Also: Advanced Process Control Ain't Easy

The migration was completed during two, scheduled, four-day shutdowns. The first included installation of one ControlLogix controller, Ethernet and ControlNet communications and FactoryTalk View supervisory HMI software. The second involved the complete plant switchover with BCI replacing GTEC's outdated I/O racks for more than 800 I/O points spread throughout the plant, installing the remaining ControlLogix PACs, and checking that each I/O point responded correctly to controller commands.

Restart procedures began on the morning of the fifth day, and Hill reports that downtime and maintenance were greatly reduced. "There are some months that we operate with zero downtime," says Hill.

In addition, PlantPAx also helped GTEC install a new production line for grain-neutral spirits. "We just tied in the new I/O, downloaded the necessary programming to the FactoryTalk View SE software and ControlLogix controllers, and double-checked the I/O points for each valve and input," says Marvin Coker, BCI's senior project engineer. As a result, GTEC has expanded its production capacity by 75%, and is now able to manufacture 35 million gallons of industrial and beverage-quality ethanol per year.

Educate and Execute

Meanwhile, Plaskett reports the Benicia's refinery's migration to Experion began with I/O realignment, including an "A cluster," covering the refinery's upstream instruments and controls, and a "B cluster," covering its blending and other operations.

"More than 2000 points were moved and rebuilt," says Plaskett. "With movement of I/O comes deleting and rebuilding of points, meaning internal IDs changed, and all references needed to be cleaned up. With only one LCN in the beginning, DOC 4000 Express worked fairly well to allow us to identify and cleanup references.

However, we also used—and I would advise doing it—a homegrown cross-reference (XREF) utility based on the Find Names utility, which allowed us to find even indirection references that we also use at Benicia. We also learned it's easiest to do this before Experion is placed on top because priming for the Experion layer is also required."

Following some in-depth training and hands-on practice, the Benecia refinery's engineers and operators set up a testing system to explore the relationship between their LCN and Experion to gain more application-specific knowledge. In addition, because a critical part of any Experion system is the configuration of the fault-tolerant Ethernet (FTE) network, Plaskett added that Valero's engineers also secured help from Honeywell Network Services to set up a hot-standby router protocol (HSRP) in a redundant configuration.

"Using our test system, we were able to conduct extensive testing of switch shutdowns, router shutdowns and FTE cable disconnects," explains Plaskett. "The testing pointed out the need to carefully check operating systems on these network devices and to test, test and test for proper operation."

The detailed design also included an enterprise model database (EMDB), as well as three clusters, A, B and OM, with redundant servers on top of each individual LCN to allow future expansion. Also, each cluster would have its own subnet address, and these would be tied together with redundant routers, allowing for reliable DSA communications, as well as protected access to/from upper-levels plant networks.

To get its actual execution moving, Benecia's engineers decided to start with their graphics conversion. "We had an extensive set of operating graphics and wanted to retain the operators' familiarity with their layout and reduce their learning curve by continuing to use our 20 years of good graphics," says Plaskett. "By interviewing, visiting and getting proposals from several vendors, we found one that had shape libraries and navigation tools that met the ASM-like look and feel we wanted to give the operators, while also being able to take our existing graphic layouts and convert them efficiently to HMIWeb graphics. For this project, we decided to convert only our primary control graphics, which included 165 for A, 195 for B and 42 for OM, which adds up to 402 graphics that were converted and migrated out of 3000 total NW displays. My advice here is that, if you have a good set of DCS graphics, don't change them too radically during upgrade projects. And, for us, though ASM-type graphics are good for handling upsets, they're extremely bland for normal monitoring by operators."

Flexible Tools = Easier Migrations

One of the primary forces aiding DCS migrations these days is more capable software and hardware that can adapt more easily to legacy systems and allow more old I/O points and other devices to continue to run in updated environments.

For example, Tsb Sugar recently decided to consolidate automation and data assets at its 18-year-old Komati Mill in Komatipoort, South Africa, and planned to migrate from its old DCS to a control system combining PLCs, SCADA and even its manufacturing execution system (MES) into a one, overall information server. This would allow Komati to measure and optimize throughput, evaluate stoppages and downtime, predict final volumes, and provide device-layer and area-level bookings to Tsb's reporting layer. However, this plan required renovating and/or integrating new Quantum PLCs, legacy Provox and DeltaV DCSs, LIMS database, and SAP enterprise resource planning (ERP) systems into one data and reporting point. As a result, Tsb and system integrator Control Systems Integration (CSI) implemented ArchestrA System Platform and Wonderware InTouch SCADA/HMI, historian and MES Performance software from Invensys Operations Management South Africa. The plant also adopted AutoSave change management software from MDT Software and Top Server OPC solution from Software Toolbox to connect the PLCs to Komati's new systems.

The project was completed in phases with first-phase implementation during a three-month, off-crop season in January to April 2011, which was enough time to remove the old Provox DCS and replace it with the new, integrated system on Komati's Extraction Line 2. "All the other systems remained the same, and System Platform was built on top of the existing systems to facilitate integration," says Samantha Rabé, CSI's systems engineer. "Overall equipment effectiveness (OEE), LIMS, Provox and DeltaV data were integrated into System Platform to show KPIs on the InTouch dual-screen displays."

Why and How to Upgrade a DCS

In his white paper, "Upgrading Your DCS: Why You May Need to Do It Sooner Than You Think," Chad Harper of system integrator Maverick Technologies lays out some reasons for migrating distributed control systems, and describes how to approach it.


Pieter san Tonder, Tsb Komati's instrumentation engineer, reports, "We're now able to manage low throughput because, for the first time, we can measure it accurately. Real-time feedback to record the plant's lost-time availability (LTA) and overall time efficiency (OTE), and have it available on managers' desktops was unthinkable a decade ago. System Platform's ability to integrate and consolidate data across the different systems and deliver it in a presentable format enables us to be proactive rather than reactive."

Meanwhile, on the software side, PCN Technology Inc. reports it's developed software algorithms and hardware modules that can be dropped into a legacy control system, and allow it to be retained, but still enable it to be accessed by an Internet protocol (IP) network. "This allows migrations at a more rapid pace because we don't have to rip and replace so much," says Daniel Drolet, PCN's executive vice president. "For example, users may have an old Seriplex or RS485 network that works OK, and we can plug into its PLCs, head or service center router to add multidrop IP communications, while still allowing the serial network to continue."

Rich Clark, principal applications consultant in Honeywell's Lifecycle Services business unit, reports, "Many DCS migration plans are using advances in the IT world, such as virtualization, open standards-based devices and HMIs using thin-client computing and eServer produces to put control data in secure zones for viewing by browser-based systems. For example, in the past three years, more migration projects are using virtualized data processing that doesn't need to be implemented all at once. There are many large base stations and applications in Windows XP and Windows 2003 with three- to five-year-old hardware, but Microsoft won't support XP after next year, so many users will need to migrate those stations. Virtualization gives users the flexibility to stretch out their migrations and go longer between the time they have to touch wires and reengineer systems."

"Rip Off the Band-Aid"

To finally upgrade its consoles and layer on its new DCS, Benecia's engineers added a pair of redundant Experion Servers (ESVTs) to their 73-node LCN, and then added two Icon stations to test them on their production system. As these worked well, they upgraded their engineer console. Finally, they added a second temporary console in the center, reaching a total of 10 Icon stations in their system. These provided stations for engineers and operators to preview and test the converted graphics, while still maintaining enough stations for operations in high demand situations.

"Once all three chunks of A's complex graphics were converted, we were ready to train our A console operators. Our graphics vendor trained the operators, and then they were physically relocated to the new center console," added Plaskett. "We left their console in place for another seven weeks, making sure all operators had a chance to work their shift cycle at the new stations before removing the stations they felt most comfortable with. These stations were not side-by-side, but in an emergency, the operator could go back to the console and operate the refinery. This satisfied our space limitation challenge. Once the seven weeks passed, we began demolishing and replacing the old A console, and these operators were moved back to their home when it was upgraded. This method of training and relocation also allowed us to satisfy our seven-week shift schedule staffing challenge."

Next, with the B cluster graphics completed and B operators trained on Experion, it was time to upgrade the B console to Icon stations. However, this could not be done until the old TPN Bridge was removed and a second set of Experion servers put into operation. This bridge removal was preceded by months of preparation. LCN nodes were relocated within cabinets, so nodes on the same cluster were in close proximity and could be re-cabled together carefully.

"Our detailed procedure for making all of these moves included over 120 LCN cable moves. Besides moving and re-cabling all of these nodes, the exact procedure we developed for removing the bridge was reviewed and blessed by Honeywell," explains Plaskett. "On D-day, all operators were moved to universal stations (USs) or used NW on ESTs. The TPN Bridge nodes and switches were all simultaneously powered off. The system complained loudly for two minutes, noting which nodes were now missing from each LCN. LCN re-cabling was done to eliminate these boxes from the LCNs. A console ESTs could not be used until all points were re-primed, since many internal IDs it had known about were now gone. It had to discover these points now as DSA points. The new B cluster servers had to be started up and primed with the points on its LCN. Once primed, they could each offer up their points on DSA. After several days of re-priming, HMIWeb graphics could be used again and B console stations could now be upgraded to Icons.

"At this point, the console upgrades could continue. The B operators who had been at their B console USs could be moved back to the center console. The USs were replaced with Icons, and the B operators were moved back to the newly converted B console. Throughout all of this, OM operations had been a subset of the B complex. After allowing the A and B complex operations to line out over a few months, the OM cluster was finally broken off and created."

Finally, Plaskett reports finding that seven weeks was enough time for Benecia's operators to train on Experion. "Though we heard of other locations where old USs and new Experion stations sit side by side for years, we essentially had to ‘rip off the Band-Aid' and remove the old stations to move through the project and complete it," she says. Also, NW on the Icon has been a blessing and a curse because operators sometimes fall back to using it when they don't think the Experion graphic is responding fast enough. So we're glad it's there to give them a sense of security, but the black NW and gray HMIWeb graphics are a continual source of discussion at our plant. Also, support is becoming a big issue, as we have to do patching, error logs and network support. And simple point rebuilding now requires an extra step, DSPPrime, and even AM point algorithm changes require new priming!

"However, we achieved a successful project completed within the challenges of space, staffing, resources and timing. We maintained an integrated view for our operators. We now have a modern, highly functional operator interface. We've continued evolution along with our Honeywell system, and our investment is secure."  

About the Author

Jim Montague | Executive Editor

Jim Montague is executive editor of Control.