While it’s not in the movies, J.R.R. Tolkien famously wrote that, “Water hot is a noble thing,” and the same goes for boilers in refineries. Oil, gas, chemicals and most other modern blessings wouldn’t be possible without old-fashioned hot water and steam, which both require clean water. To keep this crucial resource flowing cleanly and safely at a large refinery in Ontario, Canada, Teng Inc. in Mississauga upgraded its boiler feedwater treatment plant (BFWTP) this past June from an older, programmable logic controller (PLC) system to Rockwell Automation’s PlantPAx distributed control system (DCS).
Glen Rycroft, senior process control engineer at Teng, presented “Upgrade of obsolete boiler feedwater treatment plant PLC/HMI to PlantPAx 5.0” at the Rockwell Automation Process Solutions User Group conference during Automation Fair 2022 week at Chicago’s McCormick Place.
Teng is a 26-year-old, mid-sized engineering, procurement, construction and management (EPCM) firm with more than 145 staffers, who provide project lifecycle services from project development to commissioning/start-up, maintenance services and support such as inspections, optimization and turnaround services, and automation and controls including system integration from concept to commissioning.
Overcoming obsolescence
This refinery also runs 24/7, so there was no feasible way to schedule a full plant shutdown to replace its PLC. Because only a short migration window was available and the plant had to remain online throughout, the new control system would have to be robust and flexible enough to handle it.
The existing BFWTP was controlled by a 30-year-old, obsolete Honeywell IPC-620 PLC with an obsolete version of Intellution iFix human-machine interface (HMI) software. The PLC managed more than 500 I/O points, running about 30 pumps and 100 valves, while its programming was heavily sequenced for operations, such as regenerations and backwashes. The PLC was also connected to a legacy Honeywell DCS for supervisory monitoring.
“The obsolete PLC was no longer supported by the vendor, so it was hard to obtain spares. Meanwhile, PLC component failures were rapidly increasing over the past two or three years. A catastrophic PLC failure would cause a costly plant outage of more than 12 weeks,” said Rycroft. “Likewise, the HMI wasn’t communicating with the PLC because a custom driver on the HMI workstation wasn’t supported, and native files weren’t available. As a result, operator interaction was only available through the Honeywell DCS communications link to the Honeywell PLC. In addition, troubleshooting in the old PLC wasn’t intuitive, onsite staff weren’t very familiar with the software, and Teng’s support was required for detailed troubleshooting or modifications. Also, the old PLC’s programming terminal was well beyond end of life with multiple component failures, and a replacement would be difficult to obtain and set up. Finally, the PLC’s documentation was out of date or non-existent PLC documentation, including control narratives and I/O wiring diagrams.”
Evaluating new controls
To evaluate and implement a new PLC, the BFWTP would need a replacement that was also sequence-based, though this wasn’t ideal for the legacy DCS. The refinery eventually settled on Rockwell Automation’s PLCs because they were already the site standard outside the feedwater plant’s DCS as part of an earlier upgrade. Based on a review by stakeholders, it also adopted PlantPAx due to its ease of operations, enhanced troubleshooting for maintenance, ability to easily complete future modifications, and similar look and feel to a planned, plantwide DCS upgrade.
The refinery and Teng considered converting the IPC-620’s program to Studio5000 software via a third party. However, Rycroft reported they rejected this option because this type of conversion wouldn’t benefit the site because troubleshooting would still require manual scanning using ladder logic, and it wouldn’t provide PlantPAx’s enhanced HMI faceplates for operations and maintenance. Plus, the cost difference was negligible compared with migrating to PlantPAx. In addition, converting IPC-630 to Studio5000 would still require:
- Comparable testing to confirm conversion was completed correctly.
- Using an output conversion tool to maintain the old PLC’s tag naming syntax, and keeping legacy tags and logic, which would make troubleshooting messy.
- Converting tools from ladder logic to function block because the BFWTP’s site standard is function block where possible, and ladder where function block isn’t possible.
- Still having to change process/control loops manually.
- Performing system diagnostics manually.
Likewise, the refinery and Teng also rejected upgrading the feedwater plant’s iFix HMI software to its latest version because the supplier reported that migrating would have still been mostly manual due to its age. “With negligible cost difference to reprogram HMI in PlantPAx, along with multiple benefits of PlantPAx interface, upgrading the old iFix HMI did not make sense for this application,” explained Rycroft. “It also would’ve required about 10-20% more testing without pre-built object libraries.”
Picking out the pieces
Based on the BFWTP’s 500 I/O and required availability, Teng and the refinery selected redundant 1756-L83EP process controllers and racks, each with a redundant power supply. They estimated CPU utilization at 50% and memory utilization at 44%, though actual performance after commissioning was only 30-40%. Other components and software included:
- I/O modules upgraded in-kind with 25% spare I/O points for expansion;
- Modbus RTU communications to the DCS via Prosoft MVI56E-MCM to replace proprietary Honeywell communications link between the DCS and the old PLC;
- Device Level Ring (DLR) EtherNet/IP communications among racks;
- Stratix 5700 switch for communicating with the HMI workstation and engineering workstation; and
- PlantPAx 5.0 DCS with Studio5000, V.33, and FactoryTalkView SE, V.12.
“Before shipping hardware and software, we took advantage of distributor Gerrie Electric’s value-added service of staging all the hardware at their facility in Burlington, Ontario, as per network architecture drawings to reduce onsite installation and testing time,” said Rycroft. “All firmware was loaded, software and patches were installed on the HMI and engineering workstations, and preliminary switch and network configurations were completed. Processor rack redundancy was also configured and tested. Meanwhile, the factory acceptance test (FAT) was completed at Teng’s office, which includes a well-stocked automation lab for testing and training.”
Programming the plant
Before migrating the BFWTP’s new controls, Rycroft reported that Teng also performed several standard PLC and HMI upgrades. These consisted of developing the I/O database, developing and reviewing the new control narrative, developing and reviewing static graphics, and conducting more thorough redundancy testing, DLR testing, switch configuration and IP address assignments during the PLC panels and hardware FAT to speed up onsite testing.
“The original IPC-620 PLC program, which had been one of the onsite standards, was difficult to read, and with one rung per page, it was difficult to search through,” explained Rycroft. “Other PLC functions were located in separate programs, such as I/O conditioning, DCS communications, diagnostics etc. To program the new PLC, each program was divided by device containing individual PlantPAx objects. Sequences were programmed using ladder to interface with PlantPAx objects. Finally, objects were built first, which was done in bulk by using the Process Library Online Configuration Tool.”
Likewise, typical function block layouts were developed with a consistent look and feel, which was programmed for each function block diagram (FBD) for maintenance troubleshooting. All commands to actuate devices were located directly in FBD and commented according, instead of using the old program’s method of searching and cross-referencing.
“Wherever possible, all previous program functions were moved into PlantPAx objects to remove as much custom programming as we could,” said Rycroft. “Complexity of sequences drove them to be programmed separately, interfacing with PlantPAx objects where required. Each sequence was complex on its own, and generally each sequence interacted with at least one other sequence. We used the Process Library Online Configuration Tool thoroughly during bulk development for quick modifications to parameters given the number of I/O points in the system.”
In addition, interlocks and permissives were reprogrammed into PlantPAx objects for enhanced operator and maintenance troubleshooting. Also, local Honeywell PID controllers were migrated by Teng into the new PLC program as PlantPAx blocks. All loops were also retuned by Teng to further improve control. Similarly, storing the alarm database within the PLC and automatically generating it with PlantPAx objects provided many benefits, such as speeding up development by about 5%, and eliminating the need to synchronize and double check the PLC versus the HMI alarm database.
“Integrating local, loop controllers into the PLC gives the operators full control capability from the HMI, and loops are now automatically controlled via sequences, reducing dedicated operator time in the field. This was a big advance,” added Rycroft. “Two of the loops were also previously bad actors that tended to overshoot, requiring additional operator intervention. Now that they’re tied into seeing the rest of the process through the PLC, control is much smoother for these loops, which reduces or eliminates sequence faults due to these controllers.”
Once all this programming was done, the pre-commissioning software FAT was held during two days at Teng, while operations and maintenance training was completed over multiple shifts before and after commissioning at the refinery. “Between the BFWTP’s DCS and PLC, we also completed a significant amount of testing because more than 600 tags had to be transferred via the Modbus RTU link for monitoring and control,” added Rycroft. “This testing was accelerated by using virtual/simulation modes of PlantPAx objects rather than manually creating simulation logic/force bits.”
Commissioning and benefits
During the six-day migration, approximately 25% of the plant remained online, which required installing temporary facilities required for working seamlessly with the new PLC throughout the migration. “I/O wiring migrations from IPC-620 to the ControlLogix PLC required I/O power to be segregated from power for the new PLC chassis. This allowed the PLC to stay online, and let us complete any requested last-minute changes and review items with operations,” said Rycroft. “We also completed I/O loop checks, which were again sped up by using PlantPAx compared to a typical system. Everything was now well organized, easily visible and manipulated from the HMI with no time wasted on searching for I/O in the old PLC program.”
Benefits gained by migrating the feedwater plant to PlantPAx and ControlLogix included fully supported hardware and software from Rockwell Automation with local distributor Gerrie Electric directly involved in system setup and support.
“We’re no longer required to obtain third-party parts for an obsolete PLC,” said Rycroft. “The feedwater plant reduced the time that operations and maintenance spent on plant troubleshooting by 60% thanks to up-to-date equipment, improved controls, HMIs, alarms and diagnostics.
“We also accelerated the development and programming schedule by 20% by using PlantPAx 5.0 and associated tools, which also reduced expenses to the customer. In short, PlantPAx 5.0 played a major role in ensuring the migration stayed on schedule. Finally, this project’s implementation costs were estimated to be very similar to alternatives, while providing a solution with much greater added value.”
