“Could we cut over fast enough so the process didn’t get away from us? Also, we had to convince operations that we weren’t crazy.” Marathon Petroleum’s Ed Bullerdiek
Refineries and other process facilities routinely upgrade and even change control systems while the process continues to run, a procedure often called a hot cutover. But they’re usually done one loop at a time, minimizing the time each loop is in manual and the potential for the process to go out of control. Foundation Fieldbus (FF) puts multiple transmitters and valves on a single segment, raising the stakes when it’s time to change out a fieldbus interface module (FIM).
Marathon Petroleum’s Detroit refinery successfully upgraded the FF-networked system on its gas/oil hydrotreater without shutdown recently. The starting point was two Honeywell C200 controllers and 20 FIM2 Fieldbus interface modules with 220 Foundation Fieldbus devices and 403 measurements and valves; 11 Series A conventional I/O; heater/compressor SIS handshaking; and 585 control modules. Hot cutover was to two Honeywell C300 controllers and 14 FIM4s; Series C conventional I/O; and SIS communication via Peer Control Data Interface (PCDI).
“When we went to the existing system from a Bailey system, we didn’t do everything as well as we wished we had. We had 220 FF devices and 403 FF points on two C200 controllers. We had no spare capacity,” said Ed Bullerdiek, process control engineer at the refinery, to attendees of his session at Honeywell Users Group Americas 2019, this week in Dallas. The C200s were designed to be redundant, but they would no longer failover.
“Also, programming of the 585 control modules (CMs) was not up to standards. Standardized programming improves support—technicians can understand it quicker in the middle of the night,” Bullerdiek said. “It was time for an upgrade.”
Moving FF from one system to another was not unfamiliar, but they hadn’t done it on critical controls. And, on this system, the safeties and e-stops are in the DCS, “so we had to be very careful,” Bullerdiek said. “Could we cut over fast enough so the process doesn’t get away from us? Also, we had to convince operations that we weren’t crazy.”
Methodical methodology
To answer the speed question, “we had portable FIMs and C300s in a box, so we could test our hot cutover procedure,” Bullerdiek said. With some effort, “we got it down to 15 to 20 minutes from time out to time back on for each segment. If we did critical instruments first, those could be back online in 10 minutes.”
With speed established, Honeywell and Bullerdiek built a project schedule spreadsheet—a plan that described the sequence, risks and special considerations for all the segments. Working with the production team in half-day meetings over several weeks, “we risk-ranked all the segments, and documented the specific risks and their mitigations,” Bullerdiek said. “For example, one of the bypass valves is undersized. We had to cut charge rates the day we changed over that segment.”
Where a control scheme is complex, they planned to get the entire control done in one day so they wouldn’t have to revisit it. “We wrote a script for each segment to be sure we would do everything, with notes about cautions and special circumstances,” Bullerdiek said. “Make sure you have enough FIM licenses. We didn’t, but were able to empty the FIMs, harvest the licenses and move them to the next job as we went. Plan so you don’t get yourself into a corner where you need another license to get out.”
They also checked the physical condition of the work. “Can you remove the wiring covers, or are they rusted on?” Bullerdiek said. “Is there water in the segment protectors? Verify that communications are working—can you ping the box? Check the diagnostics on all the segments and fix any problems—replace any bad transmitters. If you can’t fix them, note them because, after cutover, you’ll own them.”
Hot cutover by the script
“Honeywell helped us write scripts for all 39 segments that we cut over in three weeks,” Bullerdiek said. Keep track of which segments are done and where you are in the process, so you don’t skip or repeat steps, he advised. Notify the operator of the risk level associated with risky segments, so they can be alert to any problems.
With the integrated safety interlocks, “we told the operators, ‘Whatever you do, don’t shut off the heaters,’ because we weren’t sure we could get them back on,” Bullerdiek said. “And, we brought in extra operators from other shifts to do the necessary field work while we worked on segments.”
When it’s time to cut over, first, inactivate the devices and unassign them from FF. Then delete them from the segment and start the field team moving the wires. While they do that, move and reload the devices, and move the CMs. When the wiring shows up, turn it back on.
Cutover is best done with two people to allow cross-checks and avoid mistakes. “We could usually do three segments per day, sometimes four,” Bullerdiek said. “Use a field calibrator to verify each segment. Then, stroking the valves from the control room verifies that the segment and wiring are correct.”
After cutover, there will be follow-up work. “Deleting CMs breaks all the links to other CMs, the historian and alarm groups,” Bullerdiek said. “I made a spreadsheet to keep track of these and then came back and reloaded them.”
The plant held off on the demolition work until the hot cutover was complete, so no one could get overzealous and remove something we might need, Bullerdiek said.
“We’ve done a lot of cutovers,” Bullerdiek concluded. “Production told us this was the smoothest one yet,” he said.
