Refinery marks ‘quick, clean’ cutover to Foxboro DCS

“The Schneider approach resulted in major time and money savings. We made a fast cutover and achieved smooth operation.” Monroe Energy’s Paul Maghan described the recent modernization of the refiner’s OM&S system from an older Honeywell system to an EcoStruxure Foxboro DCS.  

In 2012, the 180,000 BPD Trainer refinery, located in Trainer, Pa., was sold by Phillips 66 to new startup company Monroe Energy LLC, a subsidiary of Delta airlines. Among the many challenges Monroe Energy faced was a well-past-its-prime Honeywell TDC 3000 DCS used by the Oil Movement and Storage (OM&S) department.

“I’m not an engineer,” said Paul Maghan, special projects supervisor, Monroe Energy LLC, to attendees of his session at the 2018 Foxboro User Group conference this week in San Antonio. “I came out of school with a degree in computer science and couldn’t find a job in my field, so I started at Phillips 66 as an operator. After four years, I was promoted to console operator, which is a supervisory role, then to area lead, where I was asked to assist on this project.”

The refinery dates back to 1912 and was owned for many years by ConocoPhillips, a large corporation “where facilities had to fight for capital,” Maghan said. For many reasons, “We almost never got any.” Major failures would be repaired, but improvements were rarely made.

“Monroe is all about making huge improvements,” Maghan said. “Leadership supports us. We’ve implemented multiple projects, and we’re not afraid to try new things.”

A decent proposal

The plant was controlled by four EcoStruxure Foxboro DCS on the process units, a Wonderware system on the wastewater treatment plant, and a Honeywell TDC 3000 in the oil movement and storage (OM&S) department, which handles all the plant’s petroleum imports and exports. It includes two ship docks, a marine vapor control system, a gasoline blending header, two railcar docks, multiple pipelines and 93 storage vessels adding up to 4.5 MMbbl.

“Our Honeywell system was kind of a mess,” Maghan said. The Honeywell support contract had been allowed to lapse, and due to the system’s poorly documented programming, “We had to use engineers for console operations.” Finally, faced with deteriorating system reliability, a request for proposal (RFP) was initiated in mid-2013 to upgrade the system.

“The RFP was comprehensive on a higher level, but it had no input from operations,” Maghan said. “It had the process operations, control system architecture, control philosophy and alarms specified in detail, but it was written to simply replace the existing system.”

Schneider Electric’s RFP was chosen partly because it would be homogenous with existing systems, but also because it promised rapid cutover. “It required no bulldozing—it would use the existing architecture and cabinets,” said Maghan. Schneider also warrantied a range of existing Honeywell components. “We have more than 1,200 I/O, and we kept the Honeywell backplanes, which are included in the new system warranty,” he said.

The plant’s lost profit opportunity (LPO) cost is $150,000 per day, and if OM&S couldn’t operate, the available storage dictated a three-day window or the tanks would run over. “A similar project on another unit using a bulldozer approach took six weeks, and we wanted to avoid that,” Maghan said.

Quick, clean cutover

The Honeywell system had been in service for 26 years, and ongoing changes had been poorly documented. It included out-of-service equipment—a complete ship dock, for example, with its safety systems. It had also been personalized over the years. “One of the employees had programmed what he called ‘Dirtbag’ alarms—Dirtbag 1, Dirtbag 2, etc.,” Maghan said. “If a Dirtbag alarm came up, the operator had to call him and he would tell them what to do.”

Due to such ad hoc modifications and poor documentation, “The factory acceptance test (FAT) extended way beyond expectations,” said Maghan. “The hardware FAT went well but documentation took more time. The automatic software conversion was fast, but led to a lot of configuration errors, so we went to 100% check-out.

“Part of the problem was that our department is too small and we had too many projects going on at once. And because if the importance of OM&S, we couldn’t freeze the Honeywell console.”

Cutover was done in two steps. “In step one, we set up a temporary Foxboro I/A station in parallel with the TDC 3000, defined the critical loops, and cut them over. This let us pretest the loops and gave us a path to reverse if needed—but we didn’t have to do that,” Maghan said. Concurrent operation let the operators see the Foxboro graphics while operating the existing system.

In step two, they planned and prioritized loop checks, doubled the operator coverage, cut over the remote I/O cards and did the loop checks. Again, they could reverse if needed, but it never came to that.

Lessons linger on

In retrospect, Maghan said they could and probably should have used a third party for documentation, to ease the load. Also, “We didn’t realize that some of the new graphics would be very different—our systems have all the running pumps in green, stopped pumps are red. The standard in the power industry is the opposite, so it was confusing at first,” he said.

Maghan also wished they had involved operations earlier in the process. “We should have frozen the design so we didn’t confuse them with running changes, and we should have gone through the nomenclature with them before we cut over,” he said. Later, “We needed to make changes to the graphics to give them what they’re used to seeing,” he added. “And we gave them touchscreens but they wanted trackballs, so we got them trackballs and now we have 10 touchscreens that nobody touches.”

But in the end, “We made a fast cutover and achieved smooth operation,” Maghan said. The initial bids had all looked similar, he added, but most would have required bulldozing of the old system—the costs for which were not included. “The Schneider approach resulted in major time and money savings.”