Control system migration may be complex, but identifying, addressing and resolving its many problems can make it an especially rewarding challenge for the automation professional. A team at Total Petrochemicals & Refining has been modernizing the TDC 2000 system in its Port Arthur facility since 2005, and learned many lessons they shared this week with other attendees of the Honeywell Users Group (HUG) Americas Symposium in San Antonio, Texas.

The Port Arthur Refinery (PAR) began life in the early 1900s as an oil terminal for Spindletop. It's been a refinery since 1936 and now has a capacity of 174,000 barrels per day (174 MBPD) of transportation fuels. "By 2005, the systems we had installed between 1982 and 1992 were at and over capacity," said Randy Conley, Total supervisor, DCS/SIS/APC implementation, who presented the work on behalf of himself and fellow team members Lanny Gibson and Bert Labath. The 18,000 active I/O are on eight data Hiways – 80% TDC 2000, 20% TDC 3000 – and unit segregation had been lost.

"First, for anyone considering a DCS migration, I highly recommend the book, Control System Migrations, by David Roessler, published last year," said Conley. "I wish I'd been able to read this book before we began our project."

Project Approval Is First Hurdle

Before you can begin, you have to get the project approved, and selling a major migration is not a typical tech presentation. "You'll be talking to high-level management," Conley said. "The stakes are higher and the information needs are different – they want to know, what's in it for me?" Quantify the benefits and speak to persuade, as well as to inform.

Open with a strong, two- or three-page executive summary. "Take out the acronyms and make sure you speak their language," Conley said. "Many decision-makers won't read beyond the summary."

The following seven or 10 slides can explain the assumptions, acknowledge the costs (frame them in a positive manner), provide a simplified schedule and list recommendations. Propose "fit for purpose," not state-of-the-art, emphasize results and give a solid wrap-up as a conclusion. Move supporting details to an appendix or a hidden set of slides.

At Port Arthur, the migration planned eight steps, one per data Hiway, prioritized according to capacity, equipment age and maintenance history. The original schedule was to begin a new step every 12 months to distribute capital commitments and comply with Honeywell's announced end-of-support date. "Each step took 18 to 24 months to complete, resulting in overlap," Conley said. Since refinery outages could not be planned, "We adopted a philosophy of hot cutover at the junction box," he said. Done one loop at a time, hot junction cutover minimizes risk of upset, eases operator transition and decouples the project from operations, but also requires space and electrical power to handle both the old and new systems. Some systems did not have enough room and had to be staged at a remote location. "That added about $1 million," he said.

Choose Your Team Carefully

Conley offered specific advice about choosing a main automation contractor (MAC), manning the project team and managing the hot cutovers. "Use industry contacts to identify potential MACs," he said. His team narrowed a list of 12 to three and awarded them contracts for FEL (front-end loading, or feasibility) studies to estimate costs ±10%, then used the FEL results to prepare the capital request for the first step. They factored those results to get an estimate for the entire refinery ±50%. "After the first two steps were completed, we re-estimated the remaining steps based on our discovery items – our lessons learned," he said.

PAR's project execution team has three full-time roles: project architect, and DCS and SIS project managers. Other team members are temporarily reassigned from their regular jobs as needed. PAR contracts with a MAC and a configuration/graphics specialist to round out the project execution team. This "weak matrix" approach intimately involves plant personnel in designing and implementing the system they will later use, but can slow the project, as members must focus on the plant first, then the project.

"Operations has a key role in the project's success," Conley said. From cable tray routing and enclosure locations to cutover order and risk analysis, "Everything around the hot cutover – they're essential to that." They'll also approve graphics updates, help develop operator training, support factory/site acceptance tests (FAT/SAT), plan and implement the hot cutovers, and maintain the operations punch list. "Assign a chief operator early on," Conley advised. "The CO sells the project to coworkers, which reduces griping."

Tips for a Successful Cutover

The guidelines for determining cutover order include going from the back to the front of the unit and from simple to complex loops. If the next step includes an HMI upgrade, try to cut over all points on a graphic before moving to next one. If an issue arises, skip that loop and come back to it later.

PAR uses two hot cutover (HCO) teams – one doing the current cutover, and one planning the next – each with a systems hardware specialist, as well as two contract and one Total instrument technician. A construction coordinator directs outside activities. A DCS systems administrator directs the inside (control room) team, which includes an extra console operator, two contract configuration specialists and two MAC engineers. The teams are scheduled four 10-hour days (Monday through Thursday), and achieve about 35 points per day. Fridays are reserved for rain-outs and to address issues identified earlier in the week.

Lunches are catered to improve efficiency. "The cutover team costs about $15,000 per day," Conley says. "You can spend a lot more than you expect." Hot cutover is when you'll make a major number of discoveries. "You'll find a surprising number of things," he said. "We averaged more than 175 per Hiway on the first two. Decide how you'll handle them ahead of time, because you'll want to fix them all as you find them, and you can't do that at $15,000 per day."

Log all discrepancies (loop sheet errors, range changes, questionable configurations, graphic errors), address high-priority issues immediately using an expedited management of change (MOC) process, and hand off other issues to process support engineering for follow-up. PAR uses discoveries to improve performance on subsequent steps, so previous lessons don't have to be relearned. "By the third one, I think we were down to 12," Conley said. He offered the audience a complete list of "lessons learned" by e-mail request.

The Journey Continues

In 2008, PAR added a Deep Conversion Project (DCP) that would delay the next TDC 2000 replacement step for 30 months. The DCP required nine new Experion C300 controllers in the brownfield to handle 2,850 additional I/O. Additional greenfield units added 12,000 new I/O and two consoles. The project also replaced 45 legacy operating consoles, updated approximately 400 existing operator graphics and installed a refinery-wide fiber-optic backbone.

The central control building was modified to fit additional operator consoles, which gave PAR the opportunity to move the servers to a more secure location. In the process, "We found a lot of abandoned and mystery cable and coax under the floor," Conley said. "We had to be careful that we didn't lose anything."

The fiber-optic backbone runs on air-blown fiber (ABF) installed in a backbone of plastic multi-tubing that has as many as 19 pathways, each able to accommodate as many as 24 fiber-optic cables. Fiber can be quickly and easily blown into or removed from the individual pathways. "Multi-mode, single-mode or mixed fiber can be blown at a rate of 50 meters per minute and in continuous lengths up to 2,000 meters," Conley said. Since the cable is not pulled through the conduit, no strength members or jacketing is needed, and ABF is substantially smaller – a six-fiber ABF cable is 2 mm in diameter, as compared to 12 mm for conventional fiber. "It's guaranteed for 25 years when it's installed by a certified contractor," he added. "We've been using it since 1996."

PAR has now completed five of nine planned steps of its TDC 2000 migration, "on schedule and in budget," Conley said. "With Honeywell's Hiway Care extension, beginning in 2012, we lengthened our program schedule by planning a new step every 18 months. This extended schedule will complete our TDC 2000 conversion in 2023."

Conley left the audience with five takeaway lessons from Port Arthur:

• Read the book, Control System Migrations.
• Choose your MAC and MAC project manager carefully – rely on customer recommendations.
• Develop detailed, comprehensive RFPs for FEL development and detailed design/engineering/commissioning.
• Consider using a configuration/graphics specialty company.
• Maximize your local staff's involvement for long-term success.