Phillips 66 operates facilities and pipelines throughout Texas and across Louisiana. A project to upgrade its automation system at the finish process terminal in Pasadena, Texas, involved the truck rack and pipelines for 21 tanks storing 3.2 million barrels of gasoline, diesel, biodiesel and ethanol.
The Pasadena facility had undergone several expansions over the years, each handled by adding a new controller, typically in a new location. It used 32 SLC 500s and one PLC-5 with a Wonderware HMI, connected by a single run of fiber-optic cable. “The perimeter of the facility is over a mile, quite a large area, and data requests had to be sent from controller to controller,” said Asa Ange, CAP, engineering supervisor, Champion Technology Services, Sulphur, Louisiana, to attendees of his session at Rockwell Automation Process Solutions User Group (PSUG) 2017 this week in Houston.
[sidebar id =1]
Phillips 66 contracted with Champion Technology Services to update the system. A process system integrator for Rockwell Automation, Champion was founded in 2000 in Louisiana as a system integrator in industrial automation and controls. Over the years, it expanded its geographical and services ranges and expanded to Texas in 2008. It now offers a broad range of specialties in automation, power, information and safety systems for industries from oil & gas and chemical to food & beverage, marine and utilities.
Growth pains
At the Pasadena plant, “the communications system was a complicated spider web, with multi-hop messaging to get data from a controller in one place to where it was needed,” said Ange. “It was hard to manage and hard to troubleshoot, and frequent trips led to long periods of downtime.” A message from a truck lane might travel to the truck rack, terminal, and back to start a pump, passing through multiple interlocks in other controllers. The interdependence of the controllers meant there was no way to take a controller down without taking the whole facility down.
The Wonderware HMI was rendered in many colors with animation, which didn’t conform to current standards, and was inconsistent with other Phillips 66 facilities. “It’s very distracting when a truck drives across the screen, but I gather that approach was common at the time the HMI was made,” Ange said. At the same time, the information to the operator was limited. In particular, interlock information was inconsistent and incomplete.
The flat, single-line fiber-optic network meant one point of failure would take out everything downstream. “It would be a site outage,” Ange said. “The network screen would show a connection was down, but no information about why.”
Historical data was incomplete. “Some data logging was done within the controllers, which led to memory overloads and down controllers,” Ange said. Equipment status information was transferred from shift to shift using Post-it notes on the console.
Document, develop, implement
Along with a lack of system documentation, the project constraints included requirements to limit downtime to three days, re-use the existing I/O racks and fiber-optic network and loop-check every end device.
“We started by documenting the network. We took pictures of the insides of the cabinets and figured out every wire,” Ange said. For every I/O, Champion determined wire color, location, purpose and HMI usage. They went through the logic and added rung comments, tag descriptions and routine names. “We made a message map of all the data entering and leaving each controller; on average, each controller sent data to three others. The cause-and-effect diagram took up a 10-ft section of wall,” Ange said.
Champion then developed “control narratives” for each piece of equipment, describing its I/O, their locations, permissives and interlocks. “This was very helpful and important for writing the new code and later transferred well to PlantPAx,” Ange said.
Along with using PlantPAx logic and HMI, Champion proposed to modernize the controllers and top end of the automation system, and convert the existing PLCs to remote I/O. The new solution eliminates the SLC 500s and segregates the terminal and truck rack into two separate systems. Flex I/O remote racks were added where needed to receive segregated truck rack I/O. The new network follows the Purdue model, with levels 2 and 3 on VLAN networks, and levels 0 and 1 on a device-level ring.
A new industrial data center (IDC) based on Rockwell Automation E2000 has two physical ESXi servers, a network storage array, two network switches and redundant power supplies, “an addition made by Champion,” Ange said.
PlantPAx controllers and HMI were chosen because they adhere to HMI and Phillips 66 standards and because they’re tested and proven, come with Rockwell Automation documentation and are easy to use. “The technicians can go to classes and learn it, and take that knowledge to other facilities,” Ange said.
The HMI tones down the colors but retains the high density of the overview screens, using colors only where needed to attract attention to problems and alarms.
Startup and benefits
“Checkout and startup went well, and within the three-day limit,” Ange said. Benefits include eliminating all of the problems of the old system.
A detailed interlock list tells what caused a shutdown, with direct navigation to the specific interlock, equipment and I/O location. “Maintenance bypasses are available on the PlantPAx faceplates, so instead of hopping from controller to controller, technicians can navigate to an interlock and bypass it if needed,” Ange said.
Segmented networks allow technicians to take the truck rack down without affecting the pipelines, and redundancy keeps it reliable. “If you have a single loss, it shows you where and turns yellow; a full failure will alarm in red on that rack,” Ange said.
The new historian gives improved trending, tracks transfers and inventories, and allows technicians to look back and troubleshoot, Ange said. The asset management system does management of change, tracks terminal operations and movements, logs truck rack operations, and allows operators to log in and create notes for other operators. “No more Post-it notes,” Ange said.
The current architecture allows for easier expansions, segregates control networks by functional operations, resolves spurious trips and inconsistent operations, and conforms to current industry standards. The PlantPAx DCS provides a common framework that allows for configuration vs. programming, improved diagnostics, historical data and asset management, and “it’s future-proof due to the virtual environment,” Ange added.
Due to success at Pasadena Terminal, Champion has partnered with Phillips 66 to deploy similar solutions at Lake Charles Pipeline and other terminal facilities, Ange said. “As we speak, my co-presenter is not available because he’s installing a similar system at another facility.”
