So who do heroes look up to? The flood rescue rangers.
Its been almost a year since the engineers and operators at Coffeyville (Kansas) Resources (CVR) Energy Inc. were notified on June 30, 2007, that the Verdigris River next to their refinery and fertilizer plant was going to crest above flood stage. Today, Day 2 of the 2008 Honeywell User Group (HUG) at the Arizona Biltmore Hotel in Phoenix, Shane Scoles, CVRs manager of automation, reported on how his company and its vendors coped with and rebuilt following this catastrophic flood. He co-presented Recovery from Disaster: The Coffeyville Flood, with G. Amos Rolen, PE, Honeywell Process Solutions (HPS) program management manager, whose Houston-based team helped CVR deal with the crisis.
The water line reached several feet up in our typical control room, and our computer room floor was covered with mud. Shane Scoles of Coffeyville Resources Energy discussed the Herculean efforts by which the companys flooded refinery was made ready to operate in a mere months time.
Once they received emergency notification of the oncoming flood, CVRs automation personnel and other staff immediately began relocating as much equipment s possible to limit damage. The next day, CVR alerted its vendors that it was going to need help with it recovery efforts and named Honeywell as primary contractor for the recovery on July 2. Scoles reported that the river crested on July 3, and assessment and recovery teams were able to enter the facility and began examining the damage that afternoon. Flood depth was 10 to 20 feet near the river, so almost all of the two plants were under several feet of water.
After the river began to recede, we began pumping water back into the river and found damage both from the water and from floating debris and equipment, said Scoles. The water line reached several feet up in our typical control room, and our computer room floor was covered with mud. All of our Honeywell control equipment, cables, connectors and cards needed to be replaced or repaired. We also had many ruined LCN and I/O cabinets. Because theyre located at ground level, about 650 of our 913 valves and analyzers were damaged. Only 1,200 of our 12,000 instruments were damaged because most are close-coupled, and so they were located above the flood.
Our assessment teams just fanned out through the plant, decided on what action take with each device and tagged each one accordingly. We also needed to repair or replace several underground cables.
Rolen reported that recovery efforts really began to ramp up on July 5. Team members received immunization shots, made extensive use of their cell phones because the plants usual communications were out, compiled massive to-do lists of recovery tasks, and started to receive replacement parts. CVR and Honeywell also arranged for a massive infusion of manpower and additional contractors to help with the recovery, and on July 11, set a target date of July 31 for completing the recovery. Many of the staff, vendors and contractors worked 14-16 hour days and on weekends.
We had to divide and conquer and break up all the needed work into groups. It really helped that we had commitments from the highest management levels at all our companies to do whatever it would take to get this recovery done, said Rolen. Control equipment was done by Honeywell. PLCs were done by Keystone Controls. Valves and other Fisher equipment came from Experitec, and wiring and instrumentation came from Delta P/ACT.
These two plants had to be shut down in about four hours under emergency conditions, but a safe shutdown usually takes a week. So to stabilize the facility, we also had to use generators to complete the shutdown before the recovery.
Next, to handle the truckloads of replacement parts and equipment coming in, CVR and Honeywell set up warehouses and depots, consignment purchasing and daily equipment and staff accounting facilities for its many vendors. We had four deliveries per day coming in to two warehouses with the last coming in at midnight, said Scoles. To give you an idea of how quickly our vendors worked, we found a crack in a one valve body at 8 a.m., and we were able to get another shipped and replaced it by midnight that night.
One of the main lessons we learned was dont ever blow the water out of conduit and other equipment because it will go everywhere. In fact, we blew out one conduit section, and the water shut down three transformers. Now we recommend sucking out water instead.
Some bright spots in CVRs recovery effort included the fact that, at the time of the flood, it was the only U.S. refinery that was down. As a result, many veteran assessment experts that had worked at facilities hit by hurricanes Katrina and Rita were free to help at CVRs two plants. Also, it helped that the flood water was fresh, which is less corrosive than salt water. However, the recovery was hindered by a scarcity of potable water nearby, as well as lack of local lodging and eateries that forced most recovery workers to make hour-long commutes to and from the plants.
Even though our refinery is more than 100 years old, this was the first time that it was ever shut all the way down, added Scoles. So it was tremendously useful that CRRMs management was committed to a rapid recovery. We were replacing about 70 valves per day, and we had to recheck the loop for any wire we lifted or disturbed. So it helped that we could talk to one person, and get verbal OKs for many repairs.
Scoles and Rolen and the recovery managers and staff, vendors and subcontractors held daily meetings to refocus on which equipment they needed to have up and running first and by when they needed repairs done. Four accountants also signed off daily on pay and expenses for the recoverys 120 automation professionals, not to mention tracking the incoming truckloads of parts. One contractor reportedly installed 25,000 different parts.
The keys to our success were the daily staff meetings we had with all vendors. They allowed us to update and address any issues that came up that day, said Scoles. It also helped that we had departmental procedures implemented and daily tracking of them.
Because of their super-heroic coordination and cooperation, CVR, Honeywell and their many colleagues and contractors were able to start demobilizing the recovery facilities on August 2 and turn the final unit over to CVRs operations staff on August 3. Refining operations resumed on August 17.
