To Hell and Back

How to Rebuild a Control Room in Five Days. Lessons in Disaster Recovery and Emergency Response

Share Print Related RSS

By Jon DiPietro

Between February 7 and March 14, 2009, more than 400 bush fires across the state of Victoria, Australia, scorched over a million acres of land, killing 173 and injuring 414.  Engineers at Goulburn Valley Water (GVW), provider of urban water and wastewater services to 54 towns and cities on the outskirts of Melbourne, watched as their telemetry system from the Kilmore Dissolved Air Filtration plant reported an ambient control room temperature of 142 °F before going silent on the afternoon of Saturday, February 7.  A site visit on the following day revealed that while the treatment plant survived the fire, its control room was completely incinerated, destroying the electrical switchgear, plant HMI, laboratory, instrumentation and chemical dosing systems.  With only five days worth of water stored, an emergency response plan to rebuild the control room and recommission the plant went into action.

What followed over the course of the next five days is a case study in how proper backup and change management procedures, strong vendor relationships and dedicated, cross-trained employees can pull off near-miracles.  Armed with a full set of accurate as-built drawings, up-to-date PLC programs and HMI computer backups, GVW was able to assemble a portable control room in the parking lot of its operations center, deliver it to the site, connect thousands of control points, commission the plant and resume delivery of 10ML per day of water to the towns of Kilmore, Wandong and Heathcote Junction in five days. 

Preparation

"If you want to make an apple pie from scratch, you must first create the universe."
- Carl Sagan

In the months and, indeed, years leading up to the fire in Kilmore, GVW had built a foundation that would give it this chance at success.  There were three key areas of preparation – documentation, human resources, and relationships – and lacking any one of them would have spelled failure.

Documentation. One of the most obvious and crucial requirements is documentation, and GVW was very well prepared in this respect.  Employing strict change management controls, the IT department had current backups of the PLC programs and HMI systems running at the plant during the time of the fire.  Just as important, it had accurate "as built" drawings of the plant stored off site. 

However, one aspect in which it was not as well prepared was the settings on components such as variable-speed drives, loop controllers and flow valves.  As the saying goes, sometimes it's better to be lucky than good, and GVW had its fair share of luck.  In one case, the settings for a critical flow meter had not been stored, but after extricating it from the debris and powering it up, engineers breathed a tremendous sigh of relief to discover that it was still in working condition, and they were able to record all the settings.

Human Resources. During an emergency response, documents, plans and procedures will only get you so far.  Humans must do the work.  GVW had a good internal team with a wide range of skills, including an electrician and an instrumentation technician with over 30 years of combined experience at the company.  The IT department had made the decision years ago to "embed" some of its personnel with computer and networking experience in the operations group.  The result was a diverse team with an important balance of skill and experience that proved crucial in executing the response plan. 

As GVW's IT Manager Noel Squires describes, "A number of years back, we had a couple of near-misses with losing ladder logic, which was at a time when looking after PLCs and process controls was really considered operations and had nothing to do with IT.  But we pretty quickly realized that it's got a lot to do with IT, and that disciplines we use in IT routinely, such as like change management and backups, applied equally to process control.  So we formed the Operations IT group, and that was a little different in that we were taking these former operations people and putting them into the IT section."

Relationships. It's a cliché to say that teamwork is critical in these situations, but ideas become clichés for a reason; they are usually true.  Having said that, there were three specific relationships that had to work in order for this emergency response to work as well as it did.  The first was the horizontal relationship between disciplines.  The team of workers with electrical, SCADA, PLC, construction and IT skills already knew and worked with one another for some time.  The second was the vertical relationship between the response team and management.  While the response team was busy planning and executing, management was fetching coffee, arranging meals, filling out paperwork, providing accommodation and otherwise providing support and blocking distractions as much as possible.  Finally, the external relationships with vendors were equally vital.  The control room fire occurred on a Saturday night, which necessitated a Sunday response.  Many of the vendors abandoned family barbeques in order to open warehouses and drive to Melbourne so that parts would be at their doorstep first thing Monday morning.  In some instances, parts were delivered without purchase orders under a "gentleman's agreement" to simply settle up accounts later.  This can only be accomplished through solid customer-vendor relationships.

Planning

"Strategy without tactics is the slowest route to victory. Tactics without strategy is the noise before defeat. "  - Sun Tzu

Immediately upon learning the extent of the damage, GVW calculated that the 16ML of treated water stored on site would supply the township of Kilmore for approximately five days under stage four water restrictions.  Understanding that the plant had to be delivering treated water within five days, a team of three operations IT people bunkered down in a conference room on Sunday evening, in a scene reminiscent of the movie "Apollo 13": "We need to find a way to make this fit into that using only these…"

White Board
White-board plans for restoring a control room in five days.

Assessment. The first challenge at hand was gaining access to the site itself for inspection.  Because some of the fires were deliberately lit, massive swaths of charred land punctuated with dead bodies were declared crime scenes, and town police blockaded the roads.  Since GVW was an essential services provider, the county police in the incident control center granted its personnel permission to enter the site.  But in a classic case of bureaucratic discontinuity, the town and county police radio systems operated on different frequencies, making communication difficult.  It took several hours to propagate word through official channels.

The shipping container control room shell.
Emergency

Equipped with a laptop, data projector and whiteboard, the planning team quickly decided that its scope would be limited to ensuring successful water treatment within the allowable time frame.  In other words, "nice to have" was put on the bus.  For the next five hours, the team worked into the early morning hours of Monday devising a plan to do just that.

Strategy. In another fortuitous development, one of the GVW employees came up with idea to rebuild the control room inside of a shipping container, and as luck would have it, another had a relative who was in that business and just happened to have an insulated container close by.  This proved to be crucial in several ways.  First, it would save time by eliminating the need to construct a new building.  Second, the container could be shipped to the operations center where it was fitted in close proximity to critical personnel and equipment (the Kilmore plant was located approximately 90 minutes away).

The team drafted a conceptual design for rebuilding the control room inside of the container and sketched the layout of its components.  Next, it devised a detailed shopping list including cabinets, cables, motor starters, PLC equipment, computer equipment, lumber and a suitably sized generator (on- site power was estimated to be as many as three weeks away).  The team entered these items into a spreadsheet and emailed lists to vendors in the middle of the night in order to have as many parts delivered at daylight on Monday morning.  Finally, the planning team devised a timeline for construction, installation and commissioning.  Flexibility and contingencies were the order of the day.

 

Tactics. While the core team focused on construction of the control inside of the shipping container, the frenetic procurement efforts continued.  Vendors cooperated by driving to distant warehouses in Melbourne to pick up supplies so that construction efforts were not interrupted.  At the outset of the project, management understood that fatigue would be a risk, so they ensured proper rest by making alternate rostering arrangements.  These proved effective, as the project was completed without any injuries or near-misses.

Delivering the goods to get the system up and running.
Emergency

As the construction pace of the control room picked up Tuesday, coordination became key.  Due to the restricted space, only five or six workers could be inside the container at any given moment.  While the initial plan proved sound, not surprisingly, the electricians and technicians improvised along the way. 

In parallel with the construction of the control room, site demolition was underway to remove unsafe roof structures and walls.  Next, field wiring was carefully cut from the existing system and labeled in order to facilitate interconnection later.  Meanwhile, corporate IT was busy at work getting equipment in place for wireless connectivity at the site.

Corporate staff assisted the core team by providing a steady stream of food and refreshments.  They also delivered a morale boost by organizing a massive barbeque on the afternoon before the container was due to ship out.  Other staff members were busy arranging for on-site catering to feed the workers and campers to shelter them during sleeping hours.

Execution

"No battle plan ever survives contact with the enemy." - Field Marshall Helmuth Carl

Assembly. Construction activities proceeded more or less according to plan.  Along the way, personnel and couriers dashed to suppliers and warehouses for additional parts and additional local contractors were brought in to insure that deadlines were met.  The shipping container was in place at 5:00 p.m. on Wednesday afternoon, and the installation work commenced.  In a major change to the planning, the electrical provider restored power in three days instead of three weeks.  This meant that the team did not need to implement run-sequencing of the generator, but that a switchboard had to be located and installed prior to commissioning.

Teams worked around the clock to get the control
room working again.
Emergency

During the initial planning phase, the team estimated that there was enough water to supply the township for about five days.  They based their calculation on historical consumption rates during prior stage-four water restrictions.  However, it became clear that – for whatever reason – the towns were actually consuming water at a higher rate than expected.  GVW made the decision to begin bringing in tankers of fresh water to avoid running out.

Installation. The team dealt with a series of setbacks, including underestimating wiring challenges, incorrect solenoid valves and a fuel line issue with the generator.  It overcame these issues sometimes with brute force (i.e., more manpower) and sometimes with ingenuity.  In the end and after some long nights, the wiring was completed and the plant was ready for commissioning.

Commissioning. The commissioning began on Thursday afternoon after the generator was connected and operational.  The team members checked motor direction using two-way radios, which proved crucial throughout the commissioning process.  On Friday morning, personnel checked valve operation and performed some troubleshooting of wiring issues and mechanical problems with pumps.  By Friday afternoon, the water quality instruments were interfaced to the PLC and the plant was churning out good quality water. 

Work went on around the clock.
Emergency

At 9:00 p.m. on Friday night, the storage tank began filling as the team carefully supervised the plant into the early hours of Saturday morning.  On Saturday morning, the plant continued to run while the final commissioning tasks were carried out, including the resolution of various instrument, motor, valve and dosing issues.

Conclusion

GVW IT Manager Noel Squires said about the ordeal, "Strangely enough, when we looked back, we actually rebuilt that control room for about $440,000.  Now if you did that as a contracted job, it's probably going to be like $1.5M to $2M.  We didn't need multiple quotes, didn't need all the bureaucracy, and we just cut the chase and did the job." So the obvious lesson learned is that the next time you need to rebuild a control room, instead of bidding it out, just set it on fire!  But seriously…

The completed shipping container control room.
Emergency

With a combination of solid emergency preparedness, cross-trained and motivated personnel, strong vendor relationships, solid planning, and – yes – a little bit of luck, GVW built a brand new water treatment plant control room in a box and had the plant running in five days.  The experience of the individuals involved even allowed for the implementation of some on-the-fly improvements to the plant, so that it was actually running better after the ordeal than before. 

The fire at Kilmore and ensuing response has left an indelible impression on the organization, both reinforcing some existing practices and influencing organizational change.  The change management system, once frequently overlooked, has been institutionalized to a greater degree than it was before the fire.  Control buildings across the authority are being reassessed and redesigned to be more fire resistant.  Finally, says Squires, "And I think we're trying to avoid falling for the old mistake where you're always preparing for the disaster you just had.  You've kind of got to think, 'Well, what's the next one going to be?'"

Acknowledgements
Many thanks to Noel Squires, IT Manager, and Ryan McGowan, Operations IT Coordinator at GVW, for their time, photos, presentations and reports that were essential resources for the preparation of this story.


Jon DiPietro is Principal, Bridge-Soft, LLC

About Jon DiPietro
Jon DiPietro is the founder and principal of Bridge-Soft, LLC, which provides environmental data management software to water & wastewater utilities and environmental labs.  He is also the author of the upcoming book, [ital] Social Media for Engineers & Scientists[ital]  from Momentum Press, a frequent speaker on Internet marketing and social media, and a certified Inbound Marketing Professional.

Share Print Reprints Permissions

What are your comments?

Join the discussion today. Login Here.

Comments

No one has commented on this page yet.

RSS feed for comments on this page | RSS feed for all comments