To Hell and Back

How to Rebuild a Control Room in Five Days. Lessons in Disaster Recovery and Emergency Response

Share Print Related RSS
Page 2 of 3 1 | 2 | 3 View on one page

Planning

"Strategy without tactics is the slowest route to victory. Tactics without strategy is the noise before defeat. "  - Sun Tzu

Immediately upon learning the extent of the damage, GVW calculated that the 16ML of treated water stored on site would supply the township of Kilmore for approximately five days under stage four water restrictions.  Understanding that the plant had to be delivering treated water within five days, a team of three operations IT people bunkered down in a conference room on Sunday evening, in a scene reminiscent of the movie "Apollo 13": "We need to find a way to make this fit into that using only these…"

White Board
White-board plans for restoring a control room in five days.

Assessment. The first challenge at hand was gaining access to the site itself for inspection.  Because some of the fires were deliberately lit, massive swaths of charred land punctuated with dead bodies were declared crime scenes, and town police blockaded the roads.  Since GVW was an essential services provider, the county police in the incident control center granted its personnel permission to enter the site.  But in a classic case of bureaucratic discontinuity, the town and county police radio systems operated on different frequencies, making communication difficult.  It took several hours to propagate word through official channels.

The shipping container control room shell.
Emergency

Equipped with a laptop, data projector and whiteboard, the planning team quickly decided that its scope would be limited to ensuring successful water treatment within the allowable time frame.  In other words, "nice to have" was put on the bus.  For the next five hours, the team worked into the early morning hours of Monday devising a plan to do just that.

Strategy. In another fortuitous development, one of the GVW employees came up with idea to rebuild the control room inside of a shipping container, and as luck would have it, another had a relative who was in that business and just happened to have an insulated container close by.  This proved to be crucial in several ways.  First, it would save time by eliminating the need to construct a new building.  Second, the container could be shipped to the operations center where it was fitted in close proximity to critical personnel and equipment (the Kilmore plant was located approximately 90 minutes away).

The team drafted a conceptual design for rebuilding the control room inside of the container and sketched the layout of its components.  Next, it devised a detailed shopping list including cabinets, cables, motor starters, PLC equipment, computer equipment, lumber and a suitably sized generator (on- site power was estimated to be as many as three weeks away).  The team entered these items into a spreadsheet and emailed lists to vendors in the middle of the night in order to have as many parts delivered at daylight on Monday morning.  Finally, the planning team devised a timeline for construction, installation and commissioning.  Flexibility and contingencies were the order of the day.

 

Tactics. While the core team focused on construction of the control inside of the shipping container, the frenetic procurement efforts continued.  Vendors cooperated by driving to distant warehouses in Melbourne to pick up supplies so that construction efforts were not interrupted.  At the outset of the project, management understood that fatigue would be a risk, so they ensured proper rest by making alternate rostering arrangements.  These proved effective, as the project was completed without any injuries or near-misses.

Delivering the goods to get the system up and running.
Emergency

As the construction pace of the control room picked up Tuesday, coordination became key.  Due to the restricted space, only five or six workers could be inside the container at any given moment.  While the initial plan proved sound, not surprisingly, the electricians and technicians improvised along the way. 

In parallel with the construction of the control room, site demolition was underway to remove unsafe roof structures and walls.  Next, field wiring was carefully cut from the existing system and labeled in order to facilitate interconnection later.  Meanwhile, corporate IT was busy at work getting equipment in place for wireless connectivity at the site.

Corporate staff assisted the core team by providing a steady stream of food and refreshments.  They also delivered a morale boost by organizing a massive barbeque on the afternoon before the container was due to ship out.  Other staff members were busy arranging for on-site catering to feed the workers and campers to shelter them during sleeping hours.

Execution

"No battle plan ever survives contact with the enemy." - Field Marshall Helmuth Carl

Assembly. Construction activities proceeded more or less according to plan.  Along the way, personnel and couriers dashed to suppliers and warehouses for additional parts and additional local contractors were brought in to insure that deadlines were met.  The shipping container was in place at 5:00 p.m. on Wednesday afternoon, and the installation work commenced.  In a major change to the planning, the electrical provider restored power in three days instead of three weeks.  This meant that the team did not need to implement run-sequencing of the generator, but that a switchboard had to be located and installed prior to commissioning.

Page 2 of 3 1 | 2 | 3 View on one page
Share Print Reprints Permissions

What are your comments?

You cannot post comments until you have logged in. Login Here.

Comments

No one has commented on this page yet.

RSS feed for comments on this page | RSS feed for all comments