resizedimage215281-Tales_grayscale

Backup? What Backup?

Aug. 8, 2004
Tales From the Front: Programmers Say and Do the Darndest Things
Once upon a time I worked in an ethylene plant that had a ModComp process control computer to run supervisory control of the furnace area. A small company out of Houston wrote the control software. The furnaces ran great, but the ModComp paid for its speed and reliability by being incredibly cryptic and difficult to work on.

Every command was a three-letter acronym with multiple switches and parameters that would all go on the same command line. Online help had not been invented yet. The paper-fed operator terminal, located in the computer room, provided a hard copy of all entries.

 "Our new computer technician was standing behind the command terminal watching him type away late one evening when all of a sudden he said, 'Shoot!!' (or something similar)."

We used 67-MB four-platter removable disks that were more than a foot in diameter, came in its own desk-size cabinet, and sounded like an airplane taking off when it was running. Backups were done on a tape drive as big as me and would copy the entire operating system. All of this equipment ran in its own Halon-protected room.

We were due for a furnace control software update. Arriving late in the day and working well into the evening was the trademark of the ModComp advanced control programmer from Houston. His keyboard action sounded like a finely tuned percussion section during one of their favorite passages. Engineers found it enjoyable to stand around and marvel at him while he was working. After watching for a while, they would just shake their heads and go back to their projects. Our new computer technician was standing behind the command terminal watching him type away late one evening when all of a sudden he said, "Shoot!!" (or something similar). Without even raising his head, he put his hand behind him toward the technician and said, "Where's your backup?" The technician was startled for a second that the programmer was speaking to him at all and, after he recovered, said rather timidly, "Backup? What backup?" The last backup was done several weeks before.

Two too many zeros had been entered in a memory modify command line by the programmer and one hundred times the intended amount of memory was wiped out, taking a big hunk of the operating system in RAM with it. Well, regain furnace control, the programmer and technician spent the next -hour days in the plant rebuilding the real-time operating system.

From that point on, all you had to do was whisper the programmer’s name in the technician’s ear and full set of backups were done immediately. It was kind of nice.

On another occasion, the ModComp was having some disk communication problems and was interrupting advanced furnace control. After looking at the symptoms for a while, the contract ModComp service technician decided it must be something heat related on one of the circuit boards. He broke out the heat gun and Freon bottle. Diligently, he alternately heated and cooled the suspected circuit board, but no faults appeared.

Frustration was starting to set in as the hours passed with no results. The technician was getting sweaty and his comb-over was also starting to show the strain of awkward positions he was forced into for extended periods of time. Finally, he opened both sides of the computer cabinet and sprayed the Freon liberally to do one more cool down of the components before giving up for the day. One halon alarm went off followed quickly by a second as two sensors picked up the Freon and the halon fire protection system in the computer room discharged.

This particular halon system had a helical nozzle and what sounded like a shotgun blast to blow the rupture disk to the high-pressure storage tank. halon flooded the computer room at near supersonic speed both above and below the floor and every particle of dust was immediately suspended in the halon. Note that the door to the computer room opened into the room.

Everyone came running when the system discharged because when it does, it sets off an alarm in the control room. At the door of the computer room was the ModComp service technologist. He was standing with his own feet holding the door securely shut and rattling the door handle like we had him blocked in. A fine layer of dirt covered every square inch of his body and his comb-over was sticking strait out in every direction. We eventually convinced him we were not holding the door shut, he was, and got him to step back, open the door, and come out.

A few important things can be learned from all of this. Always make backup copies of your software. This can protect the enormous investment in time, engineering hours, process know-how, and maintenance that is represented by your running system. If at all possible, keep a copy of the software somewhere safe off site. Do backups often. Do not spray chemicals in halon protected rooms–it is simply too dangerous. Know the exit paths out of any room you are working in. Take your time. Think about what you are doing and what effect it will have on others and the plant.

Tom Peterson Principal Process Engineer

GE Plastics, Mt. Vernon, Ind.