Backup? What Backup?

Once upon a time I worked in an ethylene plant that had a ModComp process control computer to run supervisory control of the furnace area. A small company out of Houston wrote the control software. The furnaces ran great, but the ModComp paid for its speed and reliability by being incredibly cryptic and difficult to work on.

Every command was a three-letter acronym with multiple switches and parameters that would all go on the same command line. Online help had not been invented yet. The paper-fed operator terminal, located in the computer room, provided a hard copy of all entries.

We used 67-MB four-platter removable disks that were more than a foot in diameter, came in its own desk-size cabinet, and sounded like an airplane taking off when it was running. Backups were done on a tape drive as big as me and would copy the entire operating system. All of this equipment ran in its own Halon-protected room.

We were due for a furnace control software update. Arriving late in the day and working well into the evening was the trademark of the ModComp advanced control programmer from Houston. His keyboard action sounded like a finely tuned percussion section during one of their favorite passages. Engineers found it enjoyable to stand around and marvel at him while he was working. After watching for a while, they would just shake their heads and go back to their projects. Our new computer technician was standing behind the command terminal watching him type away late one evening when all of a sudden he said, "Shoot!!" (or something similar). Without even raising his head, he put his hand behind him toward the technician and said, "Where's your backup?" The technician was startled for a second that the programmer was speaking to him at all and, after he recovered, said rather timidly, "Backup? What backup?" The last backup was done several weeks before.

Two too many zeros had been entered in a memory modify command line by the programmer and one hundred times the intended amount of memory was wiped out, taking a big hunk of the operating system in RAM with it. Well, regain furnace control, the programmer and technician spent the next three sixteen-hour days in the plant rebuilding the real-time operating system.

From that point on, all you had to do was whisper the programmer’s name in the technician’s ear and full set of backups were done immediately. It was kind of nice.

On another occasion, the ModComp was having some disk communication problems and was interrupting advanced furnace control. After looking at the symptoms for a while, the contract ModComp service technician decided it must be something heat related on one of the circuit boards. He broke out the heat gun and Freon bottle. Diligently, he alternately heated and cooled the suspected circuit board, but no faults appeared.

Frustration was starting to set in as the hours passed with no results. The technician was getting sweaty and his comb-over was also starting to show the strain of awkward positions he was forced into for extended periods of time. Finally, he opened both sides of the computer cabinet and sprayed the Freon liberally to do one more cool down of the components before giving up for the day. One halon alarm went off followed quickly by a second as two sensors picked up the Freon and the halon fire protection system in the computer room discharged.