One of the main reasons for installing a control system is to accomplish “real time control.” However, what happens if the communications between the various nodes on your system are not operating in real time? And in an even worse situation what if you do not know that the communications have failed?
That is exactly what happened to a team mate on my hockey team about 10 years ago in the middle of winter. When he showed up for our next game I asked him where he had been a couple night earlier to which the answer was, “I was thawing out about a mile of frozen tailings (combination of up to 40% sand and water) line.” After the investigation was complete it was determined that what had happened is that the protocol being used to communicate between the PLC’s that controlled all the pumps along the line did not have a way of confirming communications between nodes. The communications failed so the pump station that lost communications failed to a safe state which was insufficient to keep the slurry from freezing at the winter temperatures of approximately -30 C.
Needlessly to say, when they started back up again they had watchdog timers between all the nodes to be sure all the nodes stayed connected.
The other lesson learned here is that when conducting a review of your program or participating in a HAZOP (HAZard & OPerability study) you need to keep this kind of failure in mind. Just because something is not “in the pipe” does not mean it will not affect your process so as the automation specialist in these meetings it is YOUR responsibility to be sure they are addressed.
Remember, if you can’t talk with it you can’t be controlling it either.
