Process sensor cyber issues have contributed to catastrophic events

If you’re a doctor and you can’t trust your temperature and blood pressure readings, how can you give a diagnosis?

A major distinction between the cyber security organization and facility Operations is that security is focused on Information Assurance (what is going on in the network), while Operations is concerned with Mission Assurance (what is going on with the process) --- and they are NOT the same. Currently, the security organization, be it IT or OT, takes the output of the serial-to-Ethernet converter to use as the baseline for network anomaly detection. However, process sensors (e.g., pressure, level, flow, temperature, voltage, current, etc.) have neither authentication nor security. Consequently, there have been unintentional sensor impacts from issues like sensor drift or plugging of instrument lines that cannot be detected from network monitoring or malicious attacks that change sensor configurations that also would not be detected by network anomaly detection. There are “network” monitoring approaches where network anomaly detection is coupled with review of Historian data, etc to get a more comprehensive view of the “process”. However, if the sensors are “compromised” before the serial-to-Ethernet converter, the packet capture, Historian review, etc will still be dealing with compromised data.

The lack of understanding of this serious security deficiency was evident by the reaction of the attendees at the CyberEndeavor 2017 Conference in June in Pittsburgh and the ISA Power Industry (POWID) Conference in Cleveland. However, the Cyber War Games at the Naval War College in July did recognize this issue as one of the scenarios was erroneous sensor data prevented a control valve from closing leading to a sewage spill that could not be contained.

There have been many process sensor-related cyber incidents, some with catastrophic impacts including damage and deaths. 

  • Dam failure when sensors pulled away from the wall providing misleading readings that resulted in pumps overfilling the reservoir.
  • A sensor on a valve malfunctioned and resulted in the release of 10 million gallons of untreated wastewater.
  • PLC automatically opened the reject bin chute door based on faulty sensor data dropping 10 tons of material on the truck cab resulting in a fatal injury.
  • PLC controlled the functioning of the batch system based on sensors that monitored material flow. Level sensor falsely sensed that slurry was in the reactor, resulted in the PLC sending a command to open the steam valve fatally burning the employee.

A recent case illustrates the issue. The pressure transmitter in a critical application in a large power plant failed tripping the plant. The network monitoring was not able to detect the incipient failure of the transmitter because the packets couldn’t detect the change in the sensor characteristics until the sensor failed. However, with real-time monitoring of the sensor’s electrical characteristics, the degradation of the signal would have been evident long before the sensor failed.

There is technology available to do this type of monitoring – process anomaly detection. Coupling the network anomaly detection to the process anomaly detection can provide a basis for making an informed decision about continued operation when malware has been detected. That is, do I care if malware has been detected on the network if the process is not affected?

This technology is already in use at several international facilities. The data from the field has provided several remarkable findings:

-        During a portion of the testing phase, SCADA was unavailable. However, the sensor monitoring was unaffected by the SCADA outage and continued to provide a view of the process. This technology not only provides resilience form SCADA attacks but also provides resilience by providing sensing input when SCADA is lost. As an example, SCADA unavailability was identified by NTSB as the proximate cause of the Olympic Pipeline Company gasoline pipeline rupture. Part of the SCADA failure scenario was sensor readings were set to average values when SCADA was lost so the operator could not readily detect a SCADA failure. Identification of sensor issues could have provided operators clues that SCADA was not operable.

-        The sensor monitoring was, at times, more granular than the SCADA system and was able to detect anomalies SCADA did not identify. This type of monitoring would have identified the sensor issues associated with the Taum Salk dam failure and the power plant trip.

These findings have great significance for reliability and safety as well as security. I will be discussing process sensor issues at Defcon in Las Vegas Saturday July 29th.

Joe Weiss