Background
There is a need to change the paradigm of control system cyber security from an intractable network problem to a tractable engineering issue. Solutions are needed that are not focused on network anomalies/attacks but identify control system process/sensor anomalies, or lack thereof. The ransomware scourge makes it even more imperative. The solutions need to be physics-based and not connected to Internet Protocol (IP) networks so the solutions cannot be hacked. That is, the solution doesn’t stop a ransomware attack, rather it is oblivious to the ransomware or IT attack. The solution can be part of a comprehensive contingency planning program that would include the capability for manual operation when automation operation is lost and can be used by any organization that utilizes process sensors that measure pressure, level, flow, temperature, voltage, current, motor speed, etc.
There is no cyber security, authentication, or cyber logging in process sensors. Yet Operational Technology (OT) monitoring systems assume process sensors are uncompromised, authenticated, and correct. As my colleague with forty plus years of field device experience stated, “I have spent years talking to brick walls and brick heads about the lack of security in field devices. Their response is typically that they are air gapped and that everything is safe and secure. Irrational fantasy at best. I am not alone in this quest, but I am definitely in a minority.”
This particular sensor health monitoring technology was developed for sensor and equipment monitoring but can be used as an input for cyber security as sensors are the input to OT networks. The sensor ecosystem, including the sensor networks and asset management systems, have known vulnerabilities as can be seen from the recent LOGIIC (Linking the Oil and Gas Industry to Improve Cybersecurity) process sensor study (Project 12 Safety Instrumentation) and the ISA84.09 process safety/cyber security process sensor study (see https://www.controlglobal.com/blogs/unfettered/ot-network-security-often-does-not-view-control-system-devices-and-the-process-as-their-problem). Process sensors lack authentication and cyber logging as identified in the ISA84.09 review of the ISA 62443-4-2Component Cyber Security Specification where most of the cyber security requirements currently could not be met for a state-of-the art wired safety pressure transmitter. Consequently, a sensor health monitoring system that can add authentication and logging can be an important addition to the sensor ecosystem, equipment monitoring, process safety, and cyber security.
The initial demonstration of the sensor health monitoring system was a plastic tank filled with water with a sensor attached to the tank lid connected to a Programmable Logic Controller (PLC) – a typical “toy” demonstration system. What caught my attention was when the lid was lifted with the sensor attached, the monitoring system detected a change in the electrical characteristics of the sensor yet the liquid level inside the tank had not changed.
Why is this important? Taum Salk is an earthen dam in Missouri with level sensors attached to the earthen wall of the dam. In December 2005, the Taum Salk level sensor attachments broke and the sensors came loose from the wall. The level in the reservoir hadn’t changed, but the sensors’ physical positions changed (level is measured by elevation head, i.e., height). Consequently, the apparent level reading indicated low level. As a result of the low-level indication, the SCADA system at Ameren in St. Louis automatically turned on the water pumps and kept them on until the upper reservoir overfilled and part of the earthen dam collapsed. The sensors didn’t fail, they just provided erroneous readings (see Max 737 plane crashes) that could have been detected by the change in the electrical characteristics of the sensors. The sensing technology indicated changes independent of whether the changes were due to unintentional or malicious events.
The same sensing issues occurred with Stuxnet and would have with Triton. Stuxnet was a “man-in-the-middle” attack where the operator displays were compromised indicating no change in centrifuge operation. However, sensor health monitoring would have identified centrifuges speeding up or slowing down because the electrical characteristics of the motor speed and flow sensors would have changed. The same would be true of the Triton attack in Saudi Arabia. That is, the sensor characteristics cannot be hacked and provide a real-time, ground truth assessment of sensor and associated process performance.
Problem
Ransomware and other IT-originated cyberattacks can affect control system network performance when IT networks are connected to OT networks, compromised Internet-connected devices are connected to the OT network, or critical operational data is kept on the IT network without back-ups (e.g., Colonial Pipeline). Consequently, the US Department of Homeland Security (DHS) provided recommendations to ensure the OT networks are isolated from the IT networks. OT network monitoring technologies can significantly help diagnose the health of the OT networks assuming the OT networks are available. However, an OT network that has not been completely isolated from the IT network or utilizes inadequately secured Internet-connected devices such as IOT devices can be made unavailable. In the water case (see below), it was the OT network that was impacted, yet the isolated sensor monitoring system was unaffected. This demonstrated that the isolated sensor monitoring system would not be affected by ransomware, other IT malware, or unintentional OT network impacts.
Currently, ransomware is not directed at control systems or control system OT networks. Like with the Colonial Pipeline attack, facility operations were shutdown not because the OT network or control systems were directly affected, but “through an abundance of caution” as situational awareness of the process (e,g., water system operations, manufacturing operations, pipeline conditions, etc.) could not be assured. One way to make control system operations less attractive to ransomware attackers is to be able to maintain situational awareness of the process even when the IT or OT networks are impacted. If the process facility (whether a pipeline, water system, manufacturing facility, medical facility, etc.) can continue to maintain situational awareness of the process, they can justify continued operation thus making the ransomware threat less critical. That is, a capability to monitor the "physics" of the process sensors (e.g., pressure, level, flow, temperature, voltage, current, motor speed, position, etc.) provides the situational awareness of the reliability and safety of the process independent of the IP networks. As a result, if the IP networks are unavailable for any reason (whether unintentional or malicious), there would not be a need to shut down the process facility which can make process facilities less of a ransomware target.
Solution
This off-line sensor monitoring technology was installed at a water utility to monitor large critical pumps. Cyber security was a by-product. For this off-line sensor monitoring technology, the raw, unfiltered sensor measurements were monitored and cross-compared to other sensors with known physics-based relationships. In this water case, the off-line sensor monitoring technology was being compared to the Ethernet (IP)-based Windows displays to ensure they were closely tracking. As should have been expected, the off-line monitoring system was “correctly” tracking the Windows HMI, and vice versa. However, during this comparison period, the Windows-based SCADA was lost (could have been for any reason including a malware cyberattack or ransomware attack). However, the off-line sensor monitoring continued providing situational awareness of the process even with the OT network off-line. This continued situational awareness can provide the justification to continue facility operation even if the IT and/or OT networks are compromised. In this case, it is not detecting a malware attack, but the lack of detecting any changes to the process, that provides the justification for continued operations.
Additional benefits
The real-time raw, unfiltered measurements provide additional benefits including authentication of the sensors, equipment diagnostics, and productivity improvements. Specifically,
- As the individual sensors are directly authenticated, the authentication can help with cyber security and supply chain considerations as the multiple sensors “wash out” many supply chain issues. This also may help with the hardware backdoors in the Chinese transformers to validate the sensor data was not spoofed.
- In the water plant case, the off-line sensor monitoring technology identified equipment operating issues not detected by the filtered Windows-based HMI. This also occurred at other facilities including power, chemicals, and building controls where sensor/equipment issues were discovered that were not identified by the Windows-based HMI.
- The sensor health monitoring system can detect sensor drift which is an input to sensor calibration programs and digital twin models. Additionally, the uncertainty in the sensor measurements can be reduced. By monitoring the sensor data in real time, maintenance intervals can be extended as real time sensor monitoring is essentially predictive maintenance.
- Monitoring the sensors provides a “backstop” if the IT networks haven’t been adequately isolated from OT networks.
- As there is no cyber logging in the sensors, the sensor health monitoring can provide input on potential cyber incident reporting to meet the new US Transportation Security Administration (TSA) cyber security requirements. This approach could have detected the process changes at the Olympic Pipeline gasoline pipeline rupture when the sensor displays were set to average values. Without a sensor monitoring program, it is unclear if the TSA requirements to identify potential cyber incidents can be met.
Summary
Ransomware and other IT-originated cyberattacks can affect control systems when IT networks or insecure IOT devices are connected to OT networks. Off-line sensor monitoring technology doesn’t stop a ransomware attack, rather it is oblivious to the ransomware or IT attack. The technology is not affected by IP network attacks or unintentional impacts and can provide real time situational awareness when ransomware or other IT cyberattacks occur providing a justification to continue operation making these systems less attractive to ransomware attackers. This was confirmed at a water facility where the OT system went down and the off-line sensor monitoring system continued working. Additionally, the off-line sensor monitoring technology identified equipment operating issues not detected by the filtered Windows-based HMI in water, power, chemicals, and building controls. This solution can be part of a comprehensive contingency planning program that would include the capability for manual operation when automation operation is lost and can be employed by any organization utilizing process sensors to measure pressure, level, flow, temperature, voltage, etc. If there is an interest in the sensor monitoring technology, please contact me at [email protected]
Joe Weiss
