Sensors and Sensibility – Dragos (and other OT experts) lack expertise on process sensors

Let me begin with a reminder about destructive control system attacks. The scenarios for the three major control system cyber attacks meant to damage facilities – Aurora, Stuxnet, and Triton – were known ahead of time, but the general control system populace didn’t believe they could happen. You could say the same thing for the 9/11 airplane attacks: even though they might have been recognized as theoretical possibilities, and even appeared in fiction, they weren’t at the forefront of people’s minds. In IT security, known vulnerabilities, regardless of how subtle, are recognized as problems that need to be addressed. Spectre and Meltdown were subtle but important vulnerabilities in Intel chips that had the industry scrambling. Why, we might ask, is that attention not being devoted to cyber gaps in control system Level 0,1 devices? I believe it is because of the unique aspects of these devices that are not well understood by the OT network monitoring community. Another key misunderstanding is that network security does not equal process integrity. The vast majority of the time, sensor/process upsets will not be cyber-related but it is still critical to identify them. From a cyber perspective, a sophisticated attacker will try to make a cyber attack look like a malfunction.  Consequently, engineers need to be part of the cyber security process as well as trained to recognize when process upsets could potentially be cyber-related.

I mention this as an introduction of an argument Joe Slowik from Dragos made recently in a blog post he called “Sensors and Sensibility” (https://pylos.co/2019/11/08/sensors-and-sensibility/). Specific to Joe’s post, “the basic argument can be distilled as follows: all security solutions (and for ICS operations, effective and integrity-assured processes) are reliant upon receiving accurate information from observed systems. If operations or security cannot guarantee or verify the integrity of sensor output (or the sensors themselves), then one cannot guarantee the fundamental integrity of the network or process itself. Joe goes on to state that the “dilemma of sensor security seems to be not only inaccurate in implication, but false in characterizing the problem.” As his title suggests, he’s offering a critique of something like a foundationalist epistemology: if you can’t guarantee that the sense data you’re working from are “true,” or “right,” then you can’t really know anything at all. His suggestion is that a fixation on sensor data in industrial control system security is like a fixation on sense data in the general theory of knowledge: if you insist on them, you’re never going to get anywhere. Besides, while there are sense data, and while they’re important, knowledge isn’t simply built up from an accumulation of sensory inputs.

I think Joe’s basic point is this: “Given practicalities, difficulties, and adversary economization of effort, focusing on sensor-level security or sensor-level manipulation is not only irrelevant given the existing environment, but such an emphasis actively draws resources from efforts more likely to result in improved security. For example, devoting resources and capability to building out a duplicated, ‘secure’ sensor network on top of existing devices means that one has likely invested significant resources which draws away from securing information processing and display. As a result, while one can assure the integrity of fundamental data, its subsequent transport, processing, and display is still subject to manipulation or subversion.” He’s right. Data can be corrupted in transport, processing, and display. But it can also be corrupted at the source, and if that happens, no improvements in security farther up the line will fix the problem. One of Joe’s contentions is that process sensor cyber security is not that viable because you would have to traverse IT and then OT network security. However, that doesn’t need to be the case. In fact, there are means to get to the process sensors while bypassing all OT network security. Moreover, you do not need to replicate all process sensors, just a subset of diverse sensors related by physics.

I think we can agree that process sensor inaccuracies or failures can be reliability and safety problems, whether they come from unintentional or malicious causes. There have been many catastrophic failures from process sensor failures. These include the Texas City refinery explosion, the Buncefield tank farm explosion, and the Taum Salk dam collapse. Consequently, long before there was such a thing as cyber security, people performed detailed analyses of process sensors and equipment forensics including for nuclear regulatory reasons. What I found were significant sensor failure modes that hadn’t been previously considered (among other incidents, they affected the Three Mile Island core melt) as well as undetectable equipment diagnostic failures. At the time, the only malicious causes were assumed to be from physical sabotage.

According to Joe, “Essentially, sensors merely represent one aspect of an overall chain in information gathering and exchange, starting with translating physical observations into electronic data and ending with the processing of such data by some supervising device or human being. Given this whole-of-information approach to sensors as an interlocking system rather than simply being a discrete device, there are in fact multiple possibilities for manipulation and subversion – many of which are far easier (and more probable) to execute than the nightmare Layer 0 device compromise.”  In this he’s correct. However, process sensors are their own ecosystems with their own insecure protocols, built-in back doors, required Internet access, and insecure calibrators to name just a few insecure items in this ecosystem. To date, there are no process sensors that have completed any cyber security certification testing.

Furthermore, because I was skeptical of the long-term accuracy of the process sensors, one of the projects I myself performed was to evaluate process sensors in actual (fossil-fuel) power plant operation for sensor drift – and they did within 30 days!  The only way to monitor for process sensor and equipment health is to monitor the raw sensor signals in real time, which is what was done before Ethernet networks came along. The irony is we are now less reliable and safe with OT network monitoring than we were 30 years ago when we were monitoring the raw process sensor fluctuations.

Joe adds, “Ultimately, security against even highly concerning events such as sensor manipulation or supply chain interdiction still relies on an ability to observe, monitor, and detect an adversary attempting to interact with or otherwise inject into the compromised environment to make this subversion valuable and actionable. As a result, defenders still have a number of opportunities to identify and defeat such techniques. By focusing not on the supposed sophistication or power of low-level attacks or injection but rather on the multiple, necessary dependencies to make such items valuable, controllable, or actionable, defenders remain in a position to control the engagement and defeat adversaries – so long as they are collecting relevant data, correlating it with events, and responding in a reasonable amount of time.” This is the issue. As far as I can see, the only way to monitor for sensor health and assure that the sensors haven’t already been compromised is to monitor the raw sensor signals in real time, as was done before Ethernet networks came along. However, because of the filtering of the serial-to-Ethernet convertors, it is not possible to accurately monitor sensor health at the Ethernet packet level: it must be done at the raw signal level before the filtering occurs. This can easily be overlooked if one is too quick to apply an IT template to what amounts to an engineering safety problem.

According to Joe, “Sensors serve as the fundamental observation system through which physical process or direct observable states are recorded and then translated into forms that are actionable. Such action can be taken by automated systems (e.g., a PLC operating within pre-programmed ladder logic) or a human operator, with the resulting data format (and speed of transmission and processing) dependent on the actor. Furthermore, this link is not direct, but relies upon multiple interconnecting pieces: the physical process or action, which is observed by the sensor, which then transmits translated process data via a data link to a receiver or processor entity, and then final delivery to and action by a supervising machine or human being.” Sensors are actually their own ecosystems with the physically sensor, protocols, calibrators, etc. Moreover, sensors are continuously monitoring. By the time an operator would detect a sensor problem, it can be too late for any type of safety mitigation.

Because of my process sensor background, I care about the importance of sensor data. But I’m not the only one who recognizes the need to monitor the raw process sensor signals. Work done at the US Air Force Institute of Technology demonstrated the ability to hack multiple vendors’ wired and wireless process sensors and actuators. The Russians also demonstrated hacking wired HART process sensors in their ICSCorsair demonstrations. Moreover, a joint ISA99 (cyber security) and ISA84 (process safety) committee on safety and security is explicitly addressing process sensors and other level 0,1 devices because of a belief that these devices have not been previously adequately addressed.

Joe stated: “While a true sensor or (unobserved, undetected) process alteration remain the most concerning attack vectors, they also are the most difficult, expensive, and hard to scale”.  I issued a blog on counterfeit transmitters (https://www.controlglobal.com/blogs/unfettered/the-ultimate-control-system-cyber-security-nightmare-using-process-transmitters-as-trojan-horses/). This type of attack makes scaling feasible as does hacking sensor manufacturing lines as sensor manufacturers supply sensors to numerous verticals globally. The blog was provided to representatives from the Naval War College and many attendees prior to the July 25-26, 2019 Cyber War Games at the Naval War College. On Thursday evening at the Conference, Joe and I had discussions with the senior intelligence official acting as the President of the United States (POTUS) for the War Games where I had a chance to explain the sensor issue. On Friday, the Red Team brought up the use of counterfeit SCADA parts. When this occurred, POTUS declared a Grid Security Emergency which was not done for any of the network cyber attack scenarios. Hacking process sensors is that dangerous which is why several major control system vendors are mobilizing to address this gap. Mitsubishi announced that it’s developed an algorithm which can detect malicious attacks against equipment sensors. GE has developed Digital Ghost that addresses sensor issues. Several boutique vendors are also working in this space.

According to Joe “ looking at the history of attacks (especially in ICS environments) and continued trends, all external-party attacks and modifications come at “higher levels” of the overall sensor stack, typically focusing on either processing of data in software or display of data to the receiver (human or machine). In many respects, this is analogous to the concept of signal jamming in electronic warfare.” I have a database of more than 1,200 actual control system cyber incidents of which a significant number were process sensor-related. Without understanding process sensor operations, there are no cyber forensics at this layer and little training for engineers to recognize potential cyber indications.  I would like to mention my book is entitled Protecting Industrial Control Systems from Electronic Threats as Electromagnetic Interference (EMI) and Radio Frequency Interference (RFI) have impacted control systems.

From first-hand knowledge, the Russians, Chinese, and Iranians are all aware of the gap in process sensor cyber security.  Cyber attacking process sensors is not hypothetical as they have been hacked. The safety and productivity concerns are that sensor configurations can be changed without detection and it can affect multiple processes and multiple facilities (think about the counterfeit transmitter issue). In order for Stuxnet and Triton to be successful, they required the Windows-based sensor displays and alarms be compromised. Without an independent sensor network not tied to the Windows displays you no longer have safety! In fact as Dave Bennett from Jacobs stated in his presentation to the 2018 Honeywell Users Group, (https://www.honeywellprocess.com/library/news-and-events/presentations/hug-america-2018-level-0-1-the-back-door-for-cyber-security-events.pdf), if level 0/1 is not secure, it’s not safe.

For those that haven’t heard it, I recommend you listen to the podcast -   https://www.controlglobal.com/blogs/unfettered/podcast-control-systems-cybersecurity-a-grim-gap-a-conversation-with-joe-weiss/.

OT network monitoring companies have a hole – plug it and everyone wins. Keep fighting and everyone loses. Hopefully, the good guys wake up before the bad guys attack us where there is neither protection nor monitoring. I believe that credit rating agencies and the insurance companies will find this potentially catastrophic gap very disconcerting. So thanks to Joe Slowik for his contribution to the conversation, but let’s continue to think about the role process sensors play in control and safety engineering as well as cyber security.

Joe Weiss