A Grim Gap: Cybersecurity of Level 1 Field Devices and lack of appropriate OT Expertise

Feb. 6, 2019
Getting networking and technical organizations to work together is critical, and one would think should be easy. Unfortunately “doughnut diplomacy” hasn’t worked and the gap between Engineering and IT/OT continues to exist and may even be growing. Monitoring the actual raw sensor data (engineering) along with the SANS recommendations (networks) would help close the cyber security gap, as well as improve reliability and safety.

The February 2019 issue of Power magazine had the following article: “A Grim Gap: Cybersecurity of Level 1 Field Devices” (https://www.powermag.com/a-grim-gap-cybersecurity-of-level-1-field-devices). The article is essentially an interview with me on the lack of cyber security in process sensors, actuators, and drives – Level 1 field devices. The interview also discusses the term “Operational Technology” (OT) and what it does and does not cover.

Part of the reason for the interview was to address the white papers being developed by Dragos and GE. According to one of the authors, their intent was to educate the engineers about cyber security, which is a good thing. However, what was, and continues to be missing, is education for security people about engineering issues. I hope to contribute to that discussion with this blog as not everything about process engineering is obvious. The control system cyber security focus continues to be on the networks not the process which is backwards as the process is what needs to be the focus of the protection.

An important aspect of the article is the distinction between IT/OT and Engineering. The lack of cyber security in industrial process sensors occurred because of the technology and cultural gaps between IT/OT and Engineering. Most control and automation engineers, substation engineers, instrumentation technicians, relay technicians, etc. do not consider themselves to be OT. The control system software and logic are engineering software. For example, Proportional-Integral-Derivative (PID) control system logic would not be considered OT, but the network infrastructure and monitoring of the computers containing the PID logic would be OT. Similarly, process sensors would not be OT but the sensor data after conversion to Ethernet packets would be OT. As such, the focus of OT cyber security has been on the network and workstation layers ASSUMING the sensors are secure (uncompromised), accurate, and authenticated which may not be the case. Another culture issue is the siloing of multiple functional areas affected by process sensors including cyber security, safety, alarm management, and device management organizations. Unfortunately, these silos are alive and well in many organizations including the Nuclear Regulatory Commission (NRC), IEEE, and many end-users and vendors. However, this doesn’t have to continue. A working group consisting of members from ISA84 (Process Safety), ISA99 (ICS Cyber Security), and ISA108 (Device Management) are currently working on the 3rd edition of ISA TR84.00.09, Cybersecurity Related to the Safety Lifecycle. This group has recognized that control system cybersecurity encompasses the entire system from the overlap with IT, boundary devices, networking devices, servers, Basic Process Control System (BPCS) controllers, Safety Integrated System (SIS) controllers, HMIs, AND the field devices. As such, this represents a broad set of disciplines, including, but not limited to engineering such as process control, process safety, instrumentation, electrical engineering, as well as operations, maintenance and IT aspects of control/electrical systems/networks. (One wonders why the nuclear industry isn’t participating since it has similar safety needs. In fact, I am scheduled to give a presentation February 14 at the American Nuclear Society’s 11th Nuclear Plant Instrumentation, Control, and Human-Machine Interface Technologies Conference on “The Hole in Nuclear Plant Cyber Security – Insecure Process Sensors” which addresses many of the issues being addressed in the ISA joint working group.)

Process sensors are used in all commercial, industrial, and manufacturing processes. Every process depends on what you measure, and the assumed accuracy of those measurements. Erroneous sensor measurements can have disastrous impacts as some recent events amply demonstrate. I had a discussion with a representative from FAA on the LionAir Boeing 737 plane crash that killed 189 because sensors played a key role (a sensor issue also occurred with the Air France AirBus crash). Per our discussion, the FAA representative acknowledged there wasn’t enough concern about the role of sensors in aircraft cyber security. January 25, 2019, a tailings dam at the Vale Brumadinho mine in Brazil failed with more than a hundred deaths. Vale says that instruments used to measure dam pressure had not detected any problems and that alarms did not actuate. I also had a discussion with MIT Professor John Thomas about STPA (System-Theoretic Process Analysis). This is a relatively new hazard analysis technique based on an extended model of accident causation. In addition to component failures, STPA assumes that accidents can also be caused by unsafe interactions of system components, none of which may have failed. This approach is intended to be demonstrated with nuclear plants. To date, STPA does not address sensor validity by monitoring at the raw signal layer.

Last year SANS provided a demonstration on the impact of PLC data table misinformation to compromise processes without the operator being aware. There is a similarity to Stuxnet in compromising the PLC to produce compromised operation without the operator/HMI being aware. The cases demonstrate that physical processes can be compromised by remotely changing sensor specifications such as span, range, and measurement type (Fahrenheit vs Celsius). As an example, NASA lost a $125 million Mars orbiter because a Lockheed Martin engineering team used English units of measurement while the agency's team used the more conventional metric system for a key spacecraft operation. A gap in the SANS presentation and test cases was the assumption that the actual sensor input data was valid (e.g., uncompromised, authenticated, and correct). However, sensors can be unintentionally or maliciously compromised before the data becomes Ethernet packets which requires monitoring of the sensors at the lowest level of the Purdue Reference Model. Without this monitoring, sensor issues, whether cyber or unintentional, would not be identified as being anomalous.

A very important finding of the SANS study was that non-routable protocols, (communications protocols that contain only a device address and not a network address and do not incorporate an addressing scheme for sending data from one network to another) can be reached and compromised. This is a concern because the electric industry’s cyber security standards - the North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection Standards (CIPS) - consider all non-routable protocols to be out-of-scope (not addressed). Process sensors are also out-of-scope as they are inside the Electronic Security Perimeter. These requirements need to be reassessed because without addressing sensors and non-routable protocols, it is not possible to cyber secure a substation or power plant!

Getting networking and technical organizations to work together is critical, and one would think should be easy. Unfortunately “doughnut diplomacy” hasn’t worked and the gap between Engineering and IT/OT continues to exist and may even be growing. The US National Academy of Engineering is concerned about these issues and has asked me to write an article about this gap. Monitoring the actual raw sensor data (engineering) along with the SANS recommendations (networks) would help close the cyber security gap, as well as improve reliability and safety.

Joe Weiss