A vulnerability worse than Log4j (and it can blow up facilities and shut down the grid)
Abstract
More than 3,000 smart instruments in a petrochemical facility were found to have no passwords, even by default. You simply plug in your HART communicator and change whatever you want. These changes can blow up refineries, burst pipelines, release toxic chemicals, take over electric transformers, etc.
Introduction
Those who work in or around network cyber security are aware of the principle of zero trust. It’s generally recognized as a core principle of network security. Meanwhile, process sensors have 100% trust by the control systems the sensors support and the operator displays that use the process sensor input. Not only are the sensors fully trusted, there is no process measurement integrity index that might enable facility operators to feel better about such trust.
Consequently, Weiss’ First Law of control system cyber security: Process measurement integrity = Authorization + Authentication + Accuracy. Process measurement integrity assures that any changes are made by those with permissions (Authorization), the signal comes from the sensor (Authentication), and the sensor measurement accounts for deviations, whether unintentional or malicious (Accuracy). Neither cyber security nor reliability and safety standards address process measurement integrity.
All facilities (industrial, commercial, manufacturing , hospitals, buildings, military, etc.) use process sensors to measure pressure, level, flow, temperature, voltage, current, etc. Process sensors are input to the control systems, safety systems, and control and safety system networks. These devices have been demonstrated to be cyber vulnerable but are not addressed by any existing industry or government cyber security standards.
Many in the Operational Technology (OT) cyber security community believe that the networks are important, but the process sensors are not. Examples include:
- The American Water Works Association (AWWA) cyber security standards do not address process sensors.
- The American Petroleum Institute’s (API’s) cyber security guidelines do not address process sensors.
- The North American Electric Reliability Corporation’s (NERC) Critical Infrastructure Protection (CIP) standards exclude process sensors.
- The U.S. Transportation Security Agency’s (TSA’s) pipeline cyber security requirements, rail, and airport and aviation cyber security requirements do not include process sensors.
- The International Society of Automation (ISA)/International Electrotechnical Commission (IEC) 62443 series of control system cyber security standards currently does not address the unique issues with legacy process sensors or process measurement integrity.
Consequently, Weiss’ Second Law of Control Systems Cyber Security: Garbage in from process sensors = Garbage out from networks, where “garbage” can be unintentional (e.g., sensor drift, technician errors, manufacturing defects, etc.) or malicious (physical or cyber).
Process sensor vulnerability
December 29, 2021, Ankit Suthar published the article “Are your smart instruments secured?” https://www.linkedin.com/pulse/your-smart-instruments-secured-ankit-suthar/?trackingId=7r%2Bf25P7QXKo83zDDsPZkw%3D%3D. In it, Ankit argues: “We have been doing the commissioning of more than 3,000 smart instruments (i.e., Foundation Fieldbus-FF, HART) which includes loop check, simulation, calibration, and datasheet verification, Asset Management System (AMS) configuration for each instrument. (HART - Highway Addressable Remote Transducer- is a hybrid analog and digital industrial automation open protocol. Wired HART communicates over legacy 4–20 mA analog instrumentation current loops using 1200 Baud modems). The project management consultant engineer asked about the password configuration for all the instruments. I started to dig into all the manuals and datasheets of different vendors and found out that there is no password at all in most of the instruments, even by default. You simply plug in your HART communicator and change whatever you want.”
Consider how this design vulnerability defeats a zero-trust model. There is also an additional message that passwords may not be relevant for many process sensors and other control system field devices.
There are no cyber security requirements in any industrial or government cyber security standard addressing process sensor cyber security.Regardless of how well communications are secured, if the process sensors that constitute the ground truth of any physical process are compromised or defective, it will not be possible to have a safe, reliable, or optimized process.
In 2017, ISA99 formed a special working group to determine if legacy control system devices such as process sensors could meet the requirements in ISA/IEC 62443-4-2, the component cyber security specification. Most of the major control system process instrumentation suppliers were members of this special working group. The conclusions were that the legacy field devices could not meet the standard. This led to another special working group, this time within the process safety committee, to evaluate the sensor issues in more detail. The intent of the ISA84.09 (Process Safety/Cyber Security) effort was to determine the relative conformance and applicability of the ISA 62443-4-2 component specification’s individual security requirements to legacy (what is being built today as well those already installed in the field) process sensors. Consequently, in early 2021, the ISA84.09 working group selected a state-of-the-art digital safety pressure transmitter (the same sensors identified by Ankit) ecosystem including the transmitters, host computers, field calibrators, and local sensor networks so as to determine what, if any, compensating measures might be necessary. The results were that 69 of the 138 individual cyber security requirements in ISA 62443-4-2, including fundamental cyber security requirements such as passwords, could not be met. This means that compensating controls are necessary and that alternate standards/recommendations are needed to address the legacy devices that will be in use for the next 10-15 years or longer.
These findings are not new. It is also important to recognize that cyber vulnerabilities are not confined to the protocols but to the sensors themselves. A presentation was made at the 2016 ICS Cyber Security Conference on hacking wired-HART transmitters from three different vendors. A presentation was made at the 2017 ICS Cyber Security Conference on hacking wireless-HART transmitters and digital valves.
CVEs and CVSSs are not applicable
The Common Vulnerabilities and Exposures (CVE) system provides a reference-method for tracking and rating the severity of publicly known information security vulnerabilities and exposures. Log4j has received a top severity rating because of its ease of compromise, The log4j (CVE-2021-44228) vulnerability is extremely bad in that millions of applications use Log4j for logging, and all the attacker needs to do is get the app to log a special string. Log4j has a Common Vulnerability Scoring System (CVSS) score of 10 as it is very widespread, easy to exploit, and allows for a complete takeover of systems or applications.
The lack of cyber security and authentication including the lack of password capabilities in many process sensors are hardware design issues not software vulnerabilities. As such, there needs to be a new category specifically for control system devices. Consequently, Weiss third law of Control System Cyber security is the Commonsense Risk Index (CRI): If process measurement integrity is compromised and sensor(s) can cause, or contribute to, catastrophic failures, the risk is High and must be addressed expeditiously. This may appear to be new but is not. This is how nuclear safety is addressed and how I first found process sensor safety issues.
Potential Impacts – Process industries
Process sensors without basic cyber security capabilities using multiple sensor protocols are used in all applications. There are, for example, more than 40 million devices world-wide using HART. Many of these devices are in safety applications where process sensor compromise could result in catastrophic failures in refineries, chemical plants, water treatment facilities, power plants, ships, etc.
In Ankit’s article, he describes one technician with a field communicator (no cyber security, yet connected to the Internet) who performs the calibration after getting clearances. The instrument is re-ranged and returned to service. Another technician is doing the same from the AMS without going to the field. There are other interfaces to access the instruments’ parameters besides the AMS and Field Communicator.
Ankit goes on to point out that “An attack does not necessarily [come] from outside.” Honeywell’s Sinclair Koelemij May 24, 2020 blog on the OSI PI – ICSA-20-133-02 vulnerability concurs by stating that though the advisory mentions an attack by a “local attacker”, a local attacker can easily be replaced by malware. Consequently, whether the threat is local or remote doesn’t make much difference. As HART-IP has essentially no native cyber security, cyber security capabilities are dependent on the system integrator or end-user. When an attacker gets access to the OSI PI connector, it is possible to inject other commands using HART-IP affecting the field equipment. Commands can result in modifying range, span, engineering units, and/or damping values. Some field devices even allow the low range to be set higher than the high range value. Such a change would effectively reverse the control direction. The situation can be even worse if both the field devices of the Basic Process Control System (BPCS) and the Safety Instrumented System (SIS) are connected to a common system. In this case it becomes possible to launch a simultaneous attack on the BPCS and SIS, potentially crippling both systems at the same time with potential devastating consequences for the production equipment and the safety of personnel.
Potential impacts-Grid
Consider how exploitation of a vulnerable process sensor might affect the reliability, security, and safety of the power grid. A steam turbine at a combined-cycle power plant in Florida experienced a faulty input to a control system due to a failed potential transformer (PT), resulting in oscillations that persisted until the plant operator manually removed the unit from service. The oscillation quickly evolved from a localized forced oscillation (200MW load swings) to an Interconnection-wide oscillation of approximately 0.25 Hz frequency that propagated through the entire Eastern Interconnection causing 50 MW load swings in New England.
As Ankit states: “An attack on a control system could take the form of one or more field instruments being spoofed to induce a shutdown of a piece of equipment.” Current transformers (CT) and voltage transformers (VT/PT) make up the majority of sensors in power system substations. CTs and PTs have no password security. Cyberattacks, including from spoofed signals, could have a detrimental impact on electric substation equipment including transformers. The Chinese have installed hardware backdoors in at least one of the 300 Chinese-made large electric transformers in the US grid. These backdoors bypass all cyber security. If the process sensors are compromised or sensor signals are spoofed as Ankit stated and sent to the transformer equipment via the backdoor, the transformer integrity is at risk. Compromising sensor configurations can lead to failures of controllers, voltage regulators (load tap changers), and other control system devices with no apparent indication putting the grid at risk.
Spoofed values can cause the control systems to inappropriately operate their associate protection systems. Upon initial investigation it would be extremely unlikely to determine the operation is caused by a cyberattack. However, the new digital relays and control systems record sequence of events (SOE) and store the records in memory. A protection engineer can access the SOE records and play back the individual control system values in the registers and by watching the voltage and current phasors and their symmetrical components as they occurred – this is physical not cyber forensics. With appropriate training, this more in-depth examination allows protection engineers to notice unusual anomalies that could point to the spoofing of CT and PT inputs.
Potential Impacts – Building Controls
In September 2021, the Oak Ridge National Laboratory (ORNL), Pacific Northwest National Laboratory (PNNL), and National Renewable Energy Laboratory (NREL) issued a report on process sensor issues in buildings. The report concluded that cyber security threats are increasing, and that sensor data delivery could be hacked as a result. A typical situation could include sensor data being modified by hackers and sent to the control loops, resulting in extreme control actions. To the best of the ORNL, PNNL, and NREL authors’ knowledge, no such study had examined this challenge.
Adversaries are aware
Unfriendly nations are aware of these vulnerabilities. A Russian security researcher gave a remote presentation from Moscow at the 2016 ICS Cyber Security Conference on hacking the wired HART protocol through cyber vulnerabilities in the AMS (consider what Ankit said). In October 2017, I received a “Like” on my Linked-In account from a representative from Iran on my Defcon presentation that there is no cyber security in process sensors. As mentioned, the hardware backdoors in large Chinese-made transformers enable spoofed process sensor signals to take control of the transformers.
Summary
OT cyber security’s focus on networks is necessary but it’s not sufficient. In fact, it is dangerously insufficient. Control system cyber security needs to focus on process sensors (and other control system field devices) which are critical for safety, reliability, maintenance, and cyber security, particularly as these devices have no cyber security, authentication, or cyber logging. That includes many process sensors with no password capability. The impact is not a potential compromise of networks such as the Log4j vulnerability but the ability to directly manipulate equipment causing physical damage and compromising personnel safety. Cyber security of sensors continuously needs to be examined and improved. However, any cyber improvements cannot come at the expense of control system reliability or safety. These issues currently are not being addressed by industry or government standards nor by government regulation. Where is the appropriate priority and urgency?
Joe Weiss
