OT control system field device cyber security issues are different from those that affect Internet Protocol (IP) networks. These differences need to be understood by all organizations making OT/ICS cyber security policy or recommendations. Too often, government and industry guidance given for OT cyber security is IP network-focused and assumes the guidance will apply to all of OT including legacy ICS field devices and control systems. From a cyber security perspective, legacy control systems are not just older pneumatics and 4-20 milli-amp analog sensors but also “modern” digital devices. Essentially, all control system field devices currently being provided are legacy systems with little to no cyber security. Moreover, the lack of engineering senior management participation in cyber security policy development as documented in my article in the May/June 2020 issue of PE Magazine “Attention Policymakers: Cybersecurity is more than an IT issue” continues unabated.
The US Government Accountability Office (GAO) report GAO-19-332 stated:
“Compounding the risk associated with the increased attack surface, many legacy industrial control systems were not designed with cybersecurity protections because they were not intended to be connected to networks, such as the internet. For example, many legacy devices are not able to authenticate commands to ensure that they have been sent from a valid user and may not be capable of running modern encryption protocols. In addition, some legacy devices do not have the capability to log commands sent to the devices, making it more difficult to detect malicious activity. Additionally, even in the case of more modern devices, the safety and efficiency goals of the grid and the supporting industrial control systems can conflict with the goal of security in the design and operation of industrial control systems. According to an Idaho National Laboratory analysis, grid owners and operators may not always be able to identify industrial control system vulnerabilities in a timely manner. Vulnerability scanning is often used in IT systems to validate proper system configuration and to identify any vulnerabilities that may be present. However, conventional IT vulnerability scanning can disable or shut down energy delivery systems, and testing may not always detect vulnerabilities deep within industrial control system software. Further, even if owners and operators are able to identify industrial control system cybersecurity vulnerabilities, they may not be able to address those vulnerabilities in a timely manner because certain industrial control system devices may have high availability requirements to support grid operations. These devices typically need to be taken offline to apply patches to fix cybersecurity vulnerabilities. In addition, grid owners and operators need to rigorously test the patches before applying them. Security patches are typically tested by vendors, but they can degrade or alter the functionality of industrial control systems, which can have serious consequences for grid operations.”
There have been many documented cases where applying IP network mitigations have caused very significant problems to control systems and control system field devices as mentioned in the GAO report.
Examples include:
- IT patches have compromised control systems even causing safety issues. This included a patch for a turbine control system that was not coordinated by OT (square peg) and engineering (round hole) even though the patch was tested by the networking organization before it was sent to the customer. The untested system interaction led to the loss of view of the turbine control workstation and the need to shut the turbine. However, the unintended system interactions from the “untested” patch prevented the engineer from being able to shut the turbine from the engineer’s workstation – a major safety issue.
- IT penetration testing on control system networks have caused shutdowns or damage to control systems and control system communications. In one case, a utility’s IT security group (square peg) was scanning data center assets using IP network scanning software and then expanded the scanning into large electric substations (round hole). The security group had no previous experience with scanning substations. Following the scans, the relays showed trouble, but SCADA was unaware of the problems. The port scanning of this new tool caused the real time protocol operation of the relays to stop and suspend operation at the CPU (two different relay suppliers) and left the DNP/non-real time operations alone - the worst possible circumstance. In order to clear the trouble, each relay had to be cut out and rebooted, to restore operation. Several hundred relays were affected. All the devices in each substation were affected at the same time in every case. Without knowing that a security scan was initiated, it looked like a distributed denial-of-service (DDOS) attack resulting in equipment malfunction. A grid upset with the high voltage relays unavailable could have caused a major region-wide outage damaging many large transformers and customer equipment. In another case, IT (square peg) did a penetration test and caused a denial-of-service to 6,000 control system devices (round hole). It took a total of 15 days to reset each control system device.
- Network mapping tools can impact control system field devices. In another case, variable speed drives were connected to the network. The network mapping tools (square peg) caused a buffer overflow that caused a hard failure of the drive requiring a power down and hardware damage requiring replacement of the configuration modules (round hole).
- Applying Anti-Virus software (square peg) to many legacy distributed control systems-DCSs (round hole) have caused denial-of-service conditions.
- System hardening applies to Microsoft Windows-based equipment (square peg). However, most legacy control system field devices (round hole) don’t use Windows.
These, and many other cases, demonstrate that IT security technology (square peg) that works well in a constrained IT environment may not work in an unconstrained OT/control system environment (round hole), particularly with legacy control system devices and communication protocols. There are network security tools that have been specifically designed for use in control system environments that do work well. However, even those need to be tested before use with older legacy control systems.
I continue to be very concerned that both private sector and public sector policy making organizations (square peg) simply don't have the control system cyber security technical depth to be making decisions about cyber security of control systems (round hole). This is not just a US problem. Recently, for example, Germany’s cyber security policy-making organization (square peg) conducted table-top exercises focused on power generation without any input from the power generation engineering organizations (round hole). Control system cyber security training that includes unique issues like process sensors, system interactions, and common cause failures are needed to educate both the workforce and policy makers.
These "square peg-networking" vs "round hole-engineering" issues will be the subject of my presentation October 26th in Minneapolis (https://www.cybersecuritysummit.org/).
Joe Weiss
