Over the next 100 days, DOE’s Office of Cybersecurity, Energy Security, and Emergency Response (CESER) will continue to advance technologies and systems that will provide cyber visibility, detection, and response capabilities for industrial control systems (ICSs) of electric utilities. The initiative modernizes cybersecurity defenses and encourages owners and operators to implement measures or technology that enhance their detection, mitigation, and forensic capabilities.
- Includes concrete milestones over the next 100 days for owners and operators to identify and deploy technologies and systems that enable near real time situational awareness and response capabilities in critical industrial control system (ICS) and operational technology (OT) networks;
- Reinforces and enhances the cybersecurity posture of critical infrastructure information technology (IT) networks; and
- Includes a voluntary industry effort to deploy technologies to increase visibility of threats in ICS and OT systems.
Reality Check
I am thrilled that the White House and the Secretary of Energy feel that control systems are critical enough to “immediately” address. Doing so is long overdue. Even more than twenty years late (I helped start the control system cyber security program for the electric utilities in 2000), the attention is welcome. The specific recommendations are not wrong, just not complete and may lead to a false sense of security. Moreover, if the 100-day plan is meant to “keep lights on”, why are IT networks included? IT network security issues such as the hacks of SolarWinds, Microsoft exchange servers, the Sophos web appliances, the Fortinet VPNs, and the Pulse Secure VPN vulnerabilities need to be addressed independent of OT networks issues.
As Mark Prince who was at Entergy for more than 40 years mentioned, “in most companies it would take close to 100 days just to complete the Supply Chain cycles and get products delivered not to mention designs and deployments. If software, service or hardware are being procured for Medium or High CIP impact system the Supply Chain piece is even slower than ever since CIP-013 came out in 2020. One hundred days sounds great but it’s not feasible.” Mark had a great term – this is “unobtainium”.
The existing electric industry approach to cyber secure large power plants and the transmission grid is to meet the NERC Critical Infrastructure Protection (CIP) standards. The NERC CIPs are compliance-based which can be seen from the scope which excludes control system field devices, non-routable networks, electric distribution, etc. The focus all too often is compliance not “keeping lights on”.
The control systems and control system field devices used in the electric industry are the same as used in other industries and manufacturing – solve for one industry and you solve for all. Even devices viewed by many as unique to the electric industry such as protective relays are used throughout commercial, industrial, and manufacturing facilities to protect large electrical equipment. Trusted real time situational awareness is necessary to provide cyber security as well as reliability for OT networks. These monitoring systems are very powerful and have been available for several years.
What the plan is missing
OT network cyber threat prevention
In addition to cyber visibility, detection, and response capabilities, there are various forms of prevention that should be applied including white-listing, encryption, firewalls, data diodes, Virtual Private Networks (VPNs), patch management, etc.
OT networks are generally flat and utilize insecure control system field devices. Operations require OT networks to send information from high security zones (e.g., real time data) to lower security zones (e.g., archival databases). Consequently, they require segmentation and unidirectional control of communications which are core tenets of the ISA62443 series of control system cyber security standards. As an example of this need, SolarWinds monitors network switches including OT network switches. Without adequately segmenting IT networks from OT networks, SolarWinds can impact building controls in electric utility control rooms and control centers. However, SolarWinds impacts on control systems have not been addressed by DOE or DHS.
There are many cases where critical control system workstations/servers cannot be patched for significant periods of time because of facility operational needs. There are technologies that can provide a level of security to allow the facility to continue operation with unpatched serves with critical vulnerabilities until the necessary patching can be made. As IT patch management policies can, and have, impacted OT networks and even ICS equipment, International Society of Automation (ISA) 62443-2-3 was developed to address patch management for control systems.
There are hardware backdoors for maintenance purposes that are part of the design of many ICS devices. As such, these backdoors cannot be bypassed. Consequently, some form of prevention is necessary.
Planning for securing OT networks can be done within the next 100 days.
ICS field devices
The plan makes no provision for situational awareness for the ICS field devices such as process sensors, actuators, and drives. These devices have no cyber security, authentication, or cyber log capabilities. Yet, these devices will be in use for the next 10-15 years in all industries and are the input to the OT networks which affects the integrity of the situational awareness of the OT network monitoring systems.
The oil/gas industry has taken an interest in cyber security of process sensors. ISA formed a special working group addressing the intersection of process safety and cyber security. To date, there has been no participation from the electric or nuclear sectors in the ISA84.09 process. The ISA84.09 review of the ISA 62443-4-2 Component Cyber Security Specification identified that most of the cyber security requirements currently could not be met for a state-of-the art wired safety pressure transmitter. This means rip and replace doesn’t work. Along with a colleague, I gave a presentation on process sensor cyber security March 30, 2021 at the Texas A&M Instrumentation and Automation Symposium but that was primarily for the oil/gas and chemical sectors. The LOGIIC program (for oil and gas) just issued a report on the cyber vulnerability of process sensors that validates the ISA84.09 work (and vice versa).
To date, the utility and nuclear sectors have been silent on cyber security of process sensors. One reason could be that the NERC CIP approach requires a six-sided enclosure for every asset considered a critical asset. Enclosing a server in a six-sided box makes sense, whereas a six-sided box for each process sensor in a rack of process sensors is not feasible – “unobtanium”. As a result, these critical devices have been excluded from NERC CIP cyber security programs. Yet, “compromised” process sensors have led to the destruction of a multi-million-dollar boiler feedwater pump (and extended plant downtime) and almost caused a regional European blackout. I have many more examples of process sensors involvement in catastrophic control system cyber incidents.
Russia, China, and Iran are aware of the cyber security gaps in these devices and in some cases are currently exploiting the lack of sensor authentication. China has provided the market with counterfeit pressure and differential pressure transmitters. It is hoped that getting people within the electric and nuclear sectors to understand the process sensor problems can be started within 100 days.
Physics-based events
I find it alarming there was no mention of mitigating physics-based events such as the Aurora vulnerability. Aurora can bring the grid down for 9-18 months and the information was declassified by DHS in 2015. In the 2015 Ukrainian cyberattack, Russia performed step 1 of the 2 steps of Aurora. The Aurora hardware mitigation process, including awareness and training, can be started in the next 100 days.
People issues
There was no mention of the “people” issue which can defeat any security technology. The culture gap between network security and domain engineers is alive and well. After 20 years, it won’t be overcome in 100 days. The bromide of “insecure by design” is misleading at best. ICSs were designed for safety and reliability and these systems are very good at meeting those requirements – cyber security was not part of the design requirements. Nowhere is the gap more egregious than addressing process safety. To network people, they are used to living with the “blue screen of death” and use of cyber security tools are part of their daily work. Control system engineers have very, very stringent requirements to meet safety requirements and cyber security tools are not an integral part of their daily job (even though they should be).
There has been very little information sharing of actual control system cyber incidents. If you don’t know what has occurred, how can you protect yourself? Yet, there have been more than 350 control system cyber incidents (unintentional and malicious) in the North American electric industry.
There are numerous “people issues” activities ongoing inside and outside the government that need to be coordinated.
The Supply Chain and Presidential Executive Order (EO) 13920
DOE has issued a Request for Information (RFI) concerning cyber security of the electric industry supply chain. Compromise of the ICS supply chain is not new nor has it been sufficiently addressed. China has compromised the ICS supply chain since the 2012 timeframe and the Russians since at least 2014.
There are more than 200 large Chinese-made transformers in the US bulk electric grid. This includes in offshore wind platforms and a High Voltage Direct Current (HVDC) substation. Chinese-made transformers support 10% of the New York City load. Other US cities use Chinese-made transformers to provide a significant part of their load. Michael Mabee has done a great job of digging into this problem and you can find his blog: “Chinese Transformers in the Electric Grid: Lights Out For NYC?” https://michaelmabee.info/chinese-transformers-in-the-electric-grid-lights-out-for-nyc/.
EO 13920 was issued not as a whim but because the Chinese effectively did a “Stuxnet” to a large electric transformer installed at a US utility. Specific to EO 13920, a Chinese-made transformer was found with hardware backdoors installed at a US utility in August 2019. As a result, the equipment to be addressed in EO 13920 is engineering hardware and ICS devices while networking equipment was specifically excluded. There are multiple pathways that can be used to exploit the backdoors to take control of the transformer. The potential impacts of that discovery resulted in the next large Chinese-made transformer being intercepted by DOE in early 2020 and sent to the Sandia National Laboratory (SNL) for examination. To date, there has been no disclosure of any findings from SNL on what was found in the transformer. Therefore, it is unknown what can be done to identify potential hardware backdoors or what actions should be taken when these backdoors are found. It was unnerving when senior representatives from close US allies were asking me about the Chinese transformer issue because they also have these Chinese-made transformers and were also not informed why EO 13920 was issued or what has been found to date.
Industry has assumed software bill of materials and adequate procurement specifications (even though control system device procurement guidelines do not exist) would be sufficient to address these hardware backdoor issues. They are not.
Supply chain cyber security is not just a transformer or grid issue. China has supplied pumps, valves, motors, relays, and other equipment world-wide. I previously identified where Chinese backdoors were installed in a pharmaceutical manufacturer using 5G technology. What other software or hardware backdoors exist in Chinese-made equipment used in critical applications?
Conclusion
I am happy to see cyber security of control systems being identified by the Secretary of Energy as important. Unfortunately, you cannot undo more than twenty years of inadequately addressing the cyber security of control systems and devices in 100 days. There is also the need to change electric utilities’ focus from compliance to security. Hopefully, this senior level government attention can help move the needle on securing control systems.
Joe Weiss
