To cyber secure the grid, the North American electric industry has to meet a set of standards known as the North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection (CIP) standards. These standards were approved by the Federal Energy Regulatory Commission (FERC). As NERC is fond of saying, the CIP standards are mandatory and enforceable.
The CIP standards were developed in the 2003-5 time-frame taking little to no input from the power plant, substation, or control system experts inside and outside of the utility industry. Rather, the NERC CIP developers were the IT, SCADA, NERC Coordinators, and physical security experts with the constraints of keeping the CIPs limited in cost and scope. This was done without understanding potential unintended system interactions associated with the control systems in power plants and substations. Because of the system interactions, the IT/OT organizations (network security) cannot be effective in securing the electric grid without the insights and guidance of the engineers that design and operate the grid. Very few network security professionals understand how serial and Ethernet data streams relate to each other much less what are Purdue Reference Model Level 0,1 control system devices. As a result, the important and pervasive cyber issues associated with control systems were, and continue to be, excluded. Many of the power plant and substation operational experts are still directly or indirectly kept out of the NERC CIP process. This situation will only become worse as we depend more on advanced grid modeling and control.
For transmission and large power generation known as the Bulk Electric System (BES), NERC CIP-002 is the critical CIP. CIP-002 “identifies and categorizes Bulk Electric System (BES) Cyber Systems and their associated BES Cyber Assets for the application of cyber security requirements commensurate with the adverse impact that loss, compromise, or misuse of those BES Cyber Systems could have on the reliable operation of the BES.” Purdue Reference Model Level 0,1 process sensors (e.g., pressure, level, flow, temperature, voltage, current, frequency, etc.) provide real-time input to all BES and distribution systems to maintain system reliability and safety. They also deliver information to operators via all BES and distribution operator displays for operations information.
As NERC says, violations of the NERC CIP requirements are punishable and there have been many NERC CIP violations for programmatic issues. There was also recent FERC rulemaking on making these violations public. However, a utility recently discovered THOUSANDS of process sensors directly connected to the corporate network and from there to the public Internet. Because process sensors are “cyber defenseless” with no cyber security, authentication, or cyber logging capabilities, a direct or indirect connection from the public Internet to the sensors can “open” sensors for compromise including changing sensor readings or preventing initiation of safety functions. As mentioned in my blog, https://www.controlglobal.com/blogs/unfettered/the-ultimate-control-system-cyber-security-nightmare-using-process-transmitters-as-trojan-horses/, this opening can also lead to the introduction of malware behind the firewalls – the ultimate Trojan horse. Such compromises can directly impact any BES or electric distribution system, regardless of priority. Erroneous sensor readings can also lead to compromise of regulatory compliance beyond just cyber security compliance.
In this case, the utility had brought in a diverse group of public and private industry cyber security experts to assure the cyber security of the grid including the control systems. However, because they did not ask the right questions, they did not find the sensor issue. This was because of the lack of engineering, not cyber security, insight necessary to determine the potential impact of the compromised sensors to the generation, transmission, or distribution systems. Speaking from experience as a nuclear engineer, I spent many years working on nuclear plant safety by performing detailed analyses of sensor issues that could impact safety and reliability. The complex control systems supporting the electric grid require an analysis first by engineering to determine which systems can be compromised, how an exploitation might be possible, the consequences from their exploitation, and how they might (or might not) be protected; afterward, IT cyber security can better assess their options for risk mitigation. In the case of this utility, without the proper engineering resources, they apparently did not realize that the process sensors would be exposed to the public Internet through the corporate network and the resultant potential impact on the reliability and safety of the grid.
The entire strategy of cyber security risk must be re-visited when control systems are part of the environment of critical infrastructure. While corporate firewalls are a standard, well-accepted tool to protect IT networks, they do not provide adequate protection to control systems because firewalls for IT communications do not have the same restrictive rules (or tracking of rules changes) as the firewalls protecting the Electronic Security Perimeter (ESP). However, process sensor communications, is different from Internet Protocol communications - effectively bypassing the means that corporate or ESP firewalls use to keep malicious traffic off networks. Even Deep Packet Inspection firewalls look at data structure, not necessarily the content, especially content that seems normal. It is imperative that executive leadership include proper engineering expertise to analyze process sensor issues and control system cyber risk if the integrity of the Electric Grid is to be maintained.
This is not the first time that control systems directly connected to business networks have caused unintended impacts. More than 10 years ago, a nuclear plant was automatically scrammed (shutdown) when a server on the business network rebooted while being directly connected to a critical plant controller. Moreover, cyber attacking sensors and sensor networks are known to our adversaries. Two specific examples include Russian researchers demonstrated attacking process sensor networks and an “individual” from Iran sent a “Like” to my LinkedIn account from my 2017 Defcon presentation on the lack of cyber security in process sensors.
The International Society of Automation (ISA) 99 Committee, developed the 62443 series of control system cyber security standards around the concept of network segmentation. If the utilities followed the ISA standards, current and legacy sensors, without any significant built-in cyber security features, would be protected as part of highly segmented networks with multiple layers of isolation and separation. However, there is very little utility participation in the ISA99 process.
Connecting process sensors that will be used for important functions (or connected to other sensors that are used for important functions) directly to corporate business networks or the Internet is a major organizational failure, a clear violation of generally accepted cyber security principles, and a significant safety concern. (It should be evident it takes an engineering analysis to determine which process sensors are critical and which are not.) Yet, connecting the process sensors to the corporate network thus missing many layers of cyber security does not appear to be a NERC CIP violation as it doesn’t fall into any of the violation criteria identified in the CIPs even though this is major safety issue. Ironically, it would be a NERC CIP violation to not submit paperwork in a timely fashion even though this is not a safety issue.
If the Nation is to ensure the Electric Grid is reliable and safe against cyber risk, FERC, DOE, DHS, EEI, NRECA, APPA, NARUC, and others should expeditiously require control system cyber security requirements and engineering expertise be utilized for the Bulk and electric distribution systems. This specifically includes the most vulnerable engineering devices and their lower level networks. Exposing thousands of these critical devices to the Internet should be a formal violation.
To learn more about what should be asked, please contact me at [email protected]
Joe Weiss
