The October 2, 2021 oil pipeline breach off Huntington Beach, California, was identified when workers saw a sheen of oil on the water. There was a leak detection system, but according to an Associated Press article, it wasn’t “fully functional.” I don’t know what that means. Consequently, I asked a representative from the Transportation Security Administration (TSA) if the pipeline cyber security controls covered leak detection systems and associated sensors. I was told that control system devices were outside the scope of their responsibility. This may seem surprising, but we’ve seen this before. This blog is addressing the control system cyber security gaps that prevent pipeline, rail, and aviation control system users from being able to meet TSA cyber security directives.
TSA cyber security directives
The TSA pipeline cyber security directive requires owners and operators of TSA-designated critical pipelines to implement specific mitigation measures to protect against ransomware attacks and other known threats to information technology and operational technology systems, develop and implement a cybersecurity contingency and recovery plan, and conduct a cybersecurity architecture design review. The TSA rail cyber security directive requires most passenger and freight rail operators to identify a cybersecurity point person, report incidents within 24 hours to CISA, conduct a vulnerability assessment, and develop a contingency and recovery plan in case of malicious cyber activity. TSA updated its aviation security programs to require that airport and airline operators implement the first two provisions above; that is, these operators must designate a cybersecurity coordinator and report cybersecurity incidents to CISA within 24 hours.
Inability to meet TSA security directives
It would have been expected the TSA directives would have addressed all of the critical cyber risks the pipeline, rail, and aviation operators face. However, in practice, the directives don’t apply to the control systems themselves, just the networks that serve them. (These same control system issues are not being addressed by CISA’s IT network approach to cyber security in every sector including power, water, manufacturing, etc.) For example, ransomware is an IT problem and vulnerability assessments don’t apply to Purdue Reference Model Level 0,1 devices (e.g., process sensors, actuators, drives, analyzers, etc.) which have no cyber security, authentication, or cyber logging. These catastrophic unaddressed risks have killed hundreds and cost billions of dollars.
The ability to identify incidents as cyberattacks may not be possible. Specifically, the lack of cyber logging capabilities in Level 0,1 devices, lack of cyber security training for control system engineers, and lack of the ability to correlate network anomalies with equipment diagnostics can make it difficult to identify cyberattacks. In some cases, the only difference between an event being a cyberattack rather than an unintentional incident is the motivation of the person. In the case of Stuxnet, a sophisticated cyberattack was misinterpreted as systemic design deficiencies that caused equipment malfunctions for more than a year before Stuxnet was identified as a cyberattack. In June 2017, the safety systems in a petrochemical plant in Saudi Arabia in 2017 were cyberattacked (the safety systems attacked are used extensively throughout the US in oil and gas, chemical, water/wastewater, nuclear, and other industries including in federal facilities). The petrochemical plant shutdown because of malware in the engineer’s safety system workstation. However, the shutdown initially was not identified as being caused by a cyberattack and consequently the plant was restarted with the malware still installed!
A 2021 Director of National Intelligence (DNI) National Intelligence Estimate states that Chinese-made transformers, circuit breakers, and inverters creates cyber vulnerability risks (https://www.controlglobal.com/blogs/unfettered/dni-identifies-chinese-transformers-as-cyber-vulnerable-risks-yet-doe-and-industry-ignore-the-threat/). There are more than 300 large Chinese-made transformers installed in the US electric grid (and many more throughout the world) and more than 170 million inverters have been imported from China since 2002 (more than 5 million in 2021). Electrified trains use electric transformers. Circuit breakers are used to protect critical electromechanical equipment used in pipelines and electrified rail. Inverters are used in variable frequency drives used in natural gas pipeline compressor stations and in gasoline pipeline variable speed gearboxes and reciprocating engine pumps. Yet, the control system cyber incidents in rail, pipelines, and aircraft including catastrophic events that have killed people, have not been identified as being cyber-related (my non-public database has almost 12 million control system cyber incidents). Training and culture have not caught up to these gaps.
Moreover, there has been no participation from TSA in the ISA99 control system cyber security standards efforts. How can TSA be issuing cyber security guidelines for pipelines, rail, and aviation without technical understanding of the control system issues?
Pipeline cyber security risk
There have been more than 50 control system cyber incidents (malicious and unintentional) in natural gas and liquid pipeline systems. Ironically, from a control system cyber perspective, the Colonial Pipeline incident was not important as it did not touch the OT network or the control systems. Yet the focus of many of the post-Colonial Pipeline recommendations focused on the potential connections between the IT and OT networks which is a known vulnerability. However, the lack of control system inventory (can be identified from OT network monitoring capabilities), device cyber security, control system device authentication, and control system device cyber logging prevent pipeline operators from meeting the new cyber security guidelines. There is also a lack of control system cyber security training based on actual incidents for the control system engineers, technicians, and pipeline operators to be able to recognize equipment malfunctions as possibly being cyber-related.
Through May 2021, critical pipeline operators reported more than 220 cybersecurity incidents since the TSA implemented emergency measures in the wake of the crippling ransomware attack on one of America's most important pipelines, according to TSA Administrator David Pekoske. I am not aware of any recent reports of pipeline ruptures or pipeline outages during that timeframe meaning the 220 cyber security incidents were IT incidents not affecting the operation of the pipelines. Yet the two cyber-related pipeline ruptures that killed people and destroyed structures would not have been addressed by the TSA cyber security guidelines.
There have been three pipeline cyber-related pipe ruptures that were not identified as being cyber-related and would not be addressed by the TSA requirements.
- The 1999 Bellingham, WA Olympic Pipeline Co. pipeline rupture appeared to be the result of a broadcast storm (unknown whether malicious or unintentional). It also impacted the leak detection system that alarmed 1 hour after 230,000 gallons of gasoline spilled into the creek and caught fire. The event killed three, three people went to jail, destroyed a water treatment facility, and resulted in the bankruptcy of the Olympic Pipeline Company. October 8, 2002, NTSB issued a report on the Olympic Pipeline rupture (https://www.ntsb.gov/investigations/AccidentReports/Reports/PAR0202.pdf). The report stated that SCADA was the proximate cause of the failure.
NIST’s Ron Ross had arranged for NTSB to provide MITRE’s Marshall Abrams and myself the publicly available information on the accident. We were told by NTSB this was the most complex pipeline case they had addressed because of the SCADA issues. We performed an analysis to:
- Determine what happened to SCADA, the control systems, and the process,
- Identify the NIST 800-53 controls that were violated,
- Determine if the NIST 800-53 controls could have prevented the event, and
- If not, identify additional new NIST 800-53 controls
The MITRE report can be found at https://icscsi.org/library/Documents/Case_Studies/Case%20Study%20-%20NIST%20-%20Olympic%20Pipeline%20(presentation).pdf
- The 2010 PG&E San Bruno natural gas pipeline rupture resulted from the replacement of an uninterruptible power supply that initiated control system logic that hadn’t considered the presence of a weak pipe. The PG&E natural gas pipe rupture killed eight people, destroyed a neighborhood, cost PG&E billions of dollars, and made PG&E a convicted felon. The PG&E pipe rupture had many similarities to the Olympic Pipeline Company gasoline pipeline rupture. There has been no guidance issued on either Bellingham or San Bruno to address the control system cyber security issues.
- DOD was conducting radar testing in Europe and inadvertently blew-up a large pipeline because of electromagnetic issues. There has been no guidance issued to address the ElectroMagnetic Interference (EMI)/Radio Frequency Interference (RFI) issues.
According to the Associated Press, October 2, 2021, a ruptured offshore pipeline spilled tens of thousands of gallons of crude oil off the Southern California coast did not have a fully functioning leak detection system at the time. The report compiled by the pipeline operator is investigating whether personnel or control room issues contributed to the accident. The report does not explain what was wrong with the detection system. It is not clear if cyber was involved but unintentional and malicious cyber issues with leak detection systems including hacking leak detection systems in the Huntington Beach area is not new. In 2008, a former IT consultant pled guilty to tampering with the SCADA system in Huntington Beach after the company rejected his request for permanent employment. He played a role in setting up the SCADA system that communicated between its headquarters and oil platforms, and which was also used as a leak detection system.”
Liquid and natural gas pipelines have remotely operated safety isolation valves that are used to isolate sections of pipeline from catastrophic events like pipeline ruptures. SCADA systems are used to monitor the isolation valves. In some states, if the remote control and monitoring of the valves is lost then a human has to be dispatched to the remote sites 24/7 for manual operation needs. Often the valves are located in very remote areas and voice communication can be limited over cellular. Many of these sites use cellular and satellite uplink systems for SCADA monitoring and control (potentially additional cyber vulnerabilities). Many of these critical pipeline applications utilize low cost, cyber insecure controllers with no cyber forensic capabilities. Additionally, in the case of the PG&E accident, it took more than 40 minutes to locate the manual shutoff/isolation valves following the pipe break. As a result, an unintended consequence of the PG&E event was requiring natural gas shutoff valves to be remotely automated which has now made the natural gas industry cyber vulnerable. The concern is inappropriately closing isolation valves can cause pipe failures. As natural gas is currently the largest form of power generation, compromising natural gas pipelines can affect the power grid. The control system issues are not addressed by the TSA requirements.
The Aurora vulnerability is a physics-based attack that uses no malware. The Aurora vulnerability can affect pipeline integrity. In July 2015, DHS declassified more than 800 pages of the previous classified Idaho National Laboratory Aurora test information. Two of the slides that were declassified identified that an Aurora event could damage the rotating equipment in a refinery and the other identified how Aurora could damage natural gas compressor stations. There are no cyber forensics to identify Aurora events and is not addressed by the TSA requirements.
Rail cyber security risk
There have been more than 25 control system cyber incidents in domestic and international commuter, long-distance passenger, and freight trail operations including fatal train crashes, complete loss of power to track switches and electric substations, loss of control of remote-controlled engines, etc. The 2009 DC Metro Red Line crash was due to sensor, control system, and signaling issues that have also occurred in other rail systems. As a result, I was an invited participant on a National Academy of Science Transportation Research Board panel to oversee the development of effective practices for the protection of transit infrastructure from cyber incidents. There were about 20 participants on the panel with only two (myself and the Chief Engineering Officer, Communications & Signals, from a major commuter railroad) that were control system/signals engineers. Despite the protestations of myself and the Chief Engineering Officer, the final deliverables did not address the control system issues involved in the Red Line train crash but basic IT cyber security. The TSA cyber security rail requirements are similar to the IT deliverables and do not address the control system issues.
Aviation cyber security risk
There have been more than 50 control system cyber incidents in aviation. As an example, after the two Max 737 crashes, I had a discussion with a senior FAA representative. He acknowledged that sensors weren’t part of the FAA cyber security program even though sensors were the cause of Boeing and Airbus crashes. The TSA requirements would not address these events.
Conclusions
Control system cyber incidents affecting pipelines, rail, and aviation have caused catastrophic damage resulting in multi-billion-dollar impacts and hundreds of deaths. However, the TSA response is effectively a reaction to the Colonial Pipeline cyberattack and other IT ransomware incidents that did not affect control systems or cause physical damage. Detecting cyberattacks against IT and OT networks can be done today. However, the same cannot be said for detecting control system cyber incidents (attacks and unintentional incidents) that occur with the cyber insecure control system field devices. This means the TSA security directives of reporting incidents within 24 hours to CISA, conducting meaningful vulnerability assessments, and developing contingency and recovery plans in case of identified malicious cyber activity cannot be met for control system-related incidents.
Recommendations
Control system cyber security training, contingency, and recovery plans need to be developed based on real control system cases. Process sensor integrity and authentication is needed to recognize system/equipment malfunctions that could be cyber-related. There is a need for government and industry to coordinate the myriad standards and governmental activities on critical infrastructure cyber security to assure there are no inconsistencies. Government funding to accredited standards organizations like ISA and IEEE can accelerate the standards development. It seems that this is an oversight that would be easily corrected, with tremendous benefit to the safety and security of control system processes everywhere.
Joe Weiss
