Cyber vulnerable Uninterruptible Power Supplies (UPSs) have caused physical damage to data centers

Cyber security of the control system devices in buildings and data centers including process sensors, valves, actuators, and power supplies have limited cyber security. In February 2021, Lawfare published Alpha Guardian’s Bob Hunter and my article on SolarWinds impacts on control system which focused on building controls including the Uninterruptible Power Supplies - UPSs (https://www.lawfareblog.com/solarwinds-hack-can-directly-affect-control-systems). Yet, guidance on addressing SolarWinds has been focused on IT or OT networks even though the network security guidance may not apply to these control systems devices. As a result of our concerns, Bob and I wrote Chapter 20, “Cyber-Security and Data Centers” in the 2^nd Edition of Data Center Handbook: Plan, Design, Build, and Operations of a Smart Data Center.

UPSs

UPSs are used in all types of facilities to provide continuous operation during off-site power failures. These include data centers; electric, gas, and water control centers; manufacturing facilities; as well as business IT equipment. UPSs generally provide two functions:

- They supply interim power when power is lost from “house loads” until backup generators are started.

- They smooth the voltage from the backup generators. so the servers are only fed the design voltage, rather than the fluctuating voltages and frequency produced by local generators as the load varies.

UPSs supply mission critical power and are routinely monitored using the Simple Network Management Protocol (SNMP) over Ethernet.  Unfortunately, most UPSs are still monitored using insecure SNMPv1 or v2 which CISA has declared to be highly cyber vulnerable.  Even SNMPv3 has inherent cyber flaws that make it vulnerable to attack. Further, many SNMP cards manufactured in China have backdoors that can compromise the UPSs.

Using an IoT search engine, more than 100,000 UPS/Power Distribution Units (PDUs), 50,000 Battery Monitoring Systems, and 100,000 Building Management Systems at critical facilities are viewable on the Internet. 

https://www.controlglobal.com/blogs/unfettered/solarwinds-orion-the-weaponization-of-a-network-management-system. These are very conservative numbers as APC, a subsidiary of Schneider Electric, has sold more than 20 million UPS devices worldwide.

Vulnerabilities for causing damage

There is minimal guidance for cyber securing UPSs even though compromising the UPSs can directly lead to data center equipment damage. Specifically, SNMP interface cards allow shutting down UPSs, scheduling shutdowns and restarts of the UPS; turning off power to selected UPSs and draining or disconnecting backup batteries. Remotely changing UPS settings, whether malicious or unintentional, can lead to fires or battery chemical releases that can cause facilities to be evacuated.

On March 8, 2022, Armis Security published a cyber security disclosure of three cyber vulnerabilities in Schneider-Electric’s APC Smart UPS devices. These critical vulnerabilities can allow attackers to remotely manipulate the power of millions of UPSs. In one case, Armis researchers were able to remotely ignite a Smart-UPS device.

Selected data center UPS incidents

- In the December 2015 Ukrainian grid cyberattack, the attackers compromised the UPSs in a communication facility to prevent communications.

- In May 2017, a European data center experienced a major power outage due to an electrical grid power surge that damaged the data servers. An investigation found that a UPS was over-ridden resulting in a hard power shutdown.  While the UPS is supposed to act as the first line of defense in an actual power event, it can also be used at the first line of attack in a cyber/physical attack. In this case, all UPS-supported power to servers and network equipment in the data center was shut down. This resulted in the total immediate loss of power to the facility, bypassing the backup generators and batteries. This meant that the controlled contingency migration to other facilities could not be applied. After a few minutes of this shutdown of power, the UPS was just as mysteriously turned back on in an unplanned and uncontrolled fashion.  The result was both the battery supply and the generator supply being connected in series to the power bus feeding the racks. That resulted in the data center’s servers being fed 480v instead of 240v, causing physical damage to the servers.

- In March 2021, French data center hosting company OVH suffered an extensive fire at its SBG2 facility, followed by a battery-related incident that filled its SBG1 facility with enough smoke that it was deemed irreparable less than 2 weeks later. In the case of both OVH incidents, the culprits were the UPS and battery systems. UPS units are frequently at fault for thermal events causing fire suppression system releases. An obvious question is could the UPSs have been compromised.

Summary and Recommendations

Cyber security of the control system devices in buildings and data centers including the UPSs have limited cyber security capabilities. Remotely changing UPS settings can wreak havoc on the equipment they are designed to support.

Address UPS cyber security, including sensors, as control system hardware issues. Use the latest version of SNMP buttressed by hardening the network on which the UPS is located per CISA recommendations. Provide appropriate training for engineers and technicians as these are not network data issues. Develop secure UPSs that can employ secure protocols and include hardware interlocks to prevent unsafe conditions.

Joe Weiss