As I was reviewing my blogs for a paper I was preparing, I found a nuclear power plant incident involving a station auxiliary transformer load tap changer (LTC) failure. Substation transformers have been acknowledged as the Achilles heel of the electric industry. As a result, the 2015 FAST (Fixing America’s Surface Transportation) Act contained a section on the need to create a Strategic Transformer Reserve. As LTCs are used in many substation transformers including main transformers, generation step-up transformers, station auxiliary transformers, grounding transformers, etc. and can cause transformer failures, they can impact the Strategic Transformer Reserve. Station auxiliary transformers are used in power plants (fossil, nuclear, and hydro) and other large industrial installations that have their own substations. These include refineries, paper and pulp, water treatment, etc. Station auxiliary (or standby auxiliary) transformers provide power to the auxiliary equipment during normal operation. An LTC failure can be catastrophic to the transformer and can also affect the reliability and possibly safety of the facility. It is estimated that about 20-40 percent of LTC-equipped transformer failures are caused by LTC failures. One of the causes of LTC and transformer failures is failure of the LTC motor.
In the nuclear plant case, troubleshooting determined the LTC alarm in the Main Control Room was initiated by the failure of the LTC motor from the station auxiliary transformer in the plant switchyard. The LTC motor failed due to continuous tap change demands from the transformer Automatic Voltage Control (AVC) system for more than an hour before the LTC alarm was initiated. The cause was an AVC firmware failure which caused the erroneous output signals to continuously run the LTC motor resulting in a failure of the LTC motor and ultimately the transformer.
This nuclear plant station auxiliary transformer failure was unintentional. However, with remote access, the firmware could be maliciously changed to cause the same impact. The hardware implants at the Chinese-made transformer with the hardware backdoors can enable spoofed signals to be sent to the LTC causing the same failure mode and would not be detected by cyber monitoring. The hardware backdoors could also prevent the transformer monitoring system from detecting the LTC motor continuously running like the “man-in-the-middle” Stuxnet attack.
There are many issues with the nuclear plant incident that are relevant to the Chinese-transformer case and Presidential Executive Order (EO) 13920. The EO explicitly addresses LTCs.
- The LTC used in the nuclear plant station auxiliary transformer is a common LTC used in many power plant and industrial facility applications.
- The generic failure mode of the LTC motor is applicable to all LTCs in all transformers.
- Transformer maintenance is generally not done by power plant staff but by transformer experts. With the Chinese transformers, the maintenance may be performed by the Chinese transformer vendor field staff.
- There was no discussion as to why it took more than an hour to recognize the failure. The lack of recognition is surprising since most solid-state LTC controls have the ability to indicate too many operations within a set time frame by defining settings in the control panel.
- Transformers and the LTCs are not addressed by NRC Regulatory Guide 5.71/NEI-0809, the NERC Critical Infrastructure Protection (CIP) standards, the NERC Supply Chain requirements, or other industry cyber security guidance. The agreement between FERC and NRC on nuclear plant cyber security is the transition point being the first isolation after the main generator. Consequently, the NRC/FERC agreement excludes the auxiliary transformer in the switchyard and the substation outside the fence line from being addressed by Reg Guide 5.71/NEI-0809. Yet, there is a publicly available NRC Licensing Event Report (LER) that identified the shutdown of the switchyard transformer from an LTC failure.
- There was no operator training to identify if this incident was a possible cyber incident or to address the possible cyber implications in the root cause analysis.
The industry response to the EO is focused on procurement guidelines for new equipment essentially ignoring the 200+ Chinese transformers already installed in the US grids. It also doesn’t account for the Chinese knock-off LTCs identified in https://www.controlglobal.com/blogs/unfettered/ics-cyber-security-is-the-second-coming-of-the-maginot-line-and-the-chinese-have-breached-it/.
Cyber security of LTCs and other equipment identified in the EO are significant safety and security gaps that need to be addressed. There are concerns with information infiltration through counterfeit electronics' firmware. Why is there little concern about a device that costs upwards of $5 million dollars, can take months to repair or replace, and have a debilitating impact on the grid if damaged or destroyed? Recovery of information can be obtained through backups. Recovery of transformer failures cannot be as easily repaired. The enterprise risks from the equipment identified in the EO such as the LTC and transformers can be very high. As such, the potential impacts to existing equipment and associated enterprise risk impacts may be of interest to credit rating agencies such as Moody’s.
Joe Weiss
