Are the Good Guys as Dangerous as the Bad Guys – an Almost Catastrophic Failure of the Transmission Grid

Nov. 2, 2017

A security group at a large utility with experience only scanning data center assets scanned a number of critical transmission substations. The scanning cut all communication between hundreds of relays and SCADA was unaware.

As stated many times, there are few forensic tools for ICS cyber security and fewer that can discriminate between an unintentional mistake and malicious intentions. Moreover, many of these incidents are intentional, that is, scheduled, but the end user is not aware of potential unintended consequences.

I was recently made aware of a large utility that performed security scans of a number of very critical substations. Until recently, the security group was only scanning data center assets and then expanded the scanning into NERC CIP substations, starting primarily at the 230/500KV level. The security group had no previous experience with scanning substations. No notification was given for the scanning change to the internal support groups that are responsible for this function. The OT Team was notified that substation scanning was started with a new security port scanning tool. Following the scans, the relays showed trouble, but the DNP polling was working properly and the networks in most substations were stable – SCADA was unaware of the problems. The port scanning of this new tool caused the real time protocol operation of the relays (IEEE61850/GOOSE) to stop and suspend operation at the CPU (two different relay suppliers) and left the DNP/non-real time operations alone - the worst possible circumstance. In order to clear the trouble, each relay had to be cut out and rebooted, to restore operation. Several hundred relays were affected. All the devices in each substation were affected at the same time in every case. Without knowing that a security scan was initiated, it looked like a DDOS attack resulting in equipment malfunction.

This case reinforces that:

-        IT security should NEVER be left alone in industrial operations

-        IT security should NEVER use a tool that hasn’t been thoroughly tested for use in OT environments

-        It is not always clear what is or isn’t a cyber event

-        SCADA is not a fail-safe to identify potential cyber attacks. There have been other cases where SCADA, by design, did not detect critical “malfunctions”.

What other types of catastrophic situations are as yet undiscovered that good guys can tumble into or bad guys use?

Joe Weiss

Sponsored Recommendations

Measurement instrumentation for improving hydrogen storage and transport

Hydrogen provides a decarbonization opportunity. Learn more about maximizing the potential of hydrogen.

Get Hands-On Training in Emerson's Interactive Plant Environment

Enhance the training experience and increase retention by training hands-on in Emerson's Interactive Plant Environment. Build skills here so you have them where and when it matters...

Learn About: Micro Motion™ 4700 Config I/O Coriolis Transmitter

An Advanced Transmitter that Expands Connectivity

Learn about: Micro Motion G-Series Coriolis Flow and Density Meters

The Micro Motion G-Series is designed to help you access the benefits of Coriolis technology even when available space is limited.