As stated many times, there are few forensic tools for ICS cyber security and fewer that can discriminate between an unintentional mistake and malicious intentions. Moreover, many of these incidents are intentional, that is, scheduled, but the end user is not aware of potential unintended consequences.
I was recently made aware of a large utility that performed security scans of a number of very critical substations. Until recently, the security group was only scanning data center assets and then expanded the scanning into NERC CIP substations, starting primarily at the 230/500KV level. The security group had no previous experience with scanning substations. No notification was given for the scanning change to the internal support groups that are responsible for this function. The OT Team was notified that substation scanning was started with a new security port scanning tool. Following the scans, the relays showed trouble, but the DNP polling was working properly and the networks in most substations were stable – SCADA was unaware of the problems. The port scanning of this new tool caused the real time protocol operation of the relays (IEEE61850/GOOSE) to stop and suspend operation at the CPU (two different relay suppliers) and left the DNP/non-real time operations alone - the worst possible circumstance. In order to clear the trouble, each relay had to be cut out and rebooted, to restore operation. Several hundred relays were affected. All the devices in each substation were affected at the same time in every case. Without knowing that a security scan was initiated, it looked like a DDOS attack resulting in equipment malfunction.
This case reinforces that:
- IT security should NEVER be left alone in industrial operations
- IT security should NEVER use a tool that hasn’t been thoroughly tested for use in OT environments
- It is not always clear what is or isn’t a cyber event
- SCADA is not a fail-safe to identify potential cyber attacks. There have been other cases where SCADA, by design, did not detect critical “malfunctions”.
What other types of catastrophic situations are as yet undiscovered that good guys can tumble into or bad guys use?
Joe Weiss
