I participated in the Naval War College Cyber War Games July 25-26, 2019 in Newport, RI https://usnwc.edu/News-and-Events/News/US-Naval-War-College-Holds-War-Game-Looking-at-Cyber-Defense-of-the-Private-Sector. The War Game was entitled: “Defend Forward 2019 Critical Infrastructure War Game” and was focused on the electric and finance industries. There were senior representatives from government, DOD, DHS, electric, and finance companies.

The war games were interesting. They provided not only lessons learned, but also powerful training tools for the participants. It’s worth noting that such games might well be expanded to include players beyond the executive ranks. Engineering and operations personnel would also benefit from them, and, more importantly, would make valuable contributions to the lessons learned.

These are my thoughts as they apply to control systems. I am not addressing policy or other non-technical issues.

The scenario.

The war game’s scenario was as follows: Green State, an ally of Blue State, was holding an election that would affect Red State. Red State wanted to ensure the election of their preferred candidate. The game included “noise,” that is, issues that may not have been introduced by the Red Team, as well as “injects” during the game to confuse the Blue State defenders.

The game was structured with four large, vertically integrated utilities geographically distributed throughout the country of Blue State. The initiating attacks involved malware being injected from the IT networks into the OT networks (as the IT and OT networks were connected). Each of the four utilities were affected. Consequently, mutual aid was impossible, as no utility could spare its cyber security experts to support the others. (It’s also unclear that a utility would be willing to give another cooperating company root OT-network access to their OT networks even when that company was there to render assistance.) And given the limited number of control system cyber experts, there is unlikely to be enough government technical support available to all four of the utilities.

Highlights from the game.

The Game was typical of cyber meetings and conferences where the vast majority of attendees were from cyber security or senior management. In other words, there was very little, if any participation from the Engineering or Operations organizations. For Finance, that can be acceptable. For Electric, that can be misleading and even dangerous. But one of the important lessons you learn from these kinds of exercises is what is the right mix of people and skills. That came out clearly over the course of the game.

Grid malware attribution was assumed to be from the Red Team. What was discovered after the game was that the grid malware did not in fact emanate from the Red Team but from other sources. It was, within the context of the exercise, noise. The intrusions into the Blue State Finance and Electric sectors were intended to influence the Green State elections and not to attack the grid. But the participants in the exercise interpreted those actions as Red Team’s malicious activity.

Discovery of the malware led utilities to take expensive evasive actions including a destabilizing severing of all third-party connections. Such critical third-party applications as payroll, connections for energy market participants, and vendor support were terminated. One artificiality of the game was the very long run-up (six months in scenario time for the first move), largely to reinforce the point that cyber operations require considerable preparation time. However, severing of third-party connections in response for six months caused considerable concern among the utilities about operating for extended periods in manual mode, with the associated financial and personal impacts. This was an area where Operations expertise could have been valuable as the utility actions taken to respond to the grid malware could have adversely impacted grid reliability, energy markets, and utility stock prices.

In a disturbing reaction to the malware mitigation, some of the utilities reconnected their IT networks back to their OT networks and the Internet as though nothing had happened or had been learned. Difficult or not, OT networks should never be directly connected to IT networks or the Internet. Doing so is asking for another OT malware infection.

I had provided my blog on counterfeit transmitters to the Naval War College prior to the games - https://www.controlglobal.com/blogs/unfettered/the-ultimate-control-system-cyber-security-nightmare-using-process-transmitters-as-trojan-horses/. The Counterfeit “SCADA parts” issue was “used” by the Red Team and became a part of the Energy industry discussions. In fact, within the context of the exercise the counterfeit SCADA parts resulted in a Presidential Grid Security Emergency (GSE) declaration. I found it interesting the player acting as the President of the United States (POTUS) would sign a declaration on “SCADA parts” that are not overseen by NERC CIP and the NERC Supply Chain submittal. If the acting POTUS feels the counterfeit parts issues is that important, shouldn’t NERC expeditiously change the NERC CIPs and the NERC Supply Chain submittal to include these critical devices? This is the type of significant risk that should interest credit rating agencies.

The assumption that it was the Red Team’s grid malware also led the players representing the Department of Defense to take offensive actions against the Red State. I was concerned with the utilities’ and DOD’s aggressive responses to malware that did not impact any grid or power plant operations. That is, there were no outages or damage to any grid or power plant equipment yet the responses to the malware almost spiraled out of control on both individual utility and national cases. It is critical to have participation from Operations experts to help influence utility technical responses and utility public relations experts to positively influence public-private utility relations.

Unfortunately, finding malware on grid OT networks is not unusual, which is why the IT and OT networks need to be isolated from each other. The Russians installed the BlackEnergy2 malware in our grids in the 2014 time frame, possibly even earlier. Moreover, I expect that malware is still in our grids. If Defend Forward can proactively prevent malware infections, that would be very valuable.

As an aside, many of the Finance participants had data centers under their control. However, they were unaware of the control system cyber incidents that have damaged data center equipment – https://www.controlglobal.com/blogs/unfettered/data-centers-have-been-damaged-and-they-are-not-being-adequately-cyber-secured. 

Lessons learned.

In summary, the War Games suggested the following lessons learned:

- Mutual assistance doesn’t work for cyber attacks – don’t count on it.

- Government, including National Guard and the National Labs, may not have enough trained control system cyber security personnel to support each utility affected – you’re on your own.

- Malware isn’t the only way to “turn the lights off” - denial-of-service or physics issues can do the same or worse, and they may not be detected by OT network monitoring.

- Attribution may be difficult – critical decisions were based on attribution that may not have been correct.

- Operations expertise is needed to determine electric system risk - it wasn’t readily available.

- Isolating the OT networks from the IT networks and the Internet is still not being followed – this has to change or the OT networks will be hit.

- Control system devices (e.g, process sensors, actuators, drives, etc.) are not in scope for NERC CIP or NERC Supply Chain but are critical for keeping lights on, as demonstrated by the GSE declaration – the NERC CIPs and Supply Chain need to include these devices which means expand the Electronic Security Perimeter to include these devices.

- There is a significant gap between what utilities have actually done when malware has been identified such as Black Energy2 and what they did in the war games – what can we realistically expect.

- When you conduct an exercise, design a scenario to help determine whether you’ve involved the right people in continuity of operations and incident response planning and execution. 

Control systems run the critical systems on which energy and finance (and other sectors) directly rely. Yet it’s easy to overlook the extent of cyber insecurity of these critical systems.

Notes from the real world.

While we were playing out the scenario at the War College, two real-world events occurred that are worth considering in the light of our experiences during the game.

The lack of cyber security in process sensors was new to most attendees, as were the potential effects of problems with such sensors.  Yet in the middle of July New York City experienced a power outage attributable to a failure at that level. According to Con Ed’s disclosure of July 29, 2019, the July 13, 2019 New York outage was caused by a flawed connection between the substation sensors and protective relays. There was no malware, but Con Ed sustained what amounted to an effective denial-of-service that cut power to 73,000 customers. This wasn’t malware (and the Aurora vulnerability, which can shut the grid down for months, isn’t malware either). The war game didn’t include this sort of failure in its scenario, but it’s well worth considering it for inclusion in future exercises.

And at the other end of the power delivery system, ransomware--and that is malware--affected business systems of the electrical utility serving Johannesburg, South Africa. This was widely reported as a grid takedown. It wasn’t, but for many customers it had the same effect. With the utility’s business systems unavailable, customers were unable to get their prepaid utilities (a common way of buying power in South Africa) and their lights went out. So the grid itself wasn’t “hacked,” but from the customers’ point of view, they experienced a denial-of-service condition.

The grid is a system, and it’s vulnerable from the sensor to the customer interface.

Joe Weiss