Because what occurred is common to many ICS vendors, I will not mention the ICS vendor’s name.
Five months ago, the utility hosting the ICS cyber security test bed met with one of their major ICS vendors. The first question the ICS vendor asked was if the utility has kept current with the latest security patches. (This is a much more difficult question to answer than it appears. Patching is not only temporal -how do you know if you missed the latest, but patches may not be identified as security.) The answer was the engineering supervisor did not know. The engineering supervisor mentioned they have trust in their ICS vendor and want to maintain the equipment warrantee and consequently have implemented all ICS vendor recommendations. The ICS vendor‘s technical documentation did not identify recommendations specifically as security patches. Consequently, even though the utility was up-to-date with all patching including security, the engineering supervisor did not realize that security was included. Some of the security patches actually impacted the reliability of the ICS and the overall system. Since the engineering supervisor did not realize they were security patches, he contacted the ICS vendor’s engineering team about the impact to the reliability of the systems. The ICS vendor’s engineering team also did not realize they were security patches and consequently the message was not relayed to the ICS vendor’s security team. Thus, the fact that a security patch could, and did, harm the operation of the system came as a surprise to the ICS vendor’s security manager.
Last week, another electric utility using this vendor’s latest generation of turbine controls received a security patch for a generating unit that had been operating very reliably. Because the unit had been operating very reliably, the utility did not address resilience issues such as hard-wired trips. The utility implemented the vendor’s patch and subsequently started the turbine and brought it to full power. However, things did not operate as they should. The HMI control panel data froze and did not update. After reaching full power, the turbine was no longer able to be controlled by the operator. Apparently, the security upgrade patch did not coordinate with the PLC software resulting in loss of view and loss of control. In an internal memo, the utility identified this incident as a significant near miss but did not use the word “cyber” nor disclose the cyber aspects to their ICS supplier. It is also not clear how many other organizations were sent this patch in terms of the extent of this potential problem.
There are a number of very important generic aspects to this discussion:
- Security and reliability need to be addressed in a comprehensive technical manner
- For ICSs, all patches should be for reliability and safety with security just being one aspect
- There often is a disconnect between the utility’s Operations and Security organizations
- There often is a disconnect between the ICS vendor’s Operations and Security organizations
- Security patches are not always identified by ICS vendors as such leading to unintended consequences that are not easily identified in a root cause analysis
- Security patches that are implemented by the end-user’s IT organization without input from engineering can also cause problems that are not easily identified in a root cause analysis
- There are very few organizations willing to look closely enough to even identify the issues identified here
- Often, the ICS vendor is not notified when cyber incidents occur
- Often, the ICS vendor will not explicitly mention the term “cyber” when they provide experience reports to their customers
- Often, even organizations such as NRC and NERC will not identify incidents as cyber
Much needs to be done in changing the culture of the entire industry. This includes increasing understanding, coordination, and disclosure. There will discussion of these issues at the October ICS Cyber Security Conference (www.icscybersecurityconference.com)
