The NERC CIP's are not making the grid more secure or reliable

The North American Electric Corporation (NERC) Critical Infrastructure Protection (CIP) cyber security standards were developed to increase the cyber security and reliability of the electric grid. Unfortunately, they are not doing either.

The NERC CIP’s were developed by the electric industry with industry-developed exclusions. Because the scope was the BULK electric grid and the associated exclusions, the scope of cyber security of the electric grid is quite limited.  Electric distribution is excluded (majority of Smart Grid falls under this exclusion), serial (non-routable protocols often using serial-network converters) communications are excluded, telecommunications is excluded, the “brightline” criteria exclude smaller facilities (the brightline criteria establishes minimum levels for facilities to be considered critical), etc. Additionally, the exclusions in the NERC CIPs provide a road map to attackers as they identify what is in-scope, and just as important, what is out-of-scope and consequently not addressed.  

The NERC CIP audit methodology is a very onerous and expensive process (independent of the potential fines for failing audits). This has resulted in many utilities manipulating the NERC CIP process to minimize the number of devices and facilities to be addressed. Marlene Ladendorff is a cyber security professional who developed the cyber security program at a nuclear utility and is now with the Idaho National Laboratory. Her doctoral thesis was “The Effect of North American Electric Reliability Corporation Critical Infrastructure Protection Standards on Bulk Electric System Reliability” (http://pqdtopen.proquest.com/doc/1619581598.html?FMT=AI). The strongest theme from her thesis was “Entities Removing Equipment to Avoid CIPs”. That is, removing programmable digital devices and replacing them with the older serial devices in order to avoid being classified as Critical Infrastructure or Critical Cyber Assets (CCA) and necessitate inclusion in a NERC CIP audit. Yet the same systems being removed from transmission systems to avoid the NERC CIPs are being installed for Smart Grid applications which are outside the NERC CIP scope. What does this really mean to Smart Grid security?

From Marlene’s thesis, the following examples were provided:

-        Participant 2 in her study found that a company had the most sophisticated network protection he had seen. However, NERC staff reviewed their architecture and wanted them to tear it out. It took the company 6 months to convince NERC that this was the best protection they could do for the control systems the company was operating.

-        Participant 3 outlined a situation where an exercise was cancelled by their compliance group, citing potential non-compliance issues with one of the CIP standards as the reason. The logic behind the compliance groups’ action was that if a potential weakness was found, it may need to be reported and the entity risked receiving a fine from NERC. Participant 3 questioned the compliance group about their decision, stating that it was impossible to discover and fix weaknesses if exercising and testing was not allowed to find those weaknesses. The compliance group continued to refuse the testing, resulting in a catch-22 situation.

-        Participants 5, 6, 10, and 11 experienced situations where “some of the transmission owners….are gaming the system in order to prevent the application of the CIP standards.” To accomplish this, some companies modified their networks to avoid compliance issues with CIP-003 through CIP-009.

-        With the expense involved in compliance with the CIP standards, Participant 10 pointed out that “organizations worked very hard to not have or have very little…assets that they had to protect”, assets that would fall into scope of the CIP standards. Some entities were trying so hard to keep equipment out of scope that they spent money to “rip out fiber and CAT-5 [networking cable] and replaced it with serial [cable] to get away from routable protocols” that would have brought networks into the compliance scope. Entities calculated that it would be cheaper to replace fiber and CAT-5 network cable with serial cable in order to remove equipment from the CIPs scope. Doing so eliminated the requirement to comply with CIP standards for those networks and equipment.

-        Participant 11 witnessed situations in more than a few utilities where remote access implementations were converted back to serial communications in order to reduce the amount of equipment requiring CIP compliance. Participant 7 echoed the comment from Participant 6, stating that entities took some networking hardware out and replaced it with “serial communications, only trying to skirt CIP compliance. Every entity I know plays the game that way.”

-        Some utilities are making a cost-benefit decision on providing security versus paying fines. Depending on the cost of the fine compared to the cost to install NERC CIP compliance, some utilities have made the decision to pay the fine rather than make the security improvement. What does this mean for cyber security of the electric grid?

The following are technical issues that are being propagated by the NERC CIP process:

-        Some substations employ nearly-automatic protective relay systems. These systems can sense when breakers re-close due to commands from the EMS/SCADA system, without ever receiving potentially compromised commands from the SCADA system directly.  Many utilities would like to keep the relay systems inaccessible from remote access, as they do not need to be connected and any such connection increases cyber risk. However, NERC CIP demands that passwords on protective relays change periodically. This means that utilities with hundreds to thousands of substations will most likely connect their protective systems to external networks (usually over the Internet) to support a compliance requirement that can actually compromise security. Which is the greater risk - that someone will physically break into a substation and try to guess an old password on the relay or that someone will try to hack the substation remotely?

-         Since the NERC CIP guidance requires anti-malware and anti-virus protection, some utilities are mandating protective relays to have malware protection even though adding this function will reduce the effectiveness and function of the relay. In some utilities, the security organizations are overruling the technical organizations to meet NERC CIP requirements. Are you really more secure or reliable if the protective relays don’t work?

-        Another example of the inconsistency of the NERC CIP guidance is that when it comes to grid reliability is the use of “black start” facilities. Black Start facilities are those necessary to restart the grid after a complete grid outage. This function is considered critical by grid planning and operations organizations as well as organizations within NERC. During the review of the NERC CIP Revision 5 process, ISO New England raised a concern that adopting a new requirement for specific controls for Low Impact assets could have unintended consequences, such as the withdrawal of black start resources. This would make the grid less reliable.

Some of the security hardware can affect control system performance. A NERC report identified that a device locking tool used to meet NERC CIP requirements caused a disturbance that resulted in the loss of SCADA services. This is obviously making the grid less reliable and secure.

Perhaps the most important point is there have already been four major cyber-related electric outages in the US (more than 90,000 customers). If the NERC CIPs were fully implemented, they would not have prevented any of these outages. What does that say about the efficacy of the CIP’s when the NERC CIPs do not address previous cyber-related outages, attacks such as Stuxnet, or vulnerabilities such as Aurora?

These examples clearly demonstrate the NERC CIP approach is not adequately securing the grid or even maintaining existing grid reliability. In the end, utilities need to have the freedom to implement the proper infrastructure and cyber security appropriate to maintaining a reliable system without the fear of legal exposure or examination by those who are not familiar with the utilities system operations. Who in the utility industry is willing to stand up and state “the emperor wears no clothes”?

Joe Weiss

Show Comments
Hide Comments

Join the discussion

We welcome your thoughtful comments.
All comments will display your user name.

Want to participate in the discussion?

Register for free

Log in for complete access.

Comments

  • <p>Hi Joe - this very solid post continues and extends, via the findings presented in Marlene Ladendorff's postdoc thesis, a comprehensive critique of the CIPS. Nothing better exemplifies the notion of unintended consequences of regulation like utilities swapping out deployed systems to reduce their compliance exposure. </p> <p>My QQ for you and others making the case for terminating the CIPs is: Then what? National security leaders can't and won't change course on this policy without a ready replacement strategy that demonstrably improves upon the status quo. Do you have something in mind?</p> <p>Good to be back in touch with you. ab</p>

    Reply

  • <p>Great article Joe by a person who has been there done that for years. I will be speaking in part on this very subject this week at the 4th Annual Smart Grid Cyber Security Virtual Summit 2015 (<a href="http://www.smartgridobserver.com/agenda-csvs12215.htm">http://www.smartgridobserver.com/agenda-csvs12215.htm</a>). In my session I will be speaking on a new programming language (5GL) that will give software intelligence and be able to react in microseconds. For this new and available for license patented software to work it needs very complete knowledge of the control system processes, which we call Digital Process Management (DPM). If there is anything we got from NERC/CIP was at least the road map of the processes in the power grid. </p> <p>I completely agree that compliance does not mean security. Cybersecurity is a technology problem that can only be addressed by technology. When machine actions are in milliseconds ( see my article: <a href="http://www.govtech.com/dc/articles/DARPA-Director-Calls-for-Cybersecurity-Change.html">http://www.govtech.com/dc/articles/DARPA-Director-Calls-for-Cybersecurity-Change.html</a> ) your cyber cybersecurity technology must be able to react real time during data in motion in that microseconds window to be effective. </p> <p>People can't think or work in microseconds but technology can. So throwing people and compliance at cybersecurity, with the acceptation of a complete review and knowledge of your control system processes, will have little effect in securing the power grid. Cybersecurity is a technology problem and can only be addressed by knowing your processes and using technology to authenticate, view, audit, analyse and block anomalies real time in microseconds. My next article, Will "5GL DPM Save Cybersecurity" and will be published tomorrow in Government Technology Magazine. It discusses this new technology and its need for immediate deployment. </p> <p>Good subject and don't let up on this. We need to be secure not just compliant. </p>

    Reply

  • <p>I totally disagree. There is indeed a difference in being compliant vs being secure. I know some utilities focus more on compliance than on security but I also know that before CIP there were instances where control systems were tied to the internet, had no malware prevention, and kept the default passwords. It's a long, slow and arduous process, but by complying with even the smallest of CIP measures a system becomes more secure.</p>

    Reply

  • <p>I respect you, Joe Weiss. But you are naive if you believe what you write about NERC CIP. I "visited" 56 entities and, while I'll grant you EVERYONE games the system to their advantage, all 56 were WAY, WAY, WAY (continued off the page) more secure after NERC CIP "compelled" them to allocate resources and take the threat seriously (even the threat of noncompliance if they weren't true believers). Nobody believes the standards are great, but, how can someone with a giant brain like the one you sport really, honestly, say they're "...not making the grid more secure and reliable."?????</p> <p>The "grid" is a thousand times more secure than what I observed in the mid-2000s when almost nothing was in place, and people surfed the web from HMI's like Greg stated. And I say this with conviction, even though I saw ridiculous things like Ms. Ladendorff points out (how about the removal of a real-time clock during a CIP audit to try and avoid a finding, since they forgot to file a waiver?)</p> <p>One mistake you make is to lump all "grid insecurity" into a meta-domain. The intended goal of NERC CIP is to lower risks to the BES, plain and simple. This scope does not worry about distribution since distribution does not typically threaten the BES (besides potentially far fetched scenarios). Not to say distribution outages don't hurt, or even cost lives... but they are not the BES and thus out of scope. (that said, I pray that future standards will expand the scope...)</p> <p>I've ranted enough for now, besides one respectful jab at Larry: cool sounding technology, but if you honestly think "Cybersecurity is a technology problem", you don't really understand cybersecurity. Give me the greatest and most secure technology in the world and I'll draw up a dozen scenarios where it can be abused, misused, atrophied, and faulty-processed into blatant insecurity. Cybersecurity is a multi-faceted problem that begins long before technology is in question, at the earliest stages of problem definition. It is first and foremost an attitude problem... if I misunderstood your post, my apologies.</p> <p>And I apologize to you, Joe, for totally cracking on you. I respect you and I agree with so much of what you post, but you need to put your energies into something more productive... you could really make so much more impact. For instance, I LOVE your emphasis on the control layer! Many people try to say this but you put it best (and probably you said it first). </p>

    Reply

RSS feed for comments on this page | RSS feed for all comments