Are the NERC CIPS making the grid less reliable

I have written a number of blogs about the deficiencies of the NERC CIPs and that most utilities are playing games with compliance. I continue to insist, as do many others, that compliance does not automatically equal increased security, while proven increased security should equal compliance (or there is something wrong with the compliance process). There are very few utilities, possibly only a handful, who are actually trying to secure their systems, not just play a compliance game. Consequently, this blog is not a blanket indictment of an entire industry – but pretty close. The bottom line is the infamous “million dollars per day” fine is meant to punish those facilities that can impact the reliability of the bulk electric grid. Those utilities playing games with NERC CIP-002 (which unfortunately are most of the utilities in this country) in order not to secure their facilities should be fined the million dollars per day per facility as they can affect the reliability of the bulk electric grid. The problem is real as industry has experienced numerous power plant cyber incidents as well as several large cyber-related outages. This is serious business, and we shouldn’t be playing games. What about Wall Street? What about the insurance companies? Shouldn’t investors and actuaries be concerned about these utilities’ attitude to risk management? Ostrich behavior isn’t risk management.

Let’s talk specifics:

- The NERC CIP Drafting Team, and many utilities are still fighting removal of CIP-002 and its unconscionable exclusions.

- Some utilities are removing IP connections in order to be excluded from the NERC CIPs. Many utilities have received rate increases to improve grid reliability by installing those connections. Are the utilities refunding the money?  State PUCs take note. Many states are cash strapped and could use the money if the utilities aren’t going to do what they said they would with the rate increase proceeds. Ironically, those same utilities are using IP connections for Smart Grid because the NERC CIPs exclude electric distribution. Something is wrong here.

- Some power plants are no longer providing Black Start capabilities in order not to be considered a NERC Critical Asset. This obviously impacts the reliability of the bulk electric grid and those plants should be fined a million dollars per day.
 
- Some utilities are reclassifying their control centers as control rooms in order not to be considered a NERC Critical Asset. This obviously impacts the reliability of the bulk electric grid and those utilities should be fined a million dollars per day.

- Some utilities have refused to classify their very large power plants as NERC Critical Assets. In at least one case that I specifically am aware, this plant is so big it exceeds the minimum reserves for an entire interconnect.  This places large swaths of the US at risk for outages.  As an aside, if they don’t consider their largest power plant a NERC Critical Asset, do you think they consider any of their other power plants NERC Critical Assets? This utility should face a million dollar a day fine for each unit as well as other possible penalties. I am sure that is not the only utility playing this game with their power plants. For example, I know of another plant from another utility whose single unit is significantly bigger than any single nuclear unit and it is not a NERC Critical Asset. NERC’s Michael Assante identified this inappropriate industry behavior with respect to NERC Critical Asset identification in his April 7th 2009 letter to industry, and, frankly, not much has changed.

I have also been concerned about the technical appropriateness of the NERC CIPs with respect to control systems, specifically CIP-005 and CIP-007. The concern has been that applying these IT-type standards to control systems actually could impact their reliability. There have been numerous cases where penetration testing has impacted legacy field devices. I have found my first documented case affecting non-legacy field devices. This case is from the January-June 2009 NERC Disturbance Reports: “A disturbance resulted in the loss of the utilities Energy Management System functions, including SCADA, AGC, Network Applications and ICCP. The utilities Power System Operators notified the RC and affected entities upon loss of the EMS and when EMS functionality was restored. The disturbance was caused by the implementation of a device locking security tool. The tool caused select hard drives to become unavailable resulting in the loss of the above functionality. The tool was being implemented in response to the Critical Infrastructure Protection (CIP) standards. The disturbance remediation consisted of uninstalling the device locking tool and restarting the impacted systems. To prevent the recurrence of this incident in the future, comprehensive testing will be performed to insure that tool operating characteristics are in accordance with expectations.”  There are many questions that arise:
- If testing wasn’t done before the tool was installed, isn’t that a violation?
- How many other utilities are using this tool?  How many haven’t tested it before installing it?
- What caused the hard drives to become unavailable?
- What other commercial tools could cause this same problem?
- Can this tool cause a similar problem in Smart Grid applications?
- Can this tool cause a similar problem in nuclear plant applications?

This is the reason I do not allow vendors to present at the ACS Conference unless I know the technology they will be touting has been successfully tested in a control system environment. 

Joe Weiss

 

Show Comments
Hide Comments

Join the discussion

We welcome your thoughtful comments.
All comments will display your user name.

Want to participate in the discussion?

Register for free

Log in for complete access.

Comments

  • <p>What I have an issue with is the somewhat lackadaisical CIP requirements surrounding access control and safeguarding sensitive CCA information.</p> <p> By ‘documenting’ all facets of your Cyber-enabled environment, you are mandated to directly expose vulnerabilities.  One of our analysts determined that there are no less than 80 required ‘documents’ from CIP-002 thru CIP-009.  Some of these documents call out technical exceptions, remediations, sunset plans, etc.  The process of complying with a set of cyber security requirements has actually forced engineers and network architects to shine the light directly on vulnerabilities that were probably best kept a secret.  Where once a particular vulnerability in a SCADA control network was either an obscure unknown (i.e. security by obscurity) or known only by a select few, it is now entirely exposed in required compliance documentation.  Technical Exceptions, design documents, detailed Electronic Security Perimeter diagrams, firmware and patch revision management, build documents, etc. all directly call out vulnerabilities. </p> <p></p> <p>  </p> <p>While compliance standards were designed to specifically address cyber vulnerabilities, many are left to the organization’s interpretation of the requirement, or the budget to provide adequate compensating protection and control.  In other words, not only has the vulnerability now been documented, but also the detailed temporary remediation.  For a hacker, zeroing-in on an environment’s Achilles heel is now really just a matter of getting a hold of some documentation – no need to poke around at the command prompt or ask awkward questions. </p> <p></p> <p>  </p> <p>The CIP Standards are very light on requirements surrounding access control to sensitive information.  Yes, it must have a ‘confidential’ stamp … yes, it must be kept in a secure area … yes, the access control should be reviewed annually … yes, terminated or reassigned employees must have their access revoked in short order, etc.  But what’s really stopping people who do have access from printing it, emailing it around, taking it home, using public servers to store it, faxing it, printing it, etc.?  Considering the criticality of these ‘documents’, much more should be done. </p> <p class="MsoNormal"></p> <p class="MsoNormal"> Old-school documentation is a Word document, a PDF or a spreadsheet.  New thinking should be akin to how Mr. Jobs &amp; company managed to keep the iPhone a secret until days before the release.  Strict and enforceable compliance requirements around the role-based access of Cyber documentation should be implemented. </p> <p></p> <p> I could go on … but I won’t! J</p>

    Reply

  • <p> <i>Know thyself </i>is an essential part of secure thyself.  Yes- what is exposed with documentation must also be well protected !   CIP-003, from the first version forward, asserts an expectation for an Information Protection Program (R4) addressing CCA information.   Regardless, supporting your comments, many organizations may just be months into having a basic defined program stood up, to execute protection particulars.  These programs and control expectations will need to mature as well.       </p> <p></p> <p>Related excerpt from <a href="http://www.nerc.com/files/CIP-003-1.pdf">NERC CIP-003-1</a>:   </p> <p></p> <p><b>R4. </b>Information Protection — The Responsible Entity shall implement and document a program to identify, classify, and protect information associated with Critical Cyber Assets. </p> <p> <b>R4.1. </b>The Critical Cyber Asset information to be protected shall include, at a minimum and regardless of media type, operational procedures, lists as required in Standard CIP-002, network topology or similar diagrams, floor plans of computing centers that contain Critical Cyber Assets, equipment layouts of Critical Cyber Assets, disaster recovery plans, incident response plans, and security configuration information. </p> <p> <b>R4.2. </b>The Responsible Entity shall classify information to be protected under this program based on the sensitivity of the Critical Cyber Asset information. </p> <p> <b>R4.3. </b>The Responsible Entity shall, at least annually, assess adherence to its Critical Cyber Asset information protection program, document the assessment results, and implement an action plan to remediate deficiencies identified during the assessment. </p> <p></p> <p>   </p>

    Reply

  • <p> Regulatory expectations are ratcheting up to address gaps.  NERC CSO Assante's April 2009 letter (see <a href="http://www.digitalbond.com/index.php/2009/04/07/assante-throws-down-the-gauntlet-on-cip-002/">Assante Throws Down the Gauntlet on CIP-002 </a>- DigitalBond.com) and now NERC's follow through with DRAFT NERC CIP-002-4 (see <a href="http://www.nerc.com/filez/standards/Project_2008-06_Cyber_Security_PhaseII_Standards.html">NERC Project 706- Phase II</a>), NRC/NERC MOU, etc., should mark refreshing progress for even the most ardent critics.   </p> <p> Any industry tactics such as avoiding routable protocols, skimpy risk assessment results, etc are at best a temporary stay while forming up compliance programs and coming to grips with ongoing audit cycle results and additional changes needed to address regulatory trajectory.  Not a time to stand still or slow down regardless of whether or not today's compliance audits are going well. </p> <p> More: <a href="http://thisweekinsecurity.blogspot.com/2010/01/2010-blasts-in-with-regulatory.html">2010 Blasts in with Regulatory Cybersecurity Bar Raising<i>- NERC CIP-002-4 (Project 706 Ph II) and NRC RG 5.71- both with NIST Enhancements</i></a></p> <p></p> <p>  </p>

    Reply

RSS feed for comments on this page | RSS feed for all comments