Is penetration testing sufficient to constitute a comprehensive cyber vulnerability assessment?

April 2011, CPNI (UK) and DHS (US) published “Cyber Security Assessments of Industrial Control Systems – A Good Practice Guide” (  The document is a comprehensive guide for performing penetration testing of ICSs. This implies that performing a penetration test constitutes a comprehensive cyber security assessment. This may be true in the IT space, but it certainly is not in the ICS space. From my experience, there are attack vectors that do not require Internet connections or Windows interfaces. Additionally, there are numerous non-IP cyber vulnerable communications that are not addressed by a penetration test. A penetration test would not have identified the cyber vulnerabilities in the 2006 Browns Ferry Nuclear Plant broadcast storm, the 2008 Hatch Nuclear Plant cyber incident, the 2008 Florida Outage, the 2009 DC Metro train crash, or the 2010 San Bruno natural gas pipeline failure. Moreover, it is not clear that a penetration test would identify a Stuxnet-type attack or an Aurora attack.

Shouldn’t the CPNI report be modified to state that penetration testing is part of an overall cyber security assessment program?

Joe Weiss
Show Comments
Hide Comments

Join the discussion

We welcome your thoughtful comments.
All comments will display your user name.

Want to participate in the discussion?

Register for free

Log in for complete access.


  • <p>Hi Joe,</p> <p>The answer to your question in the post title is No. In fact it rarily is the most efficient path to risk reduction until you have a mature security program.</p> <p>I think there are a few problems with the body of your post and you need to reread the post.</p> <p>1. That CPNI document does not say penetration tests are sufficient. On page 5 they list that as one type of assessment, and it is not the focus of the rest of the document. They focus on informed assessments.</p> <p>2. The document methodology is not restricted to IP. Page 13 gives a good overview of the methodology. It includes ICS security experts on the team. It includes online testing, interview, configuration review, ... the full suite and not restricted to IP.</p> <p>They even give an example, "Attack the FEP from the field equipment side (manipulate the RTU or PLC connection)".</p> <p>3. The pen test / vuln or security assessment decision is not an area where IT and Ops differs. Pen tests are a tool, but not a replacement for a security assessment in IT. There have been hundreds of articles, podcasts, discussions on when each is appropriate on an IT network.</p> <p>Dale Peterson</p> <p>Digital Bond</p> <p> </p> <p> </p> <p> </p> <p> </p>


  • <p> Here's what one of my über-gurus has to say on penetration testing: </p> <p> "the basic premise of penetration testing is that you’ve got something that you don’t understand and you’re trying to achieve an understanding of it by having some outsider— who also doesn’t understand it— attack it, simulating someone who doesn’t understand it, trying to figure it out. Now if that’s not the dumbest thing you’ve ever heard of, I don’t know what is." </p> <p>The über-guru is Marcus Ranum. My understanding is that Marcus is not familiar with control systems. Nevertheless, his insights are so fundamental that they apply to ICS security 100%. Now if that isn't something, I don't know what it is. </p> <p> Read a true classic here: <a href=""></a> </p>


RSS feed for comments on this page | RSS feed for all comments