Is penetration testing sufficient to constitute a comprehensive cyber vulnerability assessment?

April 2011, CPNI (UK) and DHS (US) published “Cyber Security Assessments of Industrial Control Systems – A Good Practice Guide” (http://www.cpni.gov.uk/documents/publications/2011/2011apr28-infosec-cyber_security_assessments_of_ics_gpg.pdf).  The document is a comprehensive guide for performing penetration testing of ICSs. This implies that performing a penetration test constitutes a comprehensive cyber security assessment. This may be true in the IT space, but it certainly is not in the ICS space. From my experience, there are attack vectors that do not require Internet connections or Windows interfaces. Additionally, there are numerous non-IP cyber vulnerable communications that are not addressed by a penetration test. A penetration test would not have identified the cyber vulnerabilities in the 2006 Browns Ferry Nuclear Plant broadcast storm, the 2008 Hatch Nuclear Plant cyber incident, the 2008 Florida Outage, the 2009 DC Metro train crash, or the 2010 San Bruno natural gas pipeline failure. Moreover, it is not clear that a penetration test would identify a Stuxnet-type attack or an Aurora attack.

Shouldn’t the CPNI report be modified to state that penetration testing is part of an overall cyber security assessment program?

Joe Weiss