As best as I can tell, there is no formal definition of a vulnerability assessment. Many IT security practitioners equate a penetration test to a vulnerability assessment. As an example, in April 2011, CPNI (UK) and DHS (US) published "Cyber Security Assessments of Industrial Control Systems - A Good Practice Guide" (http://www.cpni.gov.uk/documents/publications/2011/2011apr28-infosec-cyber_security_assessments_of_ics_gpg.pdf). The document is a comprehensive guide for
performing penetration testing of ICSs. This implies that performing a penetration test constitutes a comprehensive cyber security assessment.
During today's (1/12/12) ISA99 Leadership call, a participant from a manufacturing company stated that their IT organization equated penetration testing to vulnerability assessments and consequently is requiring penetration testing for all networks including legacy control system networks. Operations trying to convince IT that penetration testing can have detrimental impacts on legacy control systems fell on deaf ears. Many utilities and NERC CIP auditors use the verbiage in NERC CIP-007 R8 to imply penetration testing is required. Utilizing NESSUS or NMAP for legacy control system networks can, and has, caused unintentional consequences to legacy control system devices.
Control system cyber security experts from major oil, chemical, and manufacturing companies and ICS vendors have expressed concern and provided examples of unintended impacts on control system networks and devices from penetration testing and network mapping. Penetration testing has shutdown PLCs, variable frequency drives, and other legacy control system devices. In at least one case, NMAP has actually impacted the firmware of legacy field devices.
Penetration testing is to identify IT network vulnerabilities. However, from my experience, there are attack vectors that do not require Internet connections or Windows interfaces. Additionally, there are numerous non-IP cyber vulnerable communications that are not addressed by a
penetration test. A penetration test would not have identified the cyber vulnerabilities in the 2006 Browns Ferry Nuclear Plant broadcast storm, the 2008 Hatch Nuclear Plant cyber incident, the 2008 Florida Outage, the 2009 DC Metro train crash, or the 2010 San Bruno natural gas pipeline failure. Moreover, it is not clear that a penetration test would identify vulnerabilities
leading to a Stuxnet-type or Aurora attack.
There needs to be an overall vulnerability assessment program that is appropriate for the control system domain. I would also suggest that the preparation of a comprehensive definition of vulnerability assessment in an ICS environment be part of the ISA99 work plan.
Joe Weiss
