Security by obscurity, vendor disclosure, NERC requirements, etc – what a mess

May 5, 2011
Several months ago I was approached by an IT device tester interested in penetration testing control system devices. I arranged a joint program with a utility and the device tester to test several typical substation devices. The devices the utility sent to the vendor ostensibly were from vendors that had secure devices – there were no vulnerability disclosures. The quid pro quo was the utility would get the results of the testing which could help to educate their personnel about control system vulnerabilities while the device tester would have a marketing opportunity.
Several months ago I was approached by an IT device tester interested in penetration testing control system devices. I arranged a joint program with a utility and the device tester to test several typical substation devices. The devices the utility sent to the vendor ostensibly were from vendors that had secure devices – there were no vulnerability disclosures. The quid pro quo was the utility would get the results of the testing which could help to educate their personnel about control system vulnerabilities while the device tester would have a marketing opportunity. Industry would then have a chance to see what an IT device tester with no knowledge of control systems could do. 
In preparation for the September ACS Control System Cyber Security Conference, I sent a note to the device tester asking them to give a presentation at the Conference. The response from the device tester was: “We found some serious problems with the boxes” The device was a typical control system device used in electric, water, and natural gas substations and pumping stations. The IT device tester found major cyber vulnerabilities in the VxWorks operating system (not Windows) the first day! The device tester could find no vendor notifications about this vulnerability even though this vulnerability was demonstrated at a hacker conference last July.  
Complicating all of this was the fear of the device tester being sued by the vendor or being labeled an extortionist. This held up getting the results to the utility. The intent now is to have the utility deliver the report to the vendor.
VxWorks is arguably the most popular proprietary real time operating system used in industrial applications. Consequently, this vulnerability potentially affects not only that vendor but many others.
These types of devices are important for reliability and safety. Yet these devices are not required to be tested by the NERC CIPs. How can NERC be testifying to the Senate that things are OK when an IT device tester who knows that nothing about control systems can penetrate these critical field devices within a day? What other critical infrastructures are also at risk?
Joe Weiss