Tomorrow, Kurt Yeager, the ex CEO of EPRI and current Executive Director of the non-profit Galvin Electricity Initiative will provide one of the keynotes at Connectivity Week in Santa Clara. The Galvin Electricity Initiative is focused on transforming the reliability and value of U.S. electricity service and is building Perfect Power — a system that will not fail — starting in Illinois and New Mexico.

According to the Galvin Electricity Initiative, the electric power system is insecure and susceptible to even relatively unsophisticated cyber attacks. The Illinois Institute of Technology (IIT) is one of the showcase demonstrations of the “Perfect Power” approach. The October 15th Final Report on the IIT project (http://www.galvinpower.org/files/IIT_Perfect_Power_Prototype.pdf) includes detailed Failure Modes and Affects Analyses (FMEAs). I am familiar with that approach having performed FMEAs on nuclear power plant instrumentation and control system while at EPRI. I am concerned that cyber security has not been adequately addressed within the FMEAs – the closest was:

Communications:  Router failures were the only “cyber” failure identified; the impact of a router failure was deemed Moderate; it was viewed as Uncommon; and Redundancy was the solution.

To me, this is incomplete and inadequate. The Galvin Initiative is part of the “Smart Grid” initiatives meaning the power system has been “opened up” exposing multiple cyber vulnerabilities. There already have been many control system cyber incidents in utility applications including several at university power facilities. As a demonstration, I certainly would have expected more.

Joe Weiss

There will be a hearing May 21, 2:00 PM, 311 Cannon Building in Washington DC on “Implications of Cyber Vulnerabilities on the Resiliency and Security of the Electric Grid”.
Expected witnesses:

Joseph Kelliher, Chairman, Federal Energy Regulatory Commission (FERC)
Richard Sergel, Chief Executive Officer, North American Electric Reliability Corporation (NERC)
Greg Wilshusen, Director, Information Security Issues, Government Accountability Office (GAO)
Bill McCollum, Chief Operating Officer, Tennessee Valley Authority (TVA)

I suspect there will be an update from FERC about the status of the mitigation measures for the Aurora vulnerability.  If anybody has attended any conferences lately, it sounds like there was a broad range of compliance with the recommended mitigation measures.

The Committee asked FERC several months ago whether it needed any additional authority under the Federal Power Act… does Chairman Kelliher’s participation suggest that something big is coming?

Will the Committee follow up with NERC on their efforts to conduct a survey?
 
I hope there will be a discussion of the difference between NIST and NERC CIPs – with a TVA witness, it sounds like we could hear something on that.

Joe Weiss
 
 

At ZDNet, Michael Krigsman’s blog today talks about an FBI presentation posted on AboveTopSecret.com.

He abstracts some slides from the original posting…but the original posting appears to be the entire Powerpoint presentation, including naming names of counterfeiters and clearly showing how the counterfeit Cisco routers entered the US Government procurement cycle.

The most interesting part is the story of how they were used in a rehab of a weather data installation, which promptly failed on startup.

Coupling this with the discussions lately about chips designed with fatal flaw back doors, and you might begin to get really paranoid about making sure you have the right components in your network. You can have all the cyber security you want, but if the physical devices of the network itself are compromised, look out!

Control System Cyber Security and Auditors

I just returned from a presenting a short course on control system cyber security at the spring meeting of the LA Section of ISACA – the Information Systems Audit and Controls Association. ISACA represents IT auditors. Often, they will be the ones performing control system audits (including NERC CIP), at least internally. With the exception of the person that invited me to speak, the rest of the attendees had never heard the story before. They only knew about IT metrics. My suggestion to them was NIST SP800-53 as I know of no other approach I could recommend.

IT audit staff have the ears of senior management and the Board of Directors. I believe the IT audit staff can be an asset in securing control systems if approached in a teaming manner.

Joe Weiss

I just returned from participating on a panel session at Electric Power 2008 in Baltimore. Electric Power 2008 is focused on electric power generation (not transmission and distribution). Consequently, it was fascinating to hear what the generation attendees felt about security and the NERC CIPs as well as to see what the next generation of power generation technologies would look like with respect to cyber.

I thought there were three important points made during the panel session:
- Cyber is real and needs to be addressed. One utility experienced three cyber-related plant trips that resulted in significant costs
- According to a retired security manager at NSA, the NERC CIPs are not adequate and simply trying to meet compliance and not actual security is not acceptable to protect the critical infrastructure
- One of the attendees noted that his plants are not considered critical cyber assets so he was at a loss at what could he do.

Some generation managers considered NERC CIP compliance a “game” to remove assets from CIP-002 without realizing they were shooting themselves in the foot by not addressing the reliability threat. Specifically, at a meeting of plant managers, one manager of a very large coal-fired power plant was charged to ensure his plant was not considered a critical cyber asset. Another plant manager whose plant had black start capability and therefore deemed a critical cyber asset by CIP-002 considered it cost-effective to remove its black start capability.  In both cases, the plant managers didn’t consider the potential cyber threat to reliability. They only thought about the cost of NERC CIP compliance, and possible fines, if their facilities were considered critical cyber assets. This same thought process occurs with transmission managers as they unplug their IP connections thinking that will exclude them from the NERC CIPs. This approach does exclude them from the NERC CIPs as currently written. However, it also eliminates the productivity improvements IP was implemented to bring as well as maintains the potential cyber vulnerabilities of serial and other non-IP connections. This thought process of generation and transmission managers defeats the intent of the need to secure the critical infrastructure.

Informal discussions with two DCS suppliers found they felt they were secure. However, in one case, a recent factory acceptance test (FAT) had no testing for security. In a second instance with another vendor, the vendor claimed his system was secure and the utility agreed. However, when I contacted the utility engineer, he said the vendor was not addressing specific vulnerabilities. Seems like a disconnect doesn’t it?

Finally, there is a great need for senior management buy-in that security is important for reliability and the bottom line, not for the sake of compliance. We are hoping to find a few senior executives willing to carry that message.
Joe Weiss

Training the Bad Guys

Dale Peterson’s April 22nd blog had the following: “Jason Larsen’s presentation on SCADA and Control System hacking from Blackhat Federal 08 is now available.”

There has been a prevailing view that control systems are secure because they are so arcane and obscure. However, the area of “SCADA Security” is making its way into the mainstream community, and worse, the hacking community.

As long as four years ago, presentations were being made at “Black Hat” (hacker) conferences on “SCADA security”. Some of these presentations may not have been technically accurate, but they have spurred interest in the subject by individuals we would rather not be involved. In fact, about three years ago, SAIC gave a presentation at a Black Hat Conference on how to hack control systems. What made this presentation unique and scary was that it provided bit-by-bit instructions on how to hack specific control system protocols.

I personally have worked with Jason on vulnerability assessments when he was at INL and with the initial control system cyber attack demonstration at INL in 2004. Jason is very knowledgeable. 

Consequently, when I found out he had given a presentation at Black Hat, I was extremely concerned. What’s more, Jason is scheduled to give two training classes on hacking PLCs at Black Hat in Las Vegas in August.

When I asked him why, his answer was they were interested.

Of course they are. Wouldn’t a bank robber want to know how to rob a bank? But just because they are interested is not an excuse to train the “bad guys.” 

You can’t legislate ethics, but common sense should prevail.

Joe Weiss

Dueling Databases

In Fridays’ edition of DigitalBond’s blog, databases are mentioned.

“Dueling Incident Databases. Joe Weiss has his personal incident database. Wurldtech recently announced their new Delphi vulnerability database. Now Automation World reports that Eric Byres will be resurrecting the BCIT Industrial Security Incident Database thanks to some new funding source.”

I will not go into the issues of why there are multiple databases or how best to pool resources. However, I will provide my thoughts on the need for incident databases.

- I strongly believe one of the biggest reasons why so little is being done to secure industrial control systems is the lack of perceived need since there are so few cases that have been reported (Australia and Davis-Besse).

Consequently, control system cyber security is treated as the second coming of Y2K – nothing happened, it was simply a ruse for consultants and vendors to make money.

There needs to be a business case and the only way I know of making a business case is to show that it is real and has had significant economic impact.

- I strongly believe most people still view cyber security in the traditional IT form- “the 12-year old pimply-faced hacker concocting new Microsoft viruses or worms.” 

Control system cyber incidents are much more prevalent that many believe and most are unintentional. They are simply not recognized as cyber incidents. There needs to be a better definition of “cyber” that reflects what can happen to control systems.

- Existing IT databases and reporting do not address control systems.

Consequently, several years ago, DOE tasked CERT/CC and KEMA (myself and Bob Webb) to perform a scoping study for establishing a CERT for Control Systems.

It was recognized that private industry would not provide input to the government and consequently, it was strongly recommended that a CERT/CC, not US CERT be augmented with control system expertise.

This was not followed and consequently there is very little reporting of control system incidents.

- I strongly believe you can’t develop solutions if you don’t know the problems you are trying to solve.

Since there are so few publicly identified cases, most “solutions” for control systems are based on IT problems, not control systems. Specifically, they do not address legacy/field devices.

In fact, many of the solutions being touted as fixes can, and have, actually exacerbated control system reliability. Additionally, you can’t have “best practices” when you don’t know the problems you are trying to prevent.

- Following 9/11, there was a march to “connect the dots”. That is, with all of the disparate information, why couldn’t we tell what was going to happen.

Without an incident database and experts to review the incidents, it is not possible to determine if there are patterns occurring.
It became evident to me as Marshall Abrams of MITRE and I worked on the Bellingham pipe break that there were other control system cyber incidents of a similar nature.

Additionally, when I was at a process control systems users group meeting last week, I mentioned the Browns Ferry broadcast storm. That brought a reply from one of the attendees that he had a similar that resulted in a local blackout. I believe there are underlying patterns but they have not been adequately researched.
 
- The risk equation is frequency times consequence. Today, it is not possible to prudently assign values to either. An incident database can help.

Joe Weiss

Applied Control Solutions, LLC
For Immediate Release
Contact: Joe Weiss
(408) 253-7934 or joe.weiss@realtimeacs.com.
Cyber Security Conference Focusing on Potential Causes, Prevention of Recent Power Blackouts and Plant Shutdowns (Trips)
August 4-7, 2008 – Burr Ridge, IL

Applied Control Solutions, LLC announces the eighth in a series of conferences focused on cyber vulnerabilities of industrial control systems, August 4-7, 2008, in the southwest Chicago suburb of Burr Ridge, IL. This conference is sponsored by Control magazine, and www.controlglobal.com, where the Joe Weiss Unfettered blog is hosted.

Due to recent cyber-related blackouts and plant trips, along with the five year anniversary of the Northeast Blackout, potential cyber-related incidents and their prevention are now the main focus of the Conference agenda. The focus of cyber security has been on traditional cyber security including passwords, firewalls, and compliance, not system reliability. Reliability of industrial facilities (power plants, substations, chemical plants, refineries, water systems, pipelines, etc.) has focused on control system challenges, not cyber vulnerabilities. This Conference addresses the intersection of control system vulnerabilities and reliability of industrial control systems and processes.

Presentations and focus of the Conference include:

• The recent Florida cascading outage shines a bright spotlight on cyber security of relays, switches and other remotely accessible field devices. A session will be devoted to inherent cyber vulnerabilities of these devices, lack of appropriate logging, and the associated IEEE standards efforts to protect these devices.

• A recent nuclear plant automatic shutdown, resulting from a software change, brings up new questions concerning the unintended consequences of workstation and PLC reboots. With IT security, NERC CIP, NEI-0404, and other regulation pushing to expeditiously patch control system workstations, attendees will discuss if the proposed “cures” are “worse than the disease,” along with broad implications and potential solutions. Possible explanations for previously unexplained “trips” in fossil, chemical, and other process plants will also be explored.

• End-users will discuss impacts and issues unique to the application of firewalls for control system networks.

• IT practices that have impacted control systems will be discussed, including Microsoft’s recent disclosure about Excel calculation errors, unintended consequences of patching control system workstations, and scanning of control system networks.

• Case histories of control system cyber events, control system cyber security forensics (or lack thereof), demonstrated control system cyber security technologies, control system cyber security R&D, and status of government efforts on control system cyber security will be explored. This will include cyber security regulations and best practices for nuclear power plants.

• Control system hacking demonstrations will be conducted. There will be demonstrations of hacking control systems, using actual control system devices, not emulations. One demonstration will be the hack of a typical process controls safety system. The attack will traverse a firewall, causing a fault in both a typical controller and safety system without any indication at the HMI (operator displays) until it is too late (i.e., the process under control fails in a non-fail safe condition).

This Conference has application for utility and other industrial end-users, regulatory, university, and business professionals responsible for, or dealing with, control system security, IT security, and control system operations and maintenance.

Applied Control Solutions, LLC has more than 35 years of experience in developing, implementing, maintaining, and securing industrial control systems for multiple industries. Additionally, Applied Control Solutions personnel have supported government and university efforts in modernizing and securing control systems and providing training on how to secure and optimize these systems.

The Conference will be held at Marriott Burr Ridge, Burr Ridge, IL. Cost to attend is $800 for US government and university personnel and $1495 for others.

For further information, contact Joe Weiss at joe.weiss@realtimeacs.com or see www.realtimeacs.com 
 

I had a meeting Wednesday morning with an IEEE standards committee on cyber security of substation devices. Following that, Marshall Abrams from MITRE and I gave a presentation at RSA, which is billed as the world’s largest cyber security conference. I then gave a presentation at a major control system users’ group meeting. There were several other presentations at RSA on the subject of “SCADA security.” In one of the panel sessions, there was a discussion about media hype and how it is hurting the process by jading management. Following that concern, a presentation was made about how easy it was to hack the grid. It certainly succeeded in getting media hype on an approach that is dubious at best in terms of doing any damage to control systems.

As to the three meetings I attended, the reactions at all three were remarkably similar. To start with, there was a lack of appreciation of how real the problem really was. There was also a lack of understanding by the IT community of the uniquenesses of these systems and why solutions need to be tailored to these systems. More importantly, the “light started going on” with several knowledgeable control system engineers as what was actually meant by the term “cyber incident.” Once it was explained that a cyber incident means an impact on confidentiality, integrity or availability, and not just an intentional attack, several people came forward to say they had experienced problems (cyber incidents) resulting in system downtime in substations, power plants and chemical plants.

My database is increasing and the need for discussions on preventing these types of events is growing more urgent. Consequently, there will be significant discussions on actual cases at the August Cyber Security Conference in Chicago.

The following report by Ryan Singel appeared at Wired.com yesterday.

April 09, 2008
 On June 10, 1999, a 16-inch diameter steel pipeline operated by the now-defunct Olympic Pipeline Co. ruptured near Bellingham, Washington, flooding two local creeks with 237,000 gallons of gasoline. The gas ignited into a mile-and-a-half river of fire that claimed the lives of two 10-year-old boys and an 18-year-old man, and injured eight others.

Wednesday, computer-security experts who recently re-examined the Bellingham incident called its victims the first verified human casualities of a control-system computer incident.

For the complete story, click here. 

We’ve known all along something like this was bound to happen, but sometimes you hate to be right.