Where are the experts?
Several weeks ago, a conference was held by the Brookings Institute and Google on plug-in electric vehicles. In addition to the topic of plug-in vehicles, there was a discussion on cyber security of the electric grid by some very important industry, media, and government individuals. The video starts with R. James Woolsey, ex CIA Director, giving full credence to the bogus National Journal article blaming Chinese hacking for the Northeast and Florida outages. As can be seen from the attached conference minutes, Woolsey interviewed a panel of energy experts. However, this panel of energy experts did not have anyone directly from the control system cyber security community. Consequently, the answers are very interesting and also very questionable.

Specific “questionable” comments are:
Thomas Friedman, pointed out that our current grid’s inefficiencies are theoretically preventing mass disruption. 
Answer: Has he heard about the Northeast Outage and the Florida Outage? Both had cyber aspects (no, it wasn’t the Chinese). Additionally, these weren’t the only large outages caused by cyber incidents. The grid’s inefficiencies are not directly related to cyber vulnerabilities.
Vaswani adds… “The truth is, we’re not that secure now, but it’s been less of concern since currently most of the grid has only a fairly limited amount of networked control.”
Answer: What grid is he referring to - the existing grid has a significant amount of networked control.
As Sue Tierney, Managing Principal at Analysis Group, pointed out, the trend toward distributed generation is making the grid harder to take down.
Answer: Automated Metering  Infratsructure (AMI) is extremely vulnerable where it connects to distribution systems. I have given several presentations at Smart Grid conferences on the cyber vulnerability of distributed generation and demand side management.

Ironically, I just had a discussion Friday with members of the ISA S99.05 Leadership Committee about the continuing need for awareness, especially for those in “Washington”.
Joe Weiss

Cleantech Terror Alert: Hacking the Grid
Written by Craig Rubens
Posted June 26th, 2008 at 12:00 am in Policy
Science fiction writers speculate that robots will eventually take over our networks, but conspiracy theorists say our current grid is under attack from foreign hackers—conspiracy theorists and high-level intelligence officials that is, according to the cover of the National Journal. The article alleges that Chinese paramilitary hackers were responsible for two massive U.S. blackouts. The theory had enough credibility for former CIA Director-turned-venture-capitalist James Woolsey to ask a panel of energy experts what is being done to secure the grid at the Google/Brookings plug-in electric vehicle conference in Washington this month. Video of Woolsey’s question and the panel’s response below.
So just how secure is our grid? Does making our grid smarter and more interoperable increase our risk? The panel’s moderator, Thomas Friedman, pointed out that our current grid’s inefficiencies are theoretically preventing mass disruption. But the shift to IP-based, smart-grid services leverages all of the security technologies that have been developed in the IP space. Still, industry sources echo the panel’s response and say Woolsey’s is an “extremely legitimate question.”
Silver Spring Networks CTO Raj Vaswani , who was not on the panel, tells us that “IP networks are decades ahead of any proprietary solutions.” But he concedes that securing the grid is different from securing Internet services. “The threat model is extremely complex because you’ve got devices sitting out in the field potentially for years with no physical security.”
Vaswani says this isn’t suddenly a new issue with the smart grid becoming a hot topic. “It’s wrong to think we’re secure now and installing these services would somehow make us less secure,” Vaswani adds. “The truth is, we’re not that secure now, but it’s been less of concern since currently most of the grid has only a fairly limited amount of networked control.”
While the Administration has never officially said China had anything to do with the blackouts, DOE representative Andy Karsner laid out what he saw as the potential of such attacks in terrifying terms:
“This isn’t the cyber-hacking that you think of just for passwords. This is the capacity to destroy hardware in your home, at airports, at military bases, your car, if its connected through the grid.”
But it’s getting better. As Sue Tierney, Managing Principal at Analysis Group, pointed out, the trend toward distributed generation is making the grid harder to take down. With more solar panels on individual roofs and cars soon plugging into the grid to push and pull electrons, the importance and vulnerability of central substations decreases. “The sky definitely isn’t falling,” Vaswani says. “And if we are doing this right, we’ll be shoring it up.”

A litany of control system cyber issues – Are we making progress?

As I was getting ready to put the draft agenda on the website for the August Control System Cyber Security Conference (it should be up by Monday at www.realtimeacs.com), many things became clearer.

Based on the events that keep happening and industry’s response, or lack thereof, I am still pessimistic about real progress.

We still have too many very smart people who appear ambivalent on the questions of: “what is a cyber incident?”, “what is unique about securing control systems?” and most importantly “why secure control systems at all?” 

The question I keep asking myself is after the incidents that have happened, the vulnerabilities that have been disclosed, and even NERC admitting they’ve lied to Congress, why is there still so little REAL progress being made?

Here are some specific concerns and the sessions that will directly address them.

We continue to have cyber incidents; many of them complete repetitions, because there is no appropriate guidance or knowledge. The recent Hatch Nuclear Plant incident was a good example. One senior nuclear plant I&C Engineer wasn’t even informed of the incident because plant management considered it to be an IT issue. The Florida Outage is another – blame the field engineer even though the SCADA operator was unaware the protection was removed. How many people know about cyber incidents other than the three or four public cases even though there have been more than 100 actual cyber incidents? What would Einstein say about repeating the same thing and expecting a different result?

- A session on the cyber issues affecting IEDs including the Florida Outage issues of remote access and lack of appropriate logging.

- A session on the implications of the Hatch incident to nuclear and non-nuclear facilities including design assumptions.

Some of the current security research harkens back to my days as an EPRI Project Manager – we have too many solutions looking for problems, rather than looking at the ACTUAL problems and developing appropriate solutions. The vast majority of the cyber incidents that have occurred in critical infrastructure have not happened because of insecure operating systems or weak passwords. Most of the research I am aware of in electric and oil/gas would not have prevented most of the events that have occurred to date.

- A session on University-sponsored research.

Safety systems (emergency safeguards in nuclear plants, burner management in fossil plant, SIS in chemical plants, etc.) originally were hard-wired analog systems completely isolated from the control network. For economic and productivity reasons, they are now being co-mingled.

There has even been an actual cyber compromise of a safety system.

Field devices and wireless can be major sources of cyber vulnerabilities for safety systems. However, they may not be considered in the oil/gas safety system efforts similar to the exclusions in the AGA-12 Gas SCADA Encryption efforts. What should industry be doing to secure safety systems? How should control and safety networks be designed?

- A session on safety and control systems including a demonstration of hacking a safety system.

The area of control system vulnerability disclosure is still a horrible quagmire with no “right place” to go. You have one cyber security researcher make a disclosure to the Associated Press. I know of another that will make a disclosure to DHS because they don’t know where else to go, including the vendor. This is one of many reasons we need a CERT for Control Systems which we currently do not have.

- A session with Carnegie-Mellon to discuss the CERT concept and extending it to control systems.

Many vendors that are competing in the SCADA/control system security space are making claims that are questionable at best often with little to no field experience:

-         “My technology works in (fill in the blank - IT, banking, finance, defense,…) and therefore will work for industrial control systems.” Until the systems are installed in the field, how do you know they won’t affect performance of the systems they are meant to protect?

-         “My technology is NERC-compliant.” There is no such thing as NERC-compliant technology. NERC compliance is the utility’s program meeting specific criteria.

-         “Our DCSs are all tested before they leave the factory.” You only get what you specify and very few companies know enough as what to specify.

-         “I have used the INL Procurement Language specification to provide vendors specific procurement requirements.” There are no two major control system implementations or retrofits (DCS, SCADA, substation, pipeline, etc) that are identical. A control system is more than just than HMI and procurement specifications need to address those also.

Consequently, there is a need for an open discussion on the differing expectations between end-users and vendors.

- A one-on-one discussion with a utility engineer whose plant is going thru a DCS retrofit and his DCS vendor to demonstrate the differences in perspectives and the need to close the gap.

Nuclear power plants are implementing digital upgrades and Ethernet LANs often with unexpected consequences (see Browns Ferry and Hatch). Some of the smaller field changes that are not required to be analyzed can, and have had cyber consequences. The nuclear industry has not formally participated in non-nuclear cyber conferences or organizations such as ISA S99. There is a great need for the nuclear industry to include the experience and expertise of the non-nuclear community.

- An open discussion with NRC, FERC, and PNNL to discuss their concerns with cyber security in nuclear plant operations.

The issue of memory leakage discussed at ISA POWID demonstrates the need for IT and Operations to better understand each others’ needs. What is a problem to one may not be problem to the other. Without better communications, these problems will continue to appear.

- An open discussion session to discuss cross-functional issues.

Joe Weiss

The major control system suppliers are claiming they provide tested secure DCS and SCADA systems.

To my knowledge, at least four major control system suppliers, in this case 3 DCS and one SCADA, are providing less security than advertized.

In one DCS case, the vendor told me how secure their system was and specifically identified one showcase utility. Unfortunately for them, I knew the utility and the utility engineer. The engineer was so disappointed in the vendor not listening to his needs he made a presentation on security deficiencies the vendor would not address.

In the second case from a different DCS vendor, the vendor recently performed factory acceptance testing without security being addressed even though I was told by the supplier that security testing is standard procedure.

In the third case from another DCS vendor, the DCS is currently being procured and staged. The vendor claims they automatically secure their systems. However, when the utility engineer questioned the vendor, the vendor stated they would need additional funding for security and even asked the utility to delay the implementation to address security.

In the SCADA case, the vendor was using the full suite of Microsoft web services without recognizing the security implications.

What is really going on with our vendors?

Joe Weiss

Observations from beautiful, hot Scottsdale – ISA POWID Symposium

ISA POWID is the instrumentation and controls symposium for fossil and nuclear power generation. On Tuesday, ISA POWID held 6 hours of security tracks.

My general observations include:
- Nuclear Energy Institute (NEI) had another scheduling conflict which precluded that organization from sending anyone to Scottsdale. That makes at least six major non-nuclear control system cyber security conferences that NEI has managed not to be able to attend.
- Several generation utilities are removing black start capabilities so as to avoid the NERC CIP standards. CIP stands for Critical Infrastructure Protection. Shouldn’t it bother people that utilities are making the critical infrastructure less reliable in order to avoid the potential fines associated with the CIP standards?
- Several utilities have excluded all fossil power plants from being critical assets. In one case, utility management gave IT the responsibility for NERC CIP compliance including plants and substations. What gives?
- During the NERC CIP discussion, it was very evident there was confusion as to what was a critical asset and how big did it have to be. Jim Batug gave a great answer that focused on the effect any cyber asset would have on power plant operation.
- There was a very interesting discussion by two DCS vendors of the effects of memory leakage on control system performance.  Memory leakage is a well-known issue in the IT community that is of little consequence. However, this is a big deal for control systems. At least to me, this seemed to be a new issue that I have not seen in “top 10” vulnerability lists.

All in all, it was a very interesting symposium with many utilities wanting to be able to address security, but not having corporate buy-in or resources.

Joe Weiss

From the press release, verbatim:

Citect reassures its customers on the security
of their SCADA networks

Sydney, Australia [June 12, 2008] – Citect has moved to reassure its SCADA customers
they are extremely unlikely to be at risk from potential security breaches found by Core
Security Technologies in Windows-based control systems utilizing ODBC technology, so long
as their systems are protected by industry-standard security guidelines.

Citect and other SCADA and Control vendors have been communicating potential
vulnerabilities of control systems when they are connected to the internet for some time.
However, Citect believes this is only relevant to a company using ODBC technology and
directly connecting its system to the internet with no security in place – a situation unlikely in today’s business environment.

Citect’s Global CEO, Christopher Crowe, says, “The security of our customers’ control
systems is of paramount importance to us. Though we have not had any reports of breaches,
we are contacting our customers globally to confirm they have followed recommended
network security measures. We have also developed a patch for those companies that might
not be able to implement necessary network security measures promptly.”

Citect has been designing SCADA software for 21 years and educating the market about
network security. Citect follows, and recommends to its customers, industry best practices in
the development and implementation of control systems. Citect’s position on SCADA and
process control network security remains unchanged – SCADA systems, like any business
systems, must be protected from unauthorised access via the internet. They must be secured
by robust protection including firewalls, intrusion detection systems and VPNs. There are
basic security measures published by various organizations. Citect advises customers on
network security and has published whitepapers to further educate the market: Visit
www.citect.com for more information.

Thanks to Marcus Sachs for pointing me to this one—WB

In my view, this raises several questions. Why, again (remember, Core accused Wonderware of dilatory response just a couple of months ago) did Citect take five months to fix the problem? Why did Core go to the Associated Press? Does Core have an ethical problem with doing that? Advertising your services by outing people and companies you call bad actors — how do you all feel about the ethics of what Core did?

Security hole exposes utilities to Internet attackBy JORDAN ROBERTSON

____________________________

Bandolier: Is half way there good enough?

I want to specifically respond to Ralph Langer’s comments from my blog post on Severity Levels.

Ralph posted, “While I agree in general that severity cannot be established without context, experience tells me that such context can hardly be established by any kind of automated software tool. Even worse, many asset owners don’t have any realistic idea, not to say methodology, of calculating the cost of potential cyber incidents. Without having seen the Bandolier product, my guess is that it goes half the way… which is better than nothing, after all.

P.S. Why not discuss this stuff over at Digital Bond’s?”

My response:

Being honest, I want to discuss it here, because this is MY turf…and frankly, the Digital Bond folks don’t seem to want to hear it. Let me show you why I feel that way.

On Monday I had a phone call with Dale Peterson to determine if he would have been interested in a joint proposal (Digital Bond and Applied Control Solutions) to extend Bandolier to address the control system issues identified in the original blog post. I felt, and I still feel, that Bandolier is useful, but not addressing all the issues it could, and I wanted to help make it an even more useful tool—all that it could be. So you’re right, Ralph. Bandolier “is better than nothing.”

But do we want to stop at “better than nothing?”

I explained to Dale my concern that Bandolier appeared to be addressing computers and not systems. My concerns were that I thought the Bandolier approach could cost the end users significant money by having to address non-critical systems. Additionally, I thought Bandolier could provide a false sense of security by not addressing the security of the systems and facilities (why are we doing Critical Infrastructure Protection -CIP).

Dale’s response was their scope was not the security of the systems. He added that he felt it would be difficult to address the control systems issues but would be happy to use whatever result I could come up with as a plug-in to Bandolier.

I can’t help but feel that just because it would be difficult doesn’t mean we shouldn’t be addressing the system security issues, instead of trying to secure systems by securing individual computers.

According to Dale, Bandolier is a joint effort by the control system supplier who provides the optimal operating system configuration for their new systems, Digital Bond staff which reviews the configuration, and an end-user (such as TVA).

He then mentioned the purpose of Bandolier was to be the “gold standard.”

I then asked if the current “gold standard” was NIST SP800-53.

Dale said it was the NIST Federal Desktop Core Configuration (FDCC). Remember, the FDCC was developed for desktop operating systems not industrial control systems.  That’s what NIST SP800-53 and ISA99 are trying to do—write a “gold standard” for industrial control systems.

So why are we ignoring that work, from people who actually have worked with control systems? If somebody can explain this to me in short simple sentences, maybe I won’t be so confused.
I then asked Dale how Bandolier handled old, obsolete operating systems such as Windows NT4 and Windows 95. Dale said Bandolier had no files for these old systems. These older systems are still in use even with many modern DCS upgrades and often cannot be replaced – how can you ignore them??? Remember, companies like ATS (Advanced Technical Services) even still manufacture out-of-date circuit boards for industrial computers and PLCs that are 25 years old, and cannot be taken out of service…but that are networked through serial device servers on the plant floor. These systems will not go away for decades. Why are we ignoring them?

I believe the purpose of control system cyber security (CIP) is not to secure a computer, but to secure a system and/or facility. The vulnerability of a computer is important only if it leads to a compromise of a process or facility.

To explain why it is so important to secure systems not computers, I will provide two examples. Mark Hadley from DOE’s Pacific Northwest Laboratory (PNNL) and I created a list of myths about control system cyber security. One myth was that firewalls are enough protection. As Mark and others would say, ”firewalls are speed bumps,” particularly if appropriate rule sets are not used.

The hacking demonstration performed by Idaho National Lab staff at the 2004 Control System Cyber Security Conference in Idaho Falls traversed MULTIPLE sets of firewalls prior to compromising (opening and closing) the smart relays and operator displays.

Another myth is that VPNs secure networks. At the 2007 Conference in Portland, PNNL compromised OPC packets and then used a VPN to camouflage the compromised packets. The result was a compromise of a modern SCADA system (changing voltages) and the operator displays.

Both of these examples demonstrate that it’s the system that needs to be secured, not the computer. 

Here are my comparisons of what severity really means in the process control system context to what Digital Bond’s Bandolier Project defines severity to be. I hope this can begin to explain the differences:

Severe
Bandolier -This represents the most serious potential impact to the control system. A check that is non-compliant and has an internal rating of severe generally indicates that the system is at risk unless other specific mitigation measures are in place. Poorly configured directory permissions or network services, for example, can lead to system compromise and would have the severe rating.

Examples:
- Incorrectly configured permissions on critical system directories such as /etc/passwd
- Web server of FTP server with improper user restrictions

Control systems – This represents failures, omissions, or errors in design, configuration, or implementation of required programs and policies which have the potential for major equipment and/or environmental damage (millions of dollars), and/or extreme physical harm to facilities’ personnel or the public; and/or extreme economic impact (bankruptcy).

Example:
The Bellingham, WA gasoline pipeline rupture’s impact was 3 killed, $45M damage, and bankruptcy of Olympic Pipeline Company. The National Transportation Safety Board identified the cause as the operator using the operational SCADA system for development work. A Bandolier check would not have identified the problem to this obviously severe event.

Moderate
Bandolier - This category represents a variety of checks with potential control system security impact. They may not lead to system compromise in themselves, but could aid an attacker or become a more serious problem in the event of some other failure or compromise. Included in this category are items such as unnecessary services, inadequate password strength, insufficient logging, etc.

Examples:
- Network share that exposes sensitive control system information
- Incorrectly configures security event log settings
- Weak password requirements such as inadequate length or complexity requirements

Control systems – This represents failures, omissions, or errors in design, configuration, or implementation of required programs and policies which have the potential for moderate equipment and/or environmental damage (tens of thousands of dollars) with at most some physical harm to facility personnel or the public (no deaths).

Examples: –
- Maroochy (Australia) wireless hack which caused an environmental spill of moderate economic consequence. I don’t believe this would have identified by a Bandolier check.
- Browns Ferry 3 Nuclear Plant Broadcast Storm could have been caused by a bad Programmable Logic Controller (PLC) card, or insufficient bandwidth would not have been detected by Bandolier testing.

Informational
Bandolier - This category represents checks that may not pose a threat to the system or are simply informational in nature. These will typically identification checks that indicate the role or version of a particular control system application.

Example:
- Configuration file indicates that the system is serving in a particular role (e.g. historian, real time, etc…)

Control systems – This represents failures, omissions, or errors in design, configuration, or implementation of required programs and policies which have the potential for minimal damage or economic impact (less than $10,000) with no physical harm to facility personnel or the public.

Example:
- Davis Besse Nuclear Plant cyber incident caused by a contractor plugging in a laptop contaminated by the Slammer worm into the plant Safety Parameter Display System. I don’t believe this would have been identified by Bandolier testing.

I want to reiterate that ACTUAL control system cyber incidents including Bellingham, Maroochy, Browns Ferry, Hatch, the Florida Outage, etc were not caused by incorrect operating system configurations or operating system vulnerabilities. Neither were many of the other control system cyber incidents in my incident database.

Is CIP protection of critical infrastructure or “critical computers”?

Why do DOE and DHS continue to fund projects that do not address actual control system cyber incidents that have already occurred?

In fact, why don’t they want to know about these incidents, most of which have NOT been reported to government?

Why isn’t DOE funding projects like Aurora with solutions to demonstrated control system vulnerabilities?

Yes, the Bandolier project is worthwhile. Yes, Ralph, we should be developing automated tools to help us secure control systems. But the Bandolier project is not designed to go all the way to where we should be.

Joe Weiss

We have a problem.  We have efforts at all levels to secure industrial
control systems, but there isn’t much coordination.  Some efforts are
falling by the wayside.  The Roadmaps for energy and water are mostly
taking top-down approaches.  There are approaches in the middle such as
the ISA-99, and going further toward the technical side, Secure
Authentication for DNP and the AGA-12 effort.

However, I know of nearly nothing taking place at the bottom.  There are
training courses from DHS aimed at operators, but there is little
mandate to train them.

Looking at the example of Safety standards and the like, we had buy-in
from executive, engineering, and operations people.

We don’t have that happening right now in security.  We have Roadmaps
written for executives who are expected to wave their hands and make
security happen without knowing where it’s supposed to come from or how
it’s supposed to work.

We have half baked standards, still very much in development from
engineers and IT specialists.  These standards don’t know how to address
issues such as patch management because at the end of the day, we know
we can’t set a standard without coordinating with operations.

And as far as I know, the efforts to train operators on this security
issue and how it helps them is feeble.  They don’t see the need yet.

Meanwhile, legislators are turning up the heat on organizations such as
TVA for not be properly secure.  The managers are blindsided.  They
don’t know which way to turn and their IT departments are at a loss to
figure out how to set policies for reasonably safe AND secure control
system operation.

The problem is that these professions aren’t talking to each other.  The
Energy sector roadmap becomes a matter of useless handwaving if we don’t
bring engineering and operations in to the picture.  Engineering or
procurement standards won’t help if managers don’t know what it does or
if operations doesn’t know what to do with it.  And one thing is sure:
We can build economical and secure control systems, but if the operators
don’t know how to use it or fail to see the advantages, they’ll subvert
these features and all will be for nothing.

I had hoped the Roadmap documents would be broader than they turned out
to be.  I had hoped that ISA-99 would have more than just engineers and
IT specialists in it.  I had hoped that the ISAC organizations,
regulations, and legislation would push operations to train themselves
to ask for what they need.  None of this is happening on a practical
scale.

It’s time to talk turkey.  We keep bumping in to this problem.  How can
we get this discussion started?  What umbrella organizations should we
build to facilitate this discussion?

Something has to happen here.  Compare the Roadmap documents to ISA-99,
or the DHS operator training.  We’re barely speaking the same language.

If we can’t learn to build a common language and inclusive, industry
specific practices, we’re going to continue spinning our wheels.  Is
anyone from another utility interested in working with me on this?  I’m
tired of seeing the efforts of so many talented people go to waste.
======================================================================
Fair disclosure:  I participate in the Water Sector Coordinating Council
Cyber Security Working Group, ISA-99, and the DNP3 Technical Committee
and I’m employed by a large water and wastewater utility.
Jake Brodsky

URL: http://www.washingtonpost.com/wp-dyn/content/article/2008/06/05/AR2008060501958.html
Supporting URL: http://www.gao.gov/new.items/d08526.pdf

Cyber Incident Blamed for Nuclear Power Plant Shutdown
By Brian Krebs
washingtonpost.com Staff Writer
Thursday, June 5, 2008; 1:46 PM

A nuclear power plant in Georgia was recently forced into an emergency shutdown for 48 hours after a software update was installed on a single computer.

The incident occurred on March 7 at Unit 2 of the Hatch nuclear power plant near Baxley, Georgia. The trouble started after an engineer from Southern Company, which manages the technology operations for the plant, installed a software update on a computer operating on the plant’s business network.

The computer in question was used to monitor chemical and diagnostic data from one of the facility’s primary control systems, and the software update was designed to synchronize data on both systems. According to a report filed with the Nuclear Regulatory Commission, when the updated computer rebooted, it reset the data on the control system, causing safety systems to errantly interpret the lack of data as a drop in water reservoirs that cool the plant’s radioactive nuclear fuel rods. As a result, automated safety systems at the plant triggered a shutdown.

Southern Company spokeswoman Carrie Phillips said the nuclear plant’s emergency systems performed as designed, and that at no time did the malfunction endanger the security or safety of the nuclear facility. ad_icon

Phillips explained that company technicians were aware that there was full two-way communication between certain computers on the plant’s corporate and control networks. But she said the engineer who installed the update was not aware that that the software was designed to synchronize data between machines on both networks, or that a reboot in the business system computer would force a similar reset in the control system machine.

“We were investigating cyber vulnerabilities and discovered that the systems were communicating, we just had not implemented corrective action prior to the automatic [shutdown],” Phillips said. She said plant engineers have since physically removed all network connections between the affected servers.

Computer security experts say the Hatch plant incident is the latest reminder of problems that can occur when corporate computer systems at the nation’s most critical networks are connected to sensitive control systems that were never designed with security in mind.

Specifically, experts worry that vulnerabilities were introduced into the systems that regulate the electrical grid as power companies transferred control of generation and distribution equipment from internal networks to supervisory control and data acquisition, or SCADA, systems that can be accessed through the Internet or by phone lines, according to consultants and government reports.

The move to SCADA systems boosts efficiency at utilities because it allows workers to operate equipment remotely. But experts say it also exposes these once-closed systems to cyber attacks.

“Part of the challenge is we have all of this infrastructure in the control systems that was put in place in the 1980s and ’90s that was not designed with security in mind, and all of sudden these systems are being connected to [Internet-facing] business networks” said Brian Ahern, president and chief executive of Industrial Defender Inc., a Foxborough, Mass.-based SCADA security company.

Joe Weiss, managing partner at Cupertino, Calif.-based Applied Control Solutions, said Hatch is not the only plant that has suffered this type of unusual event. But he said it is one of a handful of public events of this type because the Nuclear Regulatory Commission documents all unusual events, in contrast to non-nuclear facilities that do not make their unusual events public.

“Consequently, it is expected that non-nuclear facilities have experienced similar events,” Weiss said. “The Hatch event illustrates the unintended consequences that could occur when business information technology systems interconnect with industrial control systems without adequate design considerations.”

Weiss said unplanned, automatic shutdowns such as what happened at the Hatch plant are costly, forcing utilities to purchase power from other parts of the grid to the tune of about $1 million a day. But more importantly, Weiss said, automatic shutdowns unnecessarily challenge nuclear safety systems.

“Anytime you have to shut down, especially with an automatic shutdown, you’re challenging the safety systems,” he said. “What happened [at Hatch] was absolutely what the plant was designed to do, but there’s always that chance that something could go wrong.”

The NRC has for years had regulations in place that require that all plants be able to defend against cyber attacks. But the agency is still in the final stretch of implementing more specific cyber-security regulations that would require plants to detail their plans for defending their digital networks as a condition of maintaining their operating license, said Scott Morris, deputy director for reactor security at the NRC.

“The plants are expanding their use of digital technology to put more megawatts on the grid, and because of that these lessons are going to occur,” Morris said. “But our expectation is that when these types of events happen, that [plant operators] correct the problem and share the information broadly with the rest of the industry.” ad_icon

Unplanned nuclear plant shutdowns used to be a fairly common event, but not anymore, Weiss said. In fact, he said, another shutdown of a U.S. nuclear plant was also precipitated by a cyber event. In August 2006, Unit 3 of the Browns Ferry nuclear plant went into a shutdown after two water recirculation pumps failed. An investigation found that the controllers for the pumps locked up due to a flood of computer data traffic on the plant’s internal control system network.

Weiss said many people in charge of SCADA systems have sought to downplay the threat that hackers pose to these complex networks. But he cautioned that internal, accidental cyber incidents at control system networks can be just as deadly as a carefully planned attack from the outside.

In June 1999, a steel gas pipeline ruptured near Bellingham, Wash., killing two children and an 18-year-old, and injuring eight others. A subsequent investigation found that a computer failure just prior to the accident locked out the central control room operating the pipeline, preventing technicians from relieving pressure in the pipeline.

“To people in the IT world, cyber means ‘attacks,’ but what I tell people is that in our world the predominant cyber events are unintentional,” he said. “The flip side of that is if it can happen unintentionally, it can probably be caused intentionally and be a whole lot worse.”

News of the Hatch incident also comes as the cyber-security posture of the electric and nuclear power industry is coming under increasing scrutiny from Congress and government investigators. Last month, the Government Accountability Office issued a scathing report about cyber security weaknesses at the Tennessee Valley Authority, the nation’s largest public power company and operator of three nuclear plants, including Browns Ferry.

The GAO found that TVA’s Internet-connected corporate network was linked with systems used to control power production, and that security weaknesses pervasive in the corporate side could be used by attackers to manipulate or destroy vital control systems. The agency also warned that computers on TVA’s corporate network lacked security software updates and anti-virus protection, and that firewalls and intrusion detection systems on the network were easily bypassed and failed to record suspicious activity.

What do severity ratings REALLY mean?

I read a blog on Digital Bond’s Bandolier project (www.digitalbond.com, Posted: May 27th, 2008 under Bandolier, DoE Research Project). It seems to be a good approach to identifying vulnerabilities in control system computers. The severity ratings for Bandolier are a good idea but the approach does not go far enough. Since these ratings are used for compliance reporting, it potentially could cost companies a significant amount of money without an accompanying risk reduction. The missing piece is the impact on the process or facility. I see two issues with the Bandolier approach- the first is the classification of non-critical computers as “severe”. The second is how Bandolier is used in the overall context of securing the facility. Many of the Major and Moderate control system cyber incidents I have identified in my incident database would not have been identified using an approach like Bandolier as they were not caused by traditional computer vulnerabilities but represented failures, omissions, or errors in design, configuration, or implementation of required programs and policies.

As an example of the first issue, consider a “typical” automation system like the one with which I am now working. It is a large power plant control system retrofit where security is to be considered. This facility includes many generations of control system workstations that are used in both critical and non-critical applications. Some are connected to the Process network LAN, some are connected to the Control LAN, some are connected to the Corporate LAN, and some are not even connected to a LAN. Using the Bandolier security rating approach, many workstations used in non-critical applications and not even connected to the plant control network could get a “Severe” rating if all IT appropriate security controls have not been applied. However, in the grand scheme of protecting assets (the facility and its ability to produce power), this type of rating would require significant expenditures without offering any benefit to the security of the facility.

For security approaches such as Bandolier to be effective, they need to be combined with impact on facilities. Examination of ISA SP99 requirements and risk definitions and tools such as the Idaho National Laboratory-developed Cyber Security Self-Assessment Tool (CS2SAT) make it clear that consequences must be understood in terms of the effects on facilities, major equipment health, environmental concerns, and public safety. Perhaps that was the idea of the Bandolier approach. But unless they define what they mean by “systems” to include these effects, their approach will not meet automation systems needs. I think these identified limitations in Bandolier have more to do with the different cultures of IT and Operations where IT’s focus is protecting computers while Operations focus is protecting the process and facilities. I believe the potential for extending Bandolier (or other IT-type approaches) to include impacts on facilities provides a basis for the two communities to work together. Consequently, this will be a major discussion area at the Applied Control Solutions August Control System Conference in

Joe Weiss