What's the present state of process security? Not good. There appear to be numerous looming threats and potential attacks, and not much up-to-date help from U.S. agencies or other governments. Promising standards efforts seem to be lagging or frustratingly unspecific, and many users reportedly don't update passwords, configure firewalls, restrict unfamiliar PCs or data storage media, limit access between IT and plant-floor networks, or take many other basic security precautions. No wonder there seems to be so much worry, fear and panic.
So what are conscientious process control engineers and plant managers supposed to do? Well, there are many longstanding security efforts ongoing in process applications both large and small, but almost all of the experts running actual, plant-floor security programs are unable to talk about them. Of course, this is because most organizations are concerned that describing what they're doing will be seen as a challenge to hackers and invite more unwelcome probes and possibly even destructive attacks.
Still, there are a few brave process control engineers and companies that are willing to describe their experiences and offer some badly needed advice and encouragement to their colleagues in the field.
Security at Ergon
"The problem is that, as we've become a more connected world, the process equipment controlling temperature, pressure, level and flow can convey data outside their usual areas," says Steve Elwart, director of systems engineering at Ergon Refining Inc. (www.ergon.com) in Vicksburg, Miss., which uses about 25,000 barrels per day of naphthenic crude to produce lubricants and process oils (Figure 1). "However, the need to get at control-related data can expose these devices to a lot more risk. When you begin to use Windows in control systems, its easy connectivity can get you into trouble quickly. I know of one situation where an IT department added a controls server to a corporate domain, automatically rebooted the system when it was trying to add some routine patches and almost destroyed a major piece of equipment by compromising its monitoring system."
Elwart adds that controls engineers and corporate managers need to decide what how much connectivity is acceptable by answering two questions: How valuable is the data they need and how much can they safely open their controls to the outside? To help answer these questions, he also serves as a member of the U.S. Department of Homeland Security's (DHS) Energy Sector Control Systems Working Group (ESCSWG), which has been updating its Roadmap to Achieve Energy Delivery Systems Cybersecurity (www.controlsystemsroadmap.net).
"Unfortunately, plant managers have a lot to think about, and process control security previously hasn't been on their radar as much as it should be," says Elwart. "Likewise, when everyone goes to the budget trough, the physical security, financial security and IT security guys say they must be the top priority or someone could go to jail, and so they get funded first. But when the process security guys say they need process security because it's good for availability, they come last and get the short end. This happens because the number one security issue for IT and business is access, then accuracy and, finally, availability. This is upside down for controls, of course, where availability and uptime are most important and where accuracy and access come in second and third. It's these appropriate, but reversed priorities that make process security seem to be less important than the others."
Another problem is that the workforce in many refineries and other process applications is smaller and younger, so online assistance via PC-based systems and connecting to production and control-level information is crucial for them to do their jobs.
Good Security = Good Business
Basically, Ergon looks at its own process security from a business perspective, so it views security as just another way to prevent operating interruptions and reduce downtime. "The first step in security is changing the lexicon," explains Elwart. "For example, we don't call computers PCs. We call them machines because that's what plant people are used to dealing with. PCs are what you play games on, so they aren't taken seriously with that label."
Elwart reports that his Systems Engineering department manages a complex and growing network of about 250 computers, a dozen SCADA servers, as well as other servers and related equipment used by Ergon's refining, fleeting, retail and some corporate departments. "In the past five years, we've designed everything with security in mind. Before that, all we could do was limit the access points into the network, but we also learned that you can't make a 25-year-old process control system inherently secure" says Elwart.
So two years ago, the department also began expanding beyond Ergon's existing Foxboro I/A distributed control system (DCS) installed in 1987 by adding Emerson Process Management's Delta V, so now the plant consists of roughly one half of each system. Over the past 15 to 20 years, Ergon also acquired a variety of PLCs from Allen-Bradley, Triconex and Siemens, so it's also working to integrate them into its expanded DCS and create a better interface into these controls with a combination of Modbus, Ethernet, serial communications and OPC.
Besides continuing to severely restrict external access, Elwart adds that his department also makes sure not to use administration-level accounts for routine system tasks.
"I tell people that before asking their managers for money for process security, they need to do an internal survey of all their networked equipment, perform a risk assessment (RA) for each device, evaluate its criticality, prioritize each item and limit its access to the outside world," says Elwart. "We've done it informally over the years, and we're still doing it."
Cooperation and Common Sense
Besides upgrading components, software and networking, Elwart emphasizes that one of his most powerful security tools is coordinating efforts with his overall corporate IT department. This cooperation may come a little easier at Ergon because Elwart trained as a chemical engineer and worked as a process engineer before migrating over to the computing and software side. Likewise, he adds, his Systems Engineering department has a unique blend of business and controls skills because most of the 9 to 10 staffers previously worked in completely different areas.
"IT people already have network security in mind, but we also make sure they get basic operations training, too," says Elwart. "I want everyone in our department and in IT to understand how our refinery runs 24/7 and why those calls at 2 a.m. are a big deal. This training makes everyone more sensitive to shift changes, scheduled downtimes and production runs. As a result, when control engineers open some ports between the network and control level to get some equipment to work and then leave them open, then IT can follow up and shut down the unused ports." Conversely, some control engineers and operators also get trained in business-level and SCADA security practices.
Elwart adds that the Systems Engineering and IT departments also jointly confer on scheduling upgrades and other projects to avoid causing production problems. "For example, while other IT departments may add enterprise software over a weekend and create some difficulties, we look at what runs and turnaround times are coming up and try to find the best upgrade times," explains Elwart. "For instance, we won't change Internet Protocol (IP) addresses on a terminal server on Friday, so we can avoid possibly locking out some engineers over the weekend. We also won't change IP addresses on Monday to prevent people from complaining that they didn't see the email sent on Friday. So we usually make any IP address changes sometime between Tuesday and Wednesday."
Likewise, Ergon also waits until its vendors test new software patches and then installs them during normal downtime. "This is because we've seen more operating interruptions due to spurious shutdowns caused by untested patches than we have from what they're supposed to be protecting us against," says Elwart. "If there's an emergency, we'll call the vendor in to help install a patch."
Elwart reports that other common-sense security procedures used at Ergon include locking its server room so no unauthorized people can plug into the servers inside. Similarly, the Systems Engineering department also changes the default passwords on all its devices, which removes one of the most typical pathways used by outsiders to gain unauthorized access. It also uses several levels of authentication, which means different passwords for different network areas. Ergon uses Cisco firewalls to protect its networks, but it doesn't set them and forget them. It implements well-thought-out rules that will meet Ergon's needs when it configures its firewalls and then constantly updates them.
"Management at some refineries will sometimes say, ‘We're nice people. Why would anyone want to attack us?' So they need to be shown that pretty much every industrial facility and application with a network connection has had some kind of probing or potential attack that could be a problem," adds Elwart. "I recently learned about a refinery network that had been under a continuous attack for several months. This was a brute-force attack in which the intruder was trying up to two user names per second to gain access. I believe the application's system was able to show where the attacks were coming from, and the unsophisticated hacker was actually caught."
Elwart adds that one of the main lessons he's learned from trying to improve Ergon's process security is that people can't be prevented from trying new technologies. "IT sometimes tries to tell people not to use a certain technology, but this doesn't work because many will ignore those instructions," says Elwart. "It's much better to embrace new technologies and then learn to manage them effectively. Much of this comes down to simple education. We always talk about hackers, but many security problems also come from inadvertent mistakes. For example, another refiner found some unexplained photos on a desktop and found they'd been accidentally downloaded to the PC when one of the cleaning crew plugged into a USB port to charge his smart phone. So they educated the cleaning crew not to do that sort of thing, and then also configured their PCs not to perform automatic downloads. People are willing to go along with rules if they're told why those rules are important."
Security Similar to Safety
To increase security awareness and compliance, many process organizations report that it helps their staffs to think about security in the same ways they think about process safety. Some internal security teams use business-type risk assessment (RA) matrices (RAM) based on probability and severity of incidents that are already widely used in their safety efforts. This can help production managers learn that security vulnerabilities can be equivalent in impact to the downtime caused by an over-pressurized valve fire or other accident. So the next time someone brings an unfamiliar laptop into their facility, they'll stop and think, question if it might have a virus and get it checked out.
"Process security is really the second coming of safety," says Rick Kaun, global business manager for industrial cybersecurity at Honeywell Process Solutions (http://hpsweb.honeywell.com). "But, like safety, there's no point at which you're completely secure. It's a constant lifecycle of reviewing, revising and refreshing the three main pillars of process security—technologies and services, policies and procedures, and people. For example, I helped assess a refinery a few years ago, and we found 57 security concerns, and they fixed most of them. However, when we went back two years ago, we found 53 more items, but they were mostly unsophisticated issues. So we'll go back next year, and see what's up."
Likewise, Suncor Energy Inc. (www.suncor.com) in Calgary, Alberta, Canada, reports that its recent Process Control Network/IT Security Project involved updating its security design, security maintenance and asset configuration management capabilities in its oil sands extraction, upgrading and support facilities. The firm's PCN security maintenance program included improving its testing environment, anti-virus and patch management tools, password and authentication management, security hardening guidelines, and health and security status monitoring.
"Making the case for improving cybersecurity in our budget meant better understanding the importance of our assets and what the lack of protection and security could do to them," says Cliff Pedersen, PE, Suncor's product production processes manager. "We calculated the cost of an unplanned outage if our networks were left as is, and the likely cost was several million dollars. A project of this nature, which spends money, but doesn't ‘make more oil' is difficult to sell. For a process security project to succeed, you need to illustrate the true impact of what you're doing and show the consequences of not doing it."
Brad Hegrat, principal security consultant at Rockwell Automation (www.rockwellautomation.com), adds that, "A basic RA involves inventorying an application's components, categorizing them, rating their criticality, assigning risk profiles and addressing those risks by mitigation, transfer, acceptance and avoidance. It's also crucial to do a comprehensive gap analysis of how a facility's staff sees its security role, what polices are already in place, and then assess technical needs." Besides continually updating passwords, firewalls and patching policies, Hegrat also stresses that users should put PLCs in "run mode," so they won't accept edited changes to their existing configuration or programming.
Security at Salt River
Similar to most multi-regional power plants, Salt River Project's (www.SRPnet.com) Navajo Generation Station (NGS) in Page, Ariz., always had security practices to prevent unauthorized access to its controls and backup policies for any incidents that might occur. And as PCs and software emerged and moved onto the plant floor, SRP kept its control network separated from its IT and business network.
NGS is a coal-fired plant that routinely produces about 2250 megawatts in its three 750-megwatt units and is projected to maintain a typical uptime of 92.5% over the next 10 years. The plant burns low-sulfur, bituminous coal, and its power goes to customers in Arizona, Nevada and California and also drives water pumps throughout the Central Arizona Project.
"However, a few years ago, our plant management saw the writing on the wall that NERC-CIP would eventually designate our plant as a critical asset, and they told us to begin looking towards complying with NERC-CIP Version 3," says Mike Hull, supervisor in NGS's computer control group. "So we put together an SRP security team and drafted a best practices document, so we could move in the right direction, establish a security baseline for all of SRP and have consistency across our organization about back-up strategies, patch management and physical and electronic access control."
Basically, the North American Electric Reliability Corp. (www.nerc.com) and its Critical Infrastructure Protection (CIP) program are requiring electricity producers to draft and implement security policies for their facilities that comply with its NERC-CIP standards. However, even though its security efforts are further along than other industries, critics claim that NERC-CIP allows utilities to define too many assets as non-critical and thereby avoid securing them.
Second Firewall, Second Network
Hull reports that SRP's and NGS's recipe for security began with identifying all its critical assets and their vulnerabilities and implementing indicated security devices, services and a layered infrastructure needed to protect them. The team found and documented about 3000 applicable assets. "However, this first part of updating technology is the easy part because next we have to do a performance baseline; make sure we're designing our security system and roadmap so it can be rolled into NERC-CIP compliance; and address any new vulnerabilities that show up. For example, NERC-CIP says to define an electronic perimeter and access points and decide what goes on either side. So even though we recently put firewalls around our corporate network, in late 2010, we put a second set of firewalls around our controls network. This was done so we would have defense-in-depth, or multi layers of security, so the plant-level could only push data out to the business side, but not accept any coming back in."
Mike Martinez, principal consultant for the critical infrastructure and security practice at Invensys Operations Management (www.iom.invensys.com), reports that SRP's network infrastructure changes also enabled deployment of centralized anti-virus management and back-up capabilities, improved network monitoring and a remote access- jump server with role-based user authentication for remote access. "We've been working with SRP for many years, so when we learned that these cybersecurity requirements were coming, we were able to help design and deploy a solution that would allow them to meet their exisitng best practices, while being consistent with their future need for a NERC-CIP-compliant program," says Martinez.
While many networks use two layers of firewalls, some are also installing a data-based demilitarized zone (DMZ) between their corporate local area network (LAN) and their control system LAN (Figure 2). This provides an added layer of protection because no communications take place directly from the control system LAN to the business LAN, according to the U.S. Computer Emergency Readiness Team (US-CERT) and its Control System Security Program (CSSP, www.us-cert.gov).
Ernie Rakaczky, program director for control system cyber security at Invensys reports that US-CERT is helping many suppliers find vulnerabilities in their software and is testing security risk mitigation strategies to make sure they work properly.
Besides extra firewalls, NGS's security team also added a second maintenance network built on secondary network cards in its PCs, which is another likely NERC-CIP requirement. So while the original card performs its regular control operations on the plant's dedicated Invensys Foxboro I/A DCS, the second card does back-up, maintenance, patch deployments and other tasks.
This second maintenance network touches all the same points as the process control network at NGS's three generating units, SO2 scrubbers, lake pumps and workstations. The teams also added new physical and electronic access controls and password management for logging onto cyber-related assets. For example, mechanical maintenance staff will no longer be allowed into the control room, but will have to go a clearance office to have clearances issued.
Hull adds that upcoming efforts will take SRP's and NGS's security beyond protecting equipment at one point in time to make it part of their lifecycle and obsolescence planning, too.
"I think the feeling at NGS now is that we're headed in the right direction on process security," adds Hull. "There's a lot of overhead in maintaining security and compliance. And the process never really ends, so we still have more work to finalize our security infrastructure, implement more secure solutions across the whole NGS facility, update other components, and more thoroughly define our physical and electronic security perimeters. Later on the roadmap, we'll check for vulnerable assets again and see how well we did now."
Jim Montague is Control's executive editor.