What's in your server?

By Rich Merritt, Senior Technical Editor

verybody is trying to scare the beejeezus out of you these days with talk of nasty ol&rsquo hackers and crackers, how assorted bad guys are trying to break into your process control system, and all the risks you take by using Web, Ethernet, wireless, and Microsoft technology in your control system. While we don&rsquot exactly subscribe to the theory that terrorists are plotting to destroy your recipe for making chocolate, we realize that threats do exist from hackers, viruses, and competitors. Maybe even terrorists, too.

To help you bolster your defenses, we've assembled a timely list of tips and techniques you can use to build a fortress to secure your data and protect your control system from intruders. (See Top ten defenses sidebar below.)

RELATED STORY
What do vendors say about process control security?

Find more Articles, White Papers and Industry Books on control system security listed at the bottom of this article.

Some of these tips and techniques don&rsquot cost a thing. Some are just common sense, some require a change in the way you do things, and some require the purchase of a little hardware. Justin Lowe, a security analyst at PA Consulting, explains it best: &ldquoThere is no silver bullet,&rdquo he says. &ldquoA suite of security measures are required but only around 30% of the solution is technical. The remainder is procedure, process and management.&rdquoThe Nature of the Threats
We&rsquove heard the classic stories about security problems ad nauseum: the wastewater plant in Australia, the nuclear plant in Ohio, and the SCADA system at an electric power plant in California (which turned out to be a hoax). Ernie Rakaczky, Director of Process Control Network Security at Invensys Process Systems, says they&rsquove seen a few in process control, too:

Internet worms such as CodeRed Nimda and SQL Slammer have attacked web servers
Outsiders have tapped into wireless communications paths
An intruder connected via a modem
A maintenance worker accidentally inserted a virus via an infected floppy or CD
Unauthorized personnel gained access to an unprotected PC in an unlocked lab
A remote user inadvertently introduced a virus into the network
An intruder entered through a Remote Access Services (RAS) link

Except for a few incidents like these, the process control industry has remained relatively immune from the huge number of problems that plague commercial web sites, banks, and government institutions. Maybe the bad guys haven&rsquot discovered us yet, or maybe we don&rsquot have anything they want. Or maybe companies in our industry just don&rsquot talk about it when they take a hit.

Bad guys are definitely out there. One of our contributors, a control engineer at a large Midwest refinery, is worried. &ldquoWe have been written by name on terrorist lists, so our physical security is very tight,&rdquo he says. He asked to remain anonymous as did other contributors. Nevertheless, there does not appear to be a major, organized attack on process control systems yet.

It certainly appears that the two biggest problems are (1) external random attacks by worms, viruses and similar software that roam cyberspace looking for vulnerabilities, and (2) internal problems caused by disgruntled employees, careless operators, and bad procedures.

In the first case, nobody outside is trying to destroy the chocolate recipe they don&rsquot even know you make chocolate. If they get you, you are probably just the victim of a random Internet crime. In the second case, you do it to yourself because of poor security or poor training. Both situations are preventable.

The tips and techniques that follow will help you create a fortress and tighten up security, but nothing will stop someone who is determined to take your plant down. No firewall is safe from a talented hacker, no anti-virus software gets them all, and dealing with disgruntled employees and actual terrorists is beyond the scope of this article. We can, however, help you make it more difficult for them. So let&rsquos build a fortress.

Get Off the Networks!
End users and vendors alike universally advise disconnecting your process control system from the Internet, corporate networks, business LANs, or any network not needed for actual control. One engineer at a chemical plant said it bluntly, &ldquoWe do not allow any outside connections into our control system. There are no modems and certainly no Ethernet connections to the Web or business system.&rdquo It&rsquos the fortress mentality, but it works.

Carl King, senior engineer at Cinergy Services in Owensville, Ind, agrees. &ldquoIf you do not have a strategic reason to connect your control system to your corporate network, don't connect it,&rdquo says King. &ldquoThat provides the best security for attacks from external sources. If possible, provide a separate data collection system for information that needs to be available to the corporate network from the control system. No data traffic should be allowed to directly access the control network from the corporate network or the internet.&rdquo

Some people call the separate data collection system a &ldquoreplicated system&rdquo or a &ldquoshadow server.&rdquo Essentially, data that is needed by external systems such as maintenance, ERP software or corporate IT is sent to a computer outside the control area, where a duplicate image of the real-time data base or process historian is maintained. The external systems can take whatever they want from the &ldquoshadow server&rdquo without affecting the process control system. If the shadow server is attacked by a virus or worm, this does not affect the control system.

Communications between the control system and the shadow server go through a firewall and protective software, which we will cover below.

&ldquoThe Shadow Server allows process data to come from the control LAN PI Server and be moved to the business LAN side, with replication,&rdquo says our anonymous refinery engineer.

"All the functionality of collecting process data, creating spreadsheets and other uses we make of the data are available to people on the business LAN without having any risk of them being on the control LAN pulling data. Our control system is physically isolated from our business network and the outside world.&rdquo

&ldquoWe&rsquove banned e-mail and web browsing from the control system,&rdquo says an anonymous Digital Security Advisor at a major oil company. &ldquoIf a plant operator needs to access these facilities then we provide them with a corporate desktop alongside the control system.&rdquo

Vendors agree. &ldquoIt is my recommendation that the local control networks should be 100% separated from the Internet,&rdquo says Scott Saunders, Director, Strategic Marketing at Moore Industries. &ldquoThis will stop hackers from outside coming in.&rdquo

Metso agrees, too. &ldquoA DCS should not be connected directly or indirectly to the Internet,&rdquo says Roger Leimbach, marketing manager at Metso Automation.

Siemens agrees with the shadow server concept. &ldquoSiemens recommends that customers keep their automation networks isolated from their corporate IT networks if at all possible,&rdquo says Todd Stauffer, Manager Product Marketing, Siemens Energy & Automation. &ldquoIt is also recommended that a dedicated computer be selected within the system to perform all software imports, as well as introducing externally-engineered data.&rdquo

RELATED STORY
What do vendors say about process control security?

Find more Articles, White Papers and Industry Books on control system security listed at the bottom of this article.

Working Inside a Fortress
Although complete isolation sets up a fortress, the logic is sound. Most of the threats arrive electronically, via networks, wireless or the Internet. Eric Byres, a member of the research faculty for Critical Infrastructure Security at the British Columbia Institute of Technology, says 70% of the security threats to process control and SCADA systems are external. Deloitte & Touche says that 90% of security breaches in financial companies are external. Therefore, our users and vendors say to disconnect yourself from all networks, and eliminate the major source of problems. Besides, why would you need access to the Web to control a process? &ldquoThe most secure systems we have are those that are not networked to anything else,&rdquo says Mike Larocca, control engineer at Solutia in St. Louis, Mo. &ldquoThese tend to be older systems.&rdquo Older legacy control systems are fairly immune to external attack, mainly because their networks are proprietary. &ldquoOlder systems usually were not connected to Internet protocol networks,&rdquo says King. &ldquoThey were usually proprietary in their communications protocols and as such were difficult to connect to other systems. This, however, did not make them more secure in nature. There were just fewer opportunities for systems to be attacked.&rdquo In other words, you really have to know what you are doing to get into a legacy system from the outside. You might want to reconsider replacing that old legacy system, especially if it is still working well. &ldquoUnfortunately, hardware support issues usually dictate replacement sooner rather than later,&rdquo notes King.Disconnecting yourself from networks does pose one problem: What about all the modem connections on your equipment that allow vendors to perform remote maintenance, software upgrades and diagnostics? &ldquoProtection can only be enforced if the end user institutes a âDo not connect&rsquo policy and periodically verifies that rogue modems or high speed internet and non-DCS LAN connections do not exist,&rdquo says Leimbach.Just about every piece of equipment in your plant purchased in the past few years has a modem connection. All of these seemingly innocent &ldquorogue&rdquo phone connections are, in fact, a &ldquoback door&rdquo into your system. Byres reports that the Slammer worm infiltrated at least four different control systems last year, and one of them got into a paper machine&rsquos HMI via a dial-up modem. Remember, virus writers are not targeting you specifically. They are just looking for any vulnerable port, and they scour cyberspace relentlessly and automatically. Someone like me, who sits on a cable modem eight hours a day, gets hundreds of port probes a day. Any system connected to any network will experience the same. You must devise a system that denies access to your system via back-door modems, and permits outside vendors to call up their equipment only under carefully controlled, supervised conditions. This isn&rsquot going to be easy, warns Joe Weiss, a security consultant at KEMA. In his testimony before Congress, Joe Weiss said you may not know about all the phone lines in your plant. He says an audit at an electrical utility turned up 100-200 phone lines in power plants and substations that were not owned by the utility. &ldquoThese phone lines were owned, installed and paid for by control and diagnostic system vendors,&rdquo he explained. &ldquoSince the phone lines belonged to the vendors, (they) were not identified. This is a common occurrence on many control system implementations.&rdquoThen there are all the new wireless systems with handheld PDAs that let control engineers and techs wander around the plant. What to do with wireless? Hook it into the shadow server, of course, where a security breach can&rsquot hurt anything. Finally, there are all the wonders of the Web, such as remote tuning and loop analysis software, maintenance management packages, batch management software, manufacturing execution systems, and so on, all of which need access to real time information. What to do with them? Hook them into the shadow server, too.Lock Up the Hardware
A few years ago, an engineer explained to me how control systems in some Third World countries are installed. All the hardware is installed in 19-in. racks behind padlocked bars, he said. The racks are in locked cabinets, and the cabinets are in a high-security room with a locked steel door inside a secure building. The only access operators have to the system is via HMI terminals (not PCs) in the control room, which is located elsewhere in the building. The purpose is to keep unauthorized personnel from tinkering with control settings, but the technique improves security, too. That&rsquos because even a system that is disconnected from networks is still vulnerable to software brought in by operators and technicians. &ldquoThe concern I am wrestling with is operators bringing in homemade CDs to listen to music on control system PCs,&rdquo says one control engineer. &ldquoWhat else are they bringing with them? I have found card games and the like on some computers after a slow weekend.&rdquoWeiss agrees. &ldquoAt least one facility with no external connections suffered a forced outage when a controls technician brought in an infected disk with games that shut down the plant,&rdquo he adds.Larocca says they limit access to prevent such incidents. &ldquoWe keep computers and controllers in locked or otherwise protected rooms where access to CD and DVD drives, floppy drives, USB ports, and so on is limited,&rdquo he explains. &ldquoWe also keep network hubs, switches, routers, etc. in limited access areas.&rdquoSiemens' Stauffer said that "automation system owners should also implement a standard operating procedure for ensuring that only authorized individuals have access to the automation system data. This policy should include user administration procedures which are based on Windows security (such as password expiration and lockout after number of retries) and controlling access to project data stored on the hard drive. To further prevent unauthorized access to the automation system, the key assets such as controllers, PCs, servers, and engineering workstations should be physically isolated and protected in a locked room. Additionally each controller has a physical switch can be enabled to prevent downloading of unwanted configuration changes."

RELATED STORY
What do vendors say about process control security?

Find more Articles, White Papers and Industry Books on control system security listed at the bottom of this article.

Don&rsquot Upgrade or Install Patches AutomaticallyMicrosoft says it is vital to install all the security patches they develop to keep ahead of the bad guys. &ldquoIf you are running Microsoft's most recent operating system, Windows XP SP2 or Windows Server 2003, which contain significant security enhancements, you've already taken a huge step toward reducing your security vulnerability,&rdquo says Ron Sielinski, Senior Industry Technical Strategist, Manufacturing Industry Unit, Microsoft. "Keeping your operating systems patched via Windows Update is equally important.&rdquo&ldquoStay current,&rdquo says Mike Pursiful, security analyst at Cryptek. &ldquoBy far, the greatest number of non-trivial intrusions, interruptions and systems disasters happen in environments where components are forgotten, out-of-date, and unpatched.&rdquoMicrosoft told us the latest versions of Windows are designed for better security. Bill Gates himself made security a number one priority. That may certainly be true. However, some engineers advise against making patches. Weiss says that traditional methods employed by IT departments, such as installing patches, don&rsquot always work in a process control system.&ldquoIT security policies and technologies used to secure traditional IT systems can potentially impact control systems if applied without understanding or inappropriately applying them to the control system environment,&rdquo says Weiss. &ldquoAutomatically implementing security patches on control system workstations can, and have, shut down control systems.&rdquo If you have isolated your control system into a fortress by disconnecting all the networks, you may not need to install patches on the control system PCs, but you probably should put them on the Shadow Server. &ldquoA key benefit of isolation with Microsoft based systems is that you then don't have to install every little security patch that Microsoft releases,&rdquo notes Larocca.You can be a little judicious about what patches you apply. Microsoft's security patches can be a real headache, particularly if they have not been tested with your control software.&rdquo&ldquoThis is true of any operating system used in a control system,&rdquo notes King. &ldquoHowever, ignoring security patches may be at your system&rsquos peril.&rdquoThe older the operating system, the less you may have to worry about patches. &ldquoUsers have talked to us about the operating system paradox that showed itself when the Sasser Virus was launched in 2004,&rdquo says Todd Stauffer, Manager Product Marketing, Siemens Energy & Automation. &ldquoThis virus attacked only the newer Microsoft operating systems, such as Windows 2000 and Windows XP, but left Windows NT alone. This means that users of older, seemingly less secure, operating systems were actually less vulnerable to threats since hackers do not typically target older operating systems.&rdquo&ldquoHave a well-defined policy for immediate testing of new Microsoft Security patches and Virus scanner profiles and for notification of testing results,&rdquo advises Stauffer. &ldquoHave a well-defined policy regarding whether new Microsoft Security patches can be installed as soon as they are available, or whether users must wait for compatibility test results by the host vendor.&rdquoInstall Firewalls Everywhere
Entire technical articles have been written about firewalls, and many process control vendors seem to base most of their security advice on firewalls. A hardware firewall sits between two networked devices such as between the shadow server and the control system, or between the IT network and the control network and monitors network traffic.&ldquoIf you have a strategic need to connect networks, make sure the corporate network is protected by a firewall between it and the internet,&rdquo advises Cinergy&rsquos King. &ldquoThe control system network should then be protected by a firewall between it and the corporate network.&rdquoEsssentially, a firewall examines the data and decides if it meets your criteria. Eric Byres defines three kinds of firewalls:

Packet filtering: Compares header information, including IP addresses and TCP port numbers, in each packet against a set of criteria before forwarding the packet.
Stateful inspection: Filters packets at the network layer, determine if session packets are legitimate, and evaluate contents at the application layer. Also called Dynamic Packet Filtering.
Application proxy: Examines packets at the application layer and filters traffic based on specific application rules. (For a more complete description, see Eric Byre&rsquos White Paper.)

In a nutshell, you can configure a packet to pass only specific kinds of information from specific sources. For the shadow server, for example, you probably want to establish criteria that carefully defines the kind of information that the shadow server can send to the process control system, limiting it to process control-related data in a narrowly defined secret format. This may require software on both sides to format the data properly.

It is highly unlikely that a hacker, even if he penetrates the shadow server&rsquos security, would know the secret format for talking to the control system. If he does know the format, then you are dealing with a truly determined, knowledgeable opponent, and that is beyond the scope of this article.

A firewall will stop most inappropriate traffic, if you set it up properly. But that&rsquos the key: you must set it up. &ldquoJust putting a firewall between the process control network and the rest of the network, without configuring it to know what data is essential and what is not, could waste time and money, without adding protection,&rdquo warns Invensys&rsquo Rakaczky. &ldquoEven decisions of where to implement firewalls must be policy-driven. Our cybersecurity consultants, for example, typically break out the following security zones: the public Internet, the data center, the plant network, the control network and the field I/O zone, and then deploy different brand firewalls between each. In addition to working out these configurations, some installations also require protection for sub zones.&rdquo (See the Invensys White Paper for examples of sub zones).

&ldquoWe segregated the control system from the corporate network by installation of a fully managed and monitored firewall,&rdquo says our anonymous digital security supervisor. &ldquoFully managed and monitored means that a security company examines the log files from the firewall every five minutes continuously and alerts us to any suspicious activity.&rdquo

Be careful, though. Many DCS vendors base much of their security on firewalls. Or, as one control engineer put it, &ldquoFirewalls are the best you can hope for from most vendors.&rdquo

As Rakaczky points out, you may want to put different firewalls in different locations in the system. For example, you could install shadow servers in various locations, including remote sites and individual process units, that would allow for remote multiplexing, access by vendors to equipment, and wireless access.

&ldquoKeep in mind that every access port that is allowed through the firewall is another potential security hazard, just waiting to be exploited by someone,&rdquo cautions King.

RELATED STORY
What do vendors say about process control security?

Find more Articles, White Papers and Industry Books on control system security listed at the bottom of this article.

Outside the FortressIf you subscribe to the Fortress Mentality recommended here, and plan to set up a shadow server, then all the conventional protective devices firewalls, anti-virus software, passwords, and so on should be applied with vigor to the server. The operating system should be one of the vendor&rsquos latest, and it should be kept updated with all necessary fixes, patches and service packs. Listed below are White Papers at ControlGlobal.com that go into great detail on anti-virus software, password protection, operational procedures, and other IT techniques needed to protect computers that are connected to external networks. Some of the White Papers address specific control systems so, if you have one of those, you may want to read what your vendor has to say very carefully. You may also want to ask your IT department for help in protecting the server. But be careful. As Joe Weiss warns, IT departments working without Control System Staff can do more harm than good: &ldquoPerforming system-wide diagnostics, maintenance, and/or scans can and have shutdown control systems,&rdquo he says. &ldquoPerforming penetration testing of control systems can, and has, shut down control systems.&rdquo What&rsquos worse, Weiss says, even after control system people complain about the tests, IT wants to continue performing them. IT can help if you work with them on process control systems. William Collins, control engineer at Constellation, says that IT can help. &ldquoOur IT security ran a routine vulnerability scan and found a hole in our Unix servers on the control system and knocked them down,&rdquo says Collins. He says they told IT to stay out of their control systems, but he still goes to them for help. Together, they developed a private process network with limited access to the corporate intranet. Even so, Collins offers this advice: &ldquoIf at all possible, stay disconnected from your corporate LAN.&rdquo Collins does not feel particularly secure. &ldquoOn a scale of 1-10, I&rsquod say I am a five,&rdquo he laments. &ldquoI am running Unix, which seems to be less of a target. So I feel protected from the outside, but am open for inside attacks.&rdquo &ldquoWe have experienced very few security problems, and we have no evidence of specific directed attacks,&rdquo The security advisor reports, &ldquoThe activity we observe most is from non-directed Internet worms.&rdquoAppoint a Security Czar
Except for setting up a fortress that limits access, there is no hardware or software that prevents attacks by a disgruntled employee, a bad guy that got inside your company, or even a well-meaning operator or technician that makes a mistake in configuring a controller, maintaining a device, or loading up tainted software. Such problems are dealt with by setting up operational procedures, passwords, levels of access, approval processes, and security rules and regulations. This requires a corporate commitment. &ldquoMake the integrity of your systems a business responsibility and a priority,&rdquo says Pursifull. &ldquoUnless someone is explicitly responsible for this -- and empowered to act or establish procedure -- it will not get done, except perhaps sporadically.&rdquoIn other words, appoint a security czar and give that person responsibility for control system security, enough authority to command respect, and a budget to carry out the task. Much is happening on the security front these days, and it takes a full time person to track down all the standards, recommended procedures, and approved hardware to keep your system safe. A few of the buzzwords, organizations, standards and coalitions flying around these days include Common Criteria EAL 4, FIPS 140-2, ISA 99, CIDX Cybersecurity Initiative, MS-MUG, TPEP B2, DTTS CAP, and more. Maintaining security on a process control system is a full time job, requires constant vigilance and training, and deserves a corporate commitment. The tips and techniques presented here are just the tip of the iceberg. Top ten defenses

Get off all outside networks. Disconnect your system from the corporate LAN, the Internet, wireless, and all networks not needed for control.
Allow access to the Web, ERP, IT and external systems only from a Shadow Server, which is connected to the control system through a firewall.
Eliminate all &ldquoback door&rdquo and &ldquorogue&rdquo modems. Develop a plan for allowing vendors to access their equipment remotely only under your direct supervision.
Lock up the hardware. Put all control system equipment, such as servers, routers and disk drives, in locked cabinets in a locked room and limit access.
Don&rsquot install patches, upgrades and service packs without careful thought and testing.
Reconsider replacing legacy systems that have no security problems.
Install firewalls everywhere.
Enlist the aid of your IT department. But don&rsquot let them anywhere near the control system without your supervision.
Protect the Shadow Server with firewalls, virus checkers and security updates.
Appoint a Security Czar with authority, power and a budget. Develop a control system security policy backed by senior management. Enforce it.

SECURITY WHITE PAPERSWant to read more about control system security? The following are White Papers on network security, procedures, firewalls, and other security topics.