By Jay Abshier, CISSP CBCP
, and Joe Weiss, PE CISM
, KEMA Consulting
AS CONSULTANTS, we often hear from IT professionals that their Operations counterparts won’t cooperate, and that their systems aren’t secure or in compliance with corporate standards. They tell us they fear that Operations will cause their systems to be attacked.
From Operations, we hear that IT doesn’t understand their business or their constraints, and insists on steps that will negatively impact operations. Operations also fears IT will cause their systems to be attacked.
In fact, at KEMA’s 5th Annual Conference on the Cyber Security of SCADA and Process Control Systems there were several presentations that discussed this subject, resulting in heated discussions during and after the conference. KEMA is a consulting, testing, inspection and certification firm for energy companies.
So, who’s right? Before trying to answer this question, let’s understand the two sides’ different perspectives, starting with a review of Operations’ background, and why this is even an issue.
Operations: The Early Days
Until recently, and in some shops to this day, the computing infrastructure for the control system environment was totally proprietary, and usually based on serial communications. This infrastructure was so different that very few IT organizations had the skills to support Operations. However, control system devices increasingly are being delivered or upgraded to support the TCP/IP protocol, and gateways have been written for the arcane serial protocols, so they can be encapsulated in TCP/IP.
"The inconvenience of having a disgruntled employee commit sabotage, or a worm running rampant, is far greater than the inconvenience of most security controls."
In 2003, the Energy Management System User Group surveyed 16 utilities about whether SCADA was “owned” by Operations or IT, and which provided computer and network support. The results were mixed, but a majority reported they weren’t part of corporate IT, and didn’t get support from IT on any EMS tasks. These mixed results are consistent with the informal responses received from many different utilities and other industrial organizations.
Many of the SCADA and power-plant operator/engineer workstations, as well as substation and power-plant laptop computers, appear to be the same as those used in traditional IT business systems, despite the fact they have very different applications and remote connections. Consequently, Operations’ computing infrastructure is starting to look a lot like corporate IT infrastructures. In fact, TCP/IP connections are being installed between the two, and sometimes to networks beyond, including the Internet.
Consequently, IT is taking an interest in Operations’ network, and understandably wants to treat it as part of the corporate network because they’re now connected. Also, there is often a sharing of IT infrastructure such as LANs, firewalls and routers by Operations and IT.
However, IT often lacks knowledge of operational and administrative control systems’ differing needs. Quite often, the conversation between IT and Operations about this subject ends in conflict. Now, lets examine some reasons why this conflict exists.
IT shops try to protect an environment that is constantly chaotic and changing. New devices constantly are added, new users and new applications make demands of bandwidth and firewall rules, and IT constantly worry about keeping out worms and viruses.
To accomplish this almost impossible task, IT fought long and hard to establish ground rules that everyone must follow. These include requiring each individual to have a unique user ID (without which there’s no accountability or verification of actions), requiring complex password construction (it takes only about 3 sec to break a simple password), and enforcement of rigorous patch management procedures (to make sure devices aren’t vulnerable to worms or viruses in case one gets through the perimeter). These are just a few of the many requirements that are basic to protecting corporate environments.
Implementing these requirements isn’t easy for IT or the user community. While the goal of IT, and IT Security in particular, is to not interfere with business objectives, it’s impossible to implement these requirements without some inconvenience to users. However, the inconvenience of having a disgruntled employee commit sabotage or a worm running rampant is far greater than the inconvenience of most security controls. Obviously, IT is used to meeting resistance when trying to implement inconvenient practices that are necessary for the welfare of the company and its individual users. Non-compliance is really justified very rarely. Even when it is justified, IT must worry about how to protect vulnerable devices and users.
Without Operations, it’s likely your company wouldn’t exist. Granted, other business functions add value, and some even generate income. However, without Operations generating a product–whether it’s food, paper, toothpaste, electricity, potable water, oil, refined fuels, natural gas or chemicals–none of the other business functions would have products to transport, trade, or market.
In essence, money generated to fund most other business units depends on the margin between Operations’ budget (to buy raw materials and generate product) and the products’ sale price. Therefore, Operations is pressured to keep its budget low, sometimes to a greater degree than other divisions. Consequently, one of Operations’ primary goals is stability. To achieve it, the lifespan of control systems is usually measured in years and, many times, in decades. Also, all changes to the systems, including software updates and security patches, must be approved by the vendor(s), and then rigorously tested by Operations before being applied. Therefore, the lag time from when updates or patches are available and when they can be implemented is weeks, at least. In fact, some systems are version locked due to the very expensive applications they run.
Also, many of devices critical to the control system environment don’t support “security.” Sometimes, there’s only one user ID, if they’re supported at all. And, if passwords are supported, they’re generally alpha-numeric characters, or sometimes just alphabetic characters, which are transmitted in plain text.
To make matters worse, operators controlling the processes often must respond rapidly to a situation, before it becomes a crisis. This requires the window onto the process being controlled, the operator console, to be logged on with one user ID that is shared by all the operators, while items such as screensavers and idle time lockouts are forbidden. Due to limited staffing, separation of duties is often impossible in control environments. For instance, control engineers often doubles as system administrators.
Part of the conflict between IT and Operations may be due to vague terminology. We’ve repeatedly stated that control systems must be a “more secure” environment, but what does this really mean?
IT looks at that statement and asks, “How can that be? They allow people to share user IDs, security patches haven’t been applied, and passwords are hardly ever changed.” From the IT perspective, until control systems embrace current best practices, including quick installation of security patches, complex passwords, and individual user IDs, to name a few, not only will the control systems not be secure, but they will remain a threat to the rest of the organization.
When IT hears that the control environment must be more secure, they correctly think it must be security hardened, according to IT’s best practices. When Operations hears the control environment must be more secure, they correctly think it must be protected from the outside world. Therefore, in the conflict between IT and Operations, it’s entirely possible that, from their different perspectives, they’re both correct.
How to Resolve Confusion
Perhaps we should say that greater diligence needs to be applied when protecting the control environment? Because the control environment will never be “security hardened” in the foreseeable future, the perimeter of the control environment must be defined, identified, and isolated from outside networks. Strict controls should be placed on connections allowed and on the methods of data transfer in and out of the control environment. We all should recognize that devices in the control environment are and will remain vulnerable to hacks, worms, and viruses. Therefore, appropriate steps must be taken to ensure malicious code is never introduced.
Perhaps we should define trust models? The control environment should never trust anything coming from an external environment. All data transferred into the control environment should be verified. All devices should be scanned for malicious code before they’re allowed to connect to the infrastructure. For example, these would include vendors’ and visitors’ laptops. Remote access connections should terminate in a DMZ separating corporate and control environments. And, since the control environment can’t be hardened, perhaps the corporate environment shouldn’t trust it as well.
Above all, we should educate and work with each other. IT should learn what control systems do and the constraints on them before requiring vulnerability scans be run against control devices, or that Operators have individual user IDs and Operator Consoles have password-protected screen savers. Operations should learn about the problems IT faces in protecting the corporate environment, and recognize that IT wants to make changes because vulnerabilities need to be addressed.
Everyone should work together to look critically at vulnerabilities, and devise methods that will mitigate the vulnerabilities, while not interfering with critical business functions. We need to keep in mind that sometimes procedures, processes, and physical security can mitigate vulnerabilities when a technical solution isn’t viable. We need to develop a joint effort where IT does what it does best and Operations does what it does best.