Visit www.controlglobal.com/0811_carvingupsecurity.html to Check Additional Online Security Resources.
By Jim Montague, Executive Editor
“Can we talk?” If you’re a process control engineer trying to converse and coordinate efforts with an IT technician in the same company, or vice versa, the usual response is “not likely.” However, if you’re trying to discuss security problems, solutions and coordination, then the answer is probably “never” because potential lines of communication are even more closed off.
Sure, these two groups have been shoved closer by the emergence of software and networking on the plant floor, driven principally Microsoft Windows and all the non-proprietary and commercial off-the-shelf operating systems and related technologies that followed. However, just telling process control and IT to sit down and talk about security doesn’t mean they will—or even that they’re physically or psychologically equipped to do it.
Still, many end users are seeking ways to help their IT professionals and process control engineers to find common ground and cooperate to provide better security for their applications and organizations. In fact, several of the largest oil and gas firms have already formed combined process control and IT teams, and a few have even merged these into “digital security” departments to handle all security issues.
Poor Communications = Bad Surprises
Until these saviors go mainstream, however, many security efforts will remain hamstrung by miscommunications. For example, to avoid potential accidents and prevent possible intrusions, engineers at Husky Energy’s heavy oil upgrading facility in Lloydminster, Saskatchewan, knew they were going to need continual security improvement and monitoring. However, they lacked the knowledge to do it independently, and so they enlisted outside help from Integralis and Invensys Process Systems (IPS). Lloydminster can produce up to 46,000 barrels of synthetic crude oil per day (Figure 1).
Figure 1. Husky Energy’s heavy-oil upgrader in Lloydminster, Saskatchewan, uses third-party security support from Integralis and Invensys Process Systems.
“Cybersecurity management is an ever-changing field, and so it’s difficult to stay on top of its technologies and still manage our other responsibilities,” says Don Gilmour, of Husky Energy. “I wasn’t surprised by our audit’s results on the control network side, but I was surprised by the unacceptable ease of access from the business network to the control network.”
Process Control and IT Worlds Collide
The barriers to control and IT cooperation on security are formidable because, not only do the two groups have different histories and languages, but their core missions and even their very perception of time are 180° from each other. For example, software patches are pushed to IT departments on a monthly schedule, while much control equipment is expected to run for 10-15 years without much renovation.
“I think the biggest difference between IT security and manufacturing security is that control engineers start with protecting the process, and so they’re deeply rooted in the safety and reliability of their operations. In manufacturing, process availability and physical assets are paramount,” says Eric Cosman, engineering solutions IT consultant at Dow Chemical Co. in Midland, Mich. “However, IT security focuses first on data confidentiality and then on integrity and availability. These are very different perspectives, but the magic for them to work together is simply to learn to understand the other’s point of view, and the key to this is finding their common concerns. For example, engineers see physical devices as their assets, and IT sees data as its asset, but their common goal is protecting those assets.”
Eric Byres, chief technical office of Byres Security Inc. in Lantzvile, B.C., adds that, “Everyone knows that IT security and control security are different, but it’s time to get over it, and do so intelligently. This means getting IT and controls people to put their assumptions and what’s important to them on the table, and then create cross-functional teams so they can work together.
“The good news is that a major oil company I consult with recently gave its IT security standards and practices to its process control guys, and they reported that they could live with about 90% of the security policies that IT already had in place, such as staffing, contractor and training practices, human resource policies and incident-recording procedures. So, traditional IT and process control assumptions about security only affect about 10% of what’s going on in most process applications, but that’s the fraction on which the team really has to focus. This 10% usually includes issues like password management and software patching policies, as well as firewall and network design. In fact, as the cross-functional teams at two other oil and gaa firms developed joint security policies, they also merged from separate control security and IT security departments into one overall digital security department.”
Cooperation and Coordination
Cosman reports that Dow always thought of security as important because it’s so closely tied to basic safety. “What changed in the past 10-15 years is that commodity software and IT equipment entered the plant floor and control space,” says Cosman. “We couldn’t manage these new technologies with engineering, so we needed IT expertise and people. We also found our existing networks were more commingled than we wanted, and so we needed logical separations and function limits.”
To help their controls and IT staffs cooperate on security, Dow’s engineering vice president, chief information officer and others began meeting a few years ago to sketch out roles for its company-wide Manufacturing Control Systems Security program. They sought and secured management’s commitment to the program, and then hired three staffers, including one with an IT security and auditing background, one with manufacturing and plant experience, and one with an engineering, R&D and security background. “We actively look internally for people who have the potential and willingness to work on both the control and IT sides,” adds Cosman. “One person may have a degree in computers and an aptitude for engineering, while another is a veteran engineer who can learn computing.”
In fact, with Cosman as coach, this team has implemented Dow’s security program at about 600 facilities worldwide over the past three years. This includes designing and developing firewalls, working with local IT personnel, engineers and contractors, designing and implementing industrial networks, and monitoring systems when they’re running to check for possible intrusions or problems.
“To get people to get along, we found they had to be mixed up, and then forced to live in each other’s space,” adds Cosman. In fact, three years ago, Dow’s controls engineers asked its internal PC auditors to evaluate the PCs in the firm’s process control systems. “Instead of hiding, we headed the problem off at the pass, and invited our IT auditors in,” he explains. “Many people though we were crazy, but IT learned how controls work and their priorities and needs. And, we in controls learned how to make our PCs more secure.”
To resolve the old availability versus confidentiality dispute, Cosman says Dow’s joint process control and IT team discussed what was most important in each security-related circumstance. For instance, keeping a critical process running usually is more crucial than preserving data, so this indicated where Dow needed to segregate data. “We put in firewalls, so we’d only pass essential information,” adds Cosman. “Since then, when we connect networks, we do it very deliberately. And we only connect to the network when it’s warranted by specific production and business goals.”
Practical Patch Management
Probably the most contentious security issue for controls and IT is how to handle patches or other software updates. Disputes arise over policies for verifying user identifications, granting access, handling viruses, detecting malicious software or other situations, but none seem to spark as many arguments as patches—perhaps because they arrive with such annoying frequency. Many are pushed out by Microsoft on the second Tuesday of each month.
Consequently, IT departments are accustomed to accepting and distributing patches. However, control engineers usually can’t implement a patch until it’s been tested to make sure it won’t adversely affect or damage their process application. Nate Kube, Ph.D, chief technical officer at Wurldtech Security Technologies in Vancouver, B.C., says communication policies and procedures for identifying vulnerabilities and managing patches are still being hashed out by most IT security groups, but their control security counterparts still can’t deploy the protection that the patches provide. “IT has its firewalls, intrusion detection systems (IDSs), intrusion protection systems (IPSs) and other security gear, but these devices are missing the intelligence and rule sets they need to function in process environments,” says Kube. In fact, Wurldtech creates resilience profiles for each device it tests, and these include a list of vulnerabilities, safe operating parameters and intrusion detection signatures and firewall rules sets. “This allows users to take their firewall, load it with these rule sets, and give that firewall the ability to operate in a process control application by blocking traffic that would otherwise trigger vulnerabilities in the control device,” he explains.
Successful Outsourcing
To jumpstart its own security effort, Gilmour reports that Husky first defined the project, and sought approval from its DCS supplier. “It became apparent during the audit that we were going to need more than firewalls, so we got our IT groups to assist, and decided to use IPS and Integralis for support,” says Gilmour. “We needed a definite division between our business and control networks because the IT group wasn’t about to let us have control over the rules in the IT firewall, but the rules they did have might cause problems in the control network. By implementing the firewall on the control network side, the IT group could control the traffic on their network and we can control the traffic on our network. And, because we can’t monitor security devices 24/7, we enlisted a third party to manage the security devices, immediately handle any threats and problems, and keep our security hardware and software updated.”
Husky’s new security system includes a multi-zone DMZ architecture, proxy/relay zone for external communications, and a network-based intrusion prevention system (IPS). Also, patches are reviewed by the support service, and Husky is notified for approval before any are applied. Likewise, security escalation procedures were defined and tested, and security policies and business rules were set up and tuned during deployment. “Since then, the new security system has monitored and managed Husky’s infrastructure, handled multiple patches and assessed and applied signature and rules changes,” adds Gilmour. “And, we’ve had no outages!”
Joint Security To-Do List for Controls and IT
Eric Cosman, engineering solutions IT consultant at Dow Chemical Co. in Midland, Mich., says there are several primary tasks when undertaking a joint IT and controls security project.
- Secure management support and approval for unified security upgrade project
- Perform situational analyses of affected processes, application and facility
- Settle on definition of what constitutes good security for each process and facility
- Complete gap analyses that describes how to achieve the desired security functions for those applications
- Implement new security software and components
- Assess unified security system’s performance, and reassess it regularly
Leaders relevant to this article: