Elwart reports that his Systems Engineering department manages a complex and growing network of about 250 computers, a dozen SCADA servers, as well as other servers and related equipment used by Ergon's refining, fleeting, retail and some corporate departments. "In the past five years, we've designed everything with security in mind. Before that, all we could do was limit the access points into the network, but we also learned that you can't make a 25-year-old process control system inherently secure" says Elwart.
So two years ago, the department also began expanding beyond Ergon's existing Foxboro I/A distributed control system (DCS) installed in 1987 by adding Emerson Process Management's Delta V, so now the plant consists of roughly one half of each system. Over the past 15 to 20 years, Ergon also acquired a variety of PLCs from Allen-Bradley, Triconex and Siemens, so it's also working to integrate them into its expanded DCS and create a better interface into these controls with a combination of Modbus, Ethernet, serial communications and OPC.
Besides continuing to severely restrict external access, Elwart adds that his department also makes sure not to use administration-level accounts for routine system tasks.
"I tell people that before asking their managers for money for process security, they need to do an internal survey of all their networked equipment, perform a risk assessment (RA) for each device, evaluate its criticality, prioritize each item and limit its access to the outside world," says Elwart. "We've done it informally over the years, and we're still doing it."
Cooperation and Common Sense
Besides upgrading components, software and networking, Elwart emphasizes that one of his most powerful security tools is coordinating efforts with his overall corporate IT department. This cooperation may come a little easier at Ergon because Elwart trained as a chemical engineer and worked as a process engineer before migrating over to the computing and software side. Likewise, he adds, his Systems Engineering department has a unique blend of business and controls skills because most of the 9 to 10 staffers previously worked in completely different areas.
"IT people already have network security in mind, but we also make sure they get basic operations training, too," says Elwart. "I want everyone in our department and in IT to understand how our refinery runs 24/7 and why those calls at 2 a.m. are a big deal. This training makes everyone more sensitive to shift changes, scheduled downtimes and production runs. As a result, when control engineers open some ports between the network and control level to get some equipment to work and then leave them open, then IT can follow up and shut down the unused ports." Conversely, some control engineers and operators also get trained in business-level and SCADA security practices.
Elwart adds that the Systems Engineering and IT departments also jointly confer on scheduling upgrades and other projects to avoid causing production problems. "For example, while other IT departments may add enterprise software over a weekend and create some difficulties, we look at what runs and turnaround times are coming up and try to find the best upgrade times," explains Elwart. "For instance, we won't change Internet Protocol (IP) addresses on a terminal server on Friday, so we can avoid possibly locking out some engineers over the weekend. We also won't change IP addresses on Monday to prevent people from complaining that they didn't see the email sent on Friday. So we usually make any IP address changes sometime between Tuesday and Wednesday."
Likewise, Ergon also waits until its vendors test new software patches and then installs them during normal downtime. "This is because we've seen more operating interruptions due to spurious shutdowns caused by untested patches than we have from what they're supposed to be protecting us against," says Elwart. "If there's an emergency, we'll call the vendor in to help install a patch."
Elwart reports that other common-sense security procedures used at Ergon include locking its server room so no unauthorized people can plug into the servers inside. Similarly, the Systems Engineering department also changes the default passwords on all its devices, which removes one of the most typical pathways used by outsiders to gain unauthorized access. It also uses several levels of authentication, which means different passwords for different network areas. Ergon uses Cisco firewalls to protect its networks, but it doesn't set them and forget them. It implements well-thought-out rules that will meet Ergon's needs when it configures its firewalls and then constantly updates them.
"Management at some refineries will sometimes say, ‘We're nice people. Why would anyone want to attack us?' So they need to be shown that pretty much every industrial facility and application with a network connection has had some kind of probing or potential attack that could be a problem," adds Elwart. "I recently learned about a refinery network that had been under a continuous attack for several months. This was a brute-force attack in which the intruder was trying up to two user names per second to gain access. I believe the application's system was able to show where the attacks were coming from, and the unsophisticated hacker was actually caught."
Elwart adds that one of the main lessons he's learned from trying to improve Ergon's process security is that people can't be prevented from trying new technologies. "IT sometimes tries to tell people not to use a certain technology, but this doesn't work because many will ignore those instructions," says Elwart. "It's much better to embrace new technologies and then learn to manage them effectively. Much of this comes down to simple education. We always talk about hackers, but many security problems also come from inadvertent mistakes. For example, another refiner found some unexplained photos on a desktop and found they'd been accidentally downloaded to the PC when one of the cleaning crew plugged into a USB port to charge his smart phone. So they educated the cleaning crew not to do that sort of thing, and then also configured their PCs not to perform automatic downloads. People are willing to go along with rules if they're told why those rules are important."