Traffic cops keep watch
Of course, the ultimate aim of any cybersecurity effort is the same as any other plant-floor initiative from basic loop control to advanced process optimization and safety—keep the application running as efficiently and profitably as possible. However, because there's no "set it and forget it" with cybersecurity due to constantly evolving probes and threats, a secure network and the communications traffic on it must be constantly examined for anomalous performance that could indicate unauthorized and possibly malicious activity. Earlier networking monitoring tools like IT-based simple network management protocol (SNMP) and related derivatives have given way in recent years to passive-monitoring software like SIEM that are less likely to hinder operations.
"It's OK to use SNMP, but you have to be careful because we can't have active network scans, which will lock up many legacy devices on most plant floors," says GPA's McNeil. "We try to use SIEM software wherever possible because it's passive; comes in many flavors; has antivirus and anti-malware capabilities on top; uses mature signatures and rules-based authentication; and can incorporate machine learning and artificial intelligence functions based on behaviors. This is really the future of cybersecurity because behavior-based software monitors internal data traffic—where it's from and where it's going—but it adds devices on a network to establish its normal, permitted traffic pattern, which lets it identify anomalous traffic and deliver alerts."
McNeil adds that GPA often uses SIEM software from AlienVault, works with machine-learning provider Darktrace, and can integrate these tools into users' networks. "We work with clients to examine their risks; establish their security policies and procedures; and set up education, training and enforcement."
Enlisting further help from the IT side, Matrix's McKarns reports, "Microsoft Active Directory is used to enforce security and access control policies on domain joined objects. It gives administrators one pane of glass from which to view and manage permissions for all users in a control system. It's the same tool that's been used in corporate enterprise networks for more than 20 years, and now administrators on manufacturing networks can centralize monitoring and management of their users, PCs and network devices, and apply policies to those users and devices allowing greater, more centralized control. Typically, joining machines to a domain wasn't done on plant floors in the past, but it's becoming much more common in new deployments and in situations where IT has a stronger influence."
McKarns explains, if domain services are in place that restrict permissible actions, along with good firewall and intrusion detection policies, many virus and ransomware attacks wouldn't be possible. "These threats are why ISA and ICS-CERT put out guidance that it's not a good idea to connect plant control system PCs directly to the corporate network, and recommend using firewalls and DMZs between layered networks," he adds. "Another element of defense-in-depth is active monitoring of network traffic for anomalies and possible intrusions, and being able to accordingly. The next step is finding the right mechanism for intrusion prevention. Prevention requires an automatic, heuristic software tool, which includes a learning capability and understands typical network traffic; establishes a baseline; red flags unexpected traffic; provides alerts and alarms; and can disallow the flow of unauthorized traffic through key security points. Intrusion prevention systems must be configured and deployed carefully, so they don't block critical traffic."
Matrix's Lycans adds, "Evaluating network traffic was originally more straightforward, but outputs were not always clearly formatted, there was a lot of text, and it could be hard to understand where links were and what their status was. Traffic evaluation tools are more graphical now, and present maps of devices, links and switches. We've used IntraVue for many years, and we also use SolarWinds software that displays the corporate enterprise."
McKarns adds, "Today's network traffic scans aren’t just visualized better, they also benefit from being dynamic network scans in real-time that can better detect intrusions. However, where it used to be easier to scan many switches because their settings were open and unsecured, it's more difficult now because more of them have built-in security requiring a greater amount of configuration."
From defense to anticipation
help users better understand how cybersecurity works and protect their processes, Sid Snitkin, VP and GM of enterprise advisory services at ARC Advisory Group, advises users to follow its Industrial Cybersecurity Maturity Model, which moves from reducing the likelihood of intrusions and attacks to reducing their impact (Figure 2). Its levels include:
- Secure with physical security, asset inventory, device hardening and patch management;
- Defend with unidirectional gateways, demilitarized zones (DMZ), firewalls, anti-malware and access control;
- Contain with zone firewalls, ICS device firewalls and whitelisting;
- Manage with security information and event management (SIEM) software, and incident management solutions; and
- Anticipate with anomaly and breach detection methods, and threat intelligence.
"Security investments should have specific objectives, and lower-level goals should be achieved before advancing to higher-level goals," says Snitkin. Many users and organizations discover they've bought too much cybersecurity technology, and find they've got a lot more than they have the resources to manage."
Eric Knapp, chief cybersecurity engineer at Honeywell Process Solutions, adds that, "There are many ways you can seek to better understand security events and how vulnerable you are to the latest threats. At the most simplistic level, think about your security across three major categories: people, process, and technology. Then take any of the latest incidents, such as Wanna Cry or Petya, and scenario plan how they might impact those categories. You’ll quickly uncover gaps that you can address.
For example, we've seen that many companies that had hired managed security services teams to keep security patches updated were protected against WannaCry and experienced no negative impact from the cyber attack. Those companies had recognized they didn't have the personnel to access and qualify patches, administer updates, and keep them regularly updated, so they hired in service teams to address this gap. Breaking security into categories is a practical way to tackle issues specific to your plant, and help ensure you aren't over-investing in the wrong areas.
"Another way to understand how security incidents might affect your industrial company is to leverage technical solutions to automate visibility and control over your PCN risks. You can run specialized software that calculates your risk and rapidly visualizes it, for example. Honeywell has the Industrial Cyber Security Risk Manager that measures, monitors and prioritizes risk levels across your operational assets, then offers simple guidance so teams can act on tasks that lower risk."
Bake in, don't bolt on
Yet another major suggestion emerging from sharing on cybersecurity from many players is their call for security software and capabilities to be added during design and construction of industrial components, instead of seeking to add security functions to existing devices and systems.
For example, Vista Irrigation District near San Diego, Calif., recently upgraded its SCADA system, I/O modules and networking from serial to Ethernet—and improved its security posture at the same time—with help from the same two firms that installed them more than 20 years ago, namely Opto 22 and system integrator IDAC West in Sam Marcos, Calif. The district moves more than 5 billion gallons of water per year for its 127,000 customers through more than 435 miles of pipes linking more than 30 remote sites, including 12 reservoirs, source water connection points, pump stations and flow control facilities.
“The distributed intelligence of Opto 22's SNAP PAC let us prototype, test and rollout upgrades to one remote site at a time,” says Alan LeVezu, engineering manager at IDAC West. “We were able to run existing and new systems in parallel, and cut over to the new system when we were ready. We also used IDAC's research and development test center to design and prototype new features of the updated system, which let us verify that the new controllers, I/O, and updated control software were running as intended before cutting over."
Opto 22 reports that dual, independent, 10/100 Mbps Ethernet interfaces on its SNAP PAC controllers connect to separate IP subnets, which improve cybersecurity from the outset by isolating the control and business sides of an overall network. SNAP PAC also has an optional 802.11 a/b/g wireless interface with built-in WPA2-AES security. SNAP PAC also has Ethernet traffic monitoring that allow LeVezu to display network performance statistics between Vista’s headquarters and remote sites directly on an HMI for increased visibility that gives the district near real-time situational awareness at all sites.
"SNAP PAC lets us perform process control at each individual site,” adds LeVezu. “Because each controller and brain in SNAP PAC is intelligent, we can configure setpoints directly at each site. So, if the network goes down, those sites can continue to operate autonomously. With built-in Ethernet already on the controller, we don’t need to buy added Ethernet cards or upgrade the controller to a higher-priced model to add networking capabilities”
Tony Baker, cybersecurity portfolio manager at Rockwell Automation, adds that, "We're trying to take an embedded approach by designing cybersecurity into products, in addition to establishing a product security office, which identifies inherent risks, consults product teams as to how they can mitigate those risks, conducts penetration testing, and completes validations and verification testing for implementation. In short, cybersecurity is taking on a product quality assurance role, and we have to prove that competency."
Down to the ground
Beyond incorporating security into controls and networks when they're first designed and built, several experts warn that process controls systems remain especially vulnerable at the lowest levels of the Purdue reference model, primarily Levels 0 and 1, where sensors and other basic instruments are located. As more of these modules gain microprocessors and Ethernet connections, they can also be pathways for probes, intrusions and attacks, which can be as subtle as communicating incorrect readings with potentially disastrous result if they're misinterpreted or remain unchecked.
"The policy of setting sensors to 'fail as-is,' 'fail upscale' or 'fail downscale' is commonly done. However, if safety systems are involved, the approach of setting sensors to a fixed value can cause a loss of safety," states Joe Weiss, ICS cybersecurity expert and author of Control's "Unfettered" blog in his Aug. 18 post, "Insecure process sensors can create safety, security and resilience vulnerabilities." "An inherent assumption is that sensor values are correct even though process sensors have neither authentication nor security. Cyber concerns with sensors include cyber-vulnerable protocols and smart transmitter functionality that leads to cyber vulnerabilities. Currently, ICS cybersecurity assumes process sensor input is correct. If sensor values are incorrect either because of unintentional issues, such as sensor drift, miscalibration, etc. or by sensors being compromised by a cyber attacks, then resilience, safety and security can be defeated.
"Following a rigorous approach to understanding complex system behavior over the full lifecycle and under numerous threat scenarios will expose such risks and support mitigation decisions. It's also essential to consider a review and reexamination of failure modes in many of our vital infrastructure systems."
To combat the threat of spurious sensor signals, Eddie Habibi, CEO and founder of PAS Inc. reports that engineers and managers in the process industries must conduct an awareness campaign that brings cybersecurity even further into the realm of process safety, and go beyond network traffic monitoring and intrusion detection by adopting complementary security controls that address the lowest control system levels. Beyond traditional security controls, PAS advocates continuous monitoring of field sensors and actuators (Level 0 devices) using multivariate process behavior recognition technologies to identify anomalies.
"Foundational ICS cybersecurity must include device inventories, vulnerability assessments, configuration baselines, change management and recovery procedures," says Habibi. "Our Cyber Integrity software mitigates Level 0 and 1 vulnerability risk and engineering mistakes by identifying and inventorying the user's whole control system, including its field instruments, I/O points, valves, I/O cards, control loops, and its HMIs, historians and servers. Once it captures the configurations of all network and OT devices, we have a baseline that can be monitored for any unauthorized changes."
John Cusimano, industrial cybersecurity director at aeSolutions, adds that, "Previously, people had to be convinced to address cybersecurity. Now, they want to know how to get started. The market is a lot more sophisticated now that many users already have some cybersecurity in place and are trying to improve it. However, even though many users write security policies and audit their facilities, we always discover vulnerabilities when we perform assessments in the field, such as unsecure TCP ports. Typically, we find there's good segmentation from the business network to the process control network (PCN), but not a lot of segmentation within the PCN. I'd estimate that only about 25% of PCNs are segmented into security zones as recommended by ISA and other industry standards and best practices."
Located in Greenville, S.C, aeSolutions is a consulting, engineering and CSIA-member system integrator specializing in process safety, SIS and industrial cybersecurity. It reports that its aeCyberPHA methodology is a practical application of the ISA 62443 cyber risk assessment requirements, which link realistic threat scenarios with known vulnerabilities and existing countermeasures, and couple them with credible consequences from the process hazards analysis (PHA) to determine cyber risks.