More connections mean greater vulnerability. Unfortunately, that's just the way it is for now. As digitalization, cloud computing, remote access and monitoring, the Industrial Internet of Things (IIoT) and all other mostly Ethernet-based networks continue to expand and multiply, they bring along the risk of even faster evolving probes, intrusions and cyber attacks (Figure 1).
Fortunately, many of the same cybersecurity tools, policies, software and procedures that protect less-extensively digitalized applications are just as useful for protecting those with more connections. At the very least, users can employ these basic cybersecurity methods as a starting point to which they can add more sophisticated measures.
"Digital transformation is top of mind, but practical cybersecurity has to be part of it, including assessing the present environment, integrating security functions, and continuing to understand your capabilities and posture," says Michael Lester, director of cybersecurity, governance and architecture at Emerson Automation Solutions. "This begins with a security risk assessment (RA) of each process application and setting their potential risk, and how to manage them within a cyber risk management framework."
Once a security RA documents an application's particular set of risks, the resulting remediation roadmap may advise going beyond initial password, network segmentation and two-factor authentication to call for added software patching, network access and advanced cybersecurity review for any affected cybersecurity controls. For example, Emerson's DeltaV DCS has long had patching for its control software, while its latest Version 14.3 will be certified as compliant with ISASecure's System Security Assurance (SSA) Level 1 and also use McAfee security solutions.
Figure 1: Rapidly evolving cyber probes, intrusions and attacks can come from different sources and seek to breach multiple areas of process control and business-level networks. These problems are compounded by the fact that many legacy control systems weren't built with cybersecurity in mind, and because the Industrial Internet of Things (IIoT) is creating more links and attack vectors, according to "The Cisco Connected Factory" whitepaper, available here. Sources: Cisco and Dell
"Users must prioritize the actions they'll take in response to identified threats and attacks, and create a roadmap of those actions as part of developing an overall cybersecurity posture and supporting procedures. This includes incident response and backup/recovery plans for all the people, processes and technologies in their organization during the lifecycles of each," adds Lester. "Even something simple like user account management must address each user's lifecycle from when they're first authorized to when they leave the organization."
Modernize and mitigate
Because strengthening cybersecurity typically means adding new software, components and services, one of the best times to implement them is during scheduled downtimes or when migrating to upgraded controls.
As part of its own digitalization journey, Campbell Soup Co. has been modernizing its plant floors and making them more secure, robust and reliable. It's also been moving to industrial Ethernet, employing more smart devices at the automation layer, and needing more access to IT-level network and security resources for engineering projects, according to Ted Franck, electrical and controls director, global engineering, Campbell, who spoke during the Food & Beverage Industry Forum at Automation Fair 2018 on Nov. 15 in Philadelphia.
"When you modernize the plant floor, you've got to pay attention to cybersecurity," says Franck. "The catalyst for us came when we did our global, plant-floor security audit in 2015, which resulted in board-level recognition that Campbell should take a proactive approach and helped drive and accelerate the implementation of a central governance model and team." The audit included data from the U.S. Dept. of Homeland Security's (DHS) Industrial Control System-Cyber Emergency Response Team about increasing ICS cyber attacks nationwide.
Franck reports that Campbell's secure modernization solution was to create a central governance team with cross-functional members from IT and engineering—including a dedicated team focused on the plant floor—and work with Rockwell Automation to execute detailed, onsite plant-floor assessment and remediation plans at each of its plants. "We knew we couldn't treat OT like IT. Rockwell Automation helped us define our approach moving forward," explains Franck. "They assisted in evaluating our existing architecture, and provided recommendations based on our requirements and needs."
To establish its secure, robust and reliable networks, Franck added that Campbell performed several tasks, including:
- Created its own detailed, standard architecture diagram;
- Developed its own industrial Ethernet standard to share with OEMs and system integrators;
- Installed firewalls at every plant;
- Eliminated all public Internet protocol (IP) addresses;
- Eliminated any unmanaged switch connections to its plant floors;
- Eliminated outdated operating systems and unsupported switches.
"It was also important to have the right roles and skillsets on the plant-floor team. We created a new role solely focused on Industrial Ethernet (Level 0-2). We hired an IT network professional, and focused him on the plant floor where we've standardized using Rockwell Stratix switches for our machine-level equipment. Having this resource accessible during the initial stages as we execute capital projects has been a big win for us."
Similarly, Brazilian papermaker Klabin reports its plant in Telemaco Borba, Parana state, has been using ABB's System 800xA DCS since 2006 to control equipment in its Cozinha Couche (kitchen area), where additive recipes are prepared for incorporation with paper sheets in the production process on two paper machines (Figure 2). However, because 800xA was running on an obsolete Windows XP operating system, this made it and its papermaking process a target for malicious software because patches were no longer available.
Figure 2: When papermaker Klabin recently upgraded the System 800xA DCS in the "kitchen" where paper additives are prepared at its plant in Telemaco Borba, Brazil, the company worked with ABB to implement embedded cybersecurity features, including advanced access control, white listing, security integrity monitoring and management, digital code signing, validated software patches, updated virus scanner software and third-party antivirus files. Source: Klabin and ABB
Consequently, Klabin's management decided to upgrade to the latest System 800xA, version 6.0.1, with embedded cybersecurity features, including advanced access control, white listing, and security integrity monitoring and management. It also uses the latest Microsoft Windows MS 8.1/Server 2012 R2 operating system; installation that automatically sets policies to simplify system hardening; digital code signing to ensure software legitimacy; and immediate access to approved antivirus files.Beyond providing access to the latest control system software and improvements, the 18-week upgrade's three-year service agreement also ensures better protection for 800xA against malware infections and cyber attacks by giving Klabin validated software patches, updated virus scanner software and third-party antivirus files. These are validated by ABB for relevance and system compatibility to secure Klabin's systems against cyber threats with minimal effort in evaluation, installing and testing. Operators and plant management at the Telemaco Borba plant also gained access to detailed information about installed system software, system health status, safety and security reports, field alerts and service contacts through myABB/My Control System, a secure web-based platform.
"As coordinator of the project for the upgrade and virtualization of 800xA from version 4.1 to version 6.0.1 in Cozinha Couche, I know that ABB's involvement exceeded the common barriers between customer and supplier," says Luiz Roberto Gentilin Jr., project coordinator at Klabin. "The team’s involvement was always proactive to make it viable. Attention and concern for the good progress since the formulation of the proposal, constant support of the technical area, management team and implementation team were essential to overcome the barriers that appeared during the project.”
Likewise, public utility Saudi Electric Co.'s (SEC) Qurayyah Power Plant 1 (QPP1) recently needed to migrate the Honeywell DCS managing its 4 x 625 megawatt (MW) units from version R211 to R430, and upgrade 52 Experion PKS nodes to R430. The plant's legacy network infrastructure was obsolete, which caused system reliability concerns, controller and network loading issues, cybersecurity vulnerabilities, and network component management problems. The two-month migration project also included upgrading:
- IT-based switches, routers and firewalls,
- 12 redundant C200 controllers to C300s,
- Safety Manager software to R150, eServer,
- Matrikon OPC to interface with GE Mark VI controls,
- 11 video screens to 55-inch LEDs, and
- Cybersecurity functions and remote service nodes.
SEC reports its migration at QPP1 improved the utility's operations, addressed cybersecurity vulnerabilities, improved display call-up and process parameter update rate, and improved controller loading by 35% by using the C300s.
Software secures software?
For most of the 15 or so years that cyber threats and attacks have emerged and spread as increasingly serious problems, most cybersecurity efforts focused on protecting devices and the networks linking them. More recently, some observers speculate that software is simultaneously becoming cybersecurity's latest weak point, as well as its new frontier and eventual savior. Basically, just as software can allow connections between devices and networks, and permit vulnerabilities if not managed properly, its instructions can also be used in increasingly sophisticated ways to set up better cybersecurity measures and maintain them, too.
For instance, software instructions are the engine that drive established tools for data and network management, such as Simple Network Management Protocol (SNMP) and its many cousins, as well as Security Information and Event Management (SIEM) software for real-time incident analysis and management solutions. Both can be instrumental in aiding cybersecurity efforts.
More recent software-based methods include MQ Telemetry Transport, which is a simple, lightweight, publish-subscribe messaging protocol designed for constrained devices and low-bandwidth, high-latency or unreliable networks. Another is Advanced Message Queuing Protocol, which is a published, open-source standard for asynchronous messaging by wire. It enables encrypted and interoperable messaging between organizations and applications, and is used in client/server messaging and in Internet of Things (IoT) device management. Also, REpresentational State Transfer (REST) defines constraints for creating web services and RESTful application program interface (API) uses HTTP requests to get, put, post and delete data, and uses codes to allow software programs to communicate with each other. All can include instructions and rules for improving security.
To learn how software can aid cybersecurity, Xavier Mesrobian, vice president of sales and marketing at Skkynet explains, "It's important to separate the idea of access to data from access to a network. Users look at IT network security, and they want something similar for control networks. However, it's not possible because, if you provide continuous network access like IT does to access your data, you expose an attack surface on your controls."
This is why, in addition to network segmentation, control systems are typically protected by defense-in-depth strategies employing multiple layers. "The question is how many layers to implement to prevent intrusions and fight back against the hackers. It's a real cat-and-mouse game," says Mesrobian. "Control system owners would like to eliminate both the complexity and the exposure of this network design. Unfortunately, they're constrained by the client/server architecture inherent in process control protocols like OPC UA.
"Instead, the data should be emitted or pushed to a rendezvous point such as a proxy or broker located in a secure location outside an application or plant where it can be safely manipulated. This lets plant operators provide data to remote users, but completely firewall and lock-down their operations. There is no attack surface because users of the data only interact with the broker."
Mesrobian points out that Skkynet's SkkyHub and DataHub solutions separate data from their networks in exactly this way, so users don't have to open firewalls on their operations. "Our new DataHub version 9 adds both MQTT client and broker functions," he adds. "This allows MQTT applications to talk directly to Modbus TCP, OPC and OPC UA without requiring users to write code.”
Meanwhile, Eric Knapp, director of strategic innovation, Honeywell Industrial Cybersecurity, Honeywell Process Solutions (HPS), adds that, "For a long time, the threat was to the networks, and industry learned to highly segment and isolate them, so the networks weren't as big a problem. What the bad guys realized is it's still easier to get in via other avenues like USB drives, which is why we developed SMX. This addresses the right approach—how is information moving around in a plant?"
Honeywell reports its Secure Media Exchange (SMX) protects users and facilities against USB-based threats without needing complex procedures or restrictions. It provides multi-layered protection for managing USB security, letting users plug in and check devices for approved use in the facility. Contractors check-in their USB drive by plugging it into an SMX Intelligence Gateway. The device analyzes files using techniques included with Honeywell’s Advanced Threat Intelligence Exchange (ATIX), a secure, hybrid-cloud threat analysis service. SMX Client Software installed on plant Windows devices provides another layer of protection. Knapp adds that some SMX capabilities are also available via cloud-computing services.
"In our recent 'Honeywell Industrial USB Threat Report,' we compared typical antivirus (AV) software, and found that up to 11% of the threats we found weren't detectable by traditional AV," adds Knapp. "This is why SMX aggregates the best available, cause-based, threat-detections tools in addition to local detection."
Cloud to the rescue, too
Even though cloud-computing services require more and closer-integrated networking links, and these usually come with increased cybersecurity risk, these powerful virtual and digital tools can also improve security at the same time.
The primary elements of cybersecurity are pretty widely understood at this point, but they still bear repeating, just to help make sure no specific parts have been neglected, especially in the face of ever-evolving threats and potential attacks:
- Disable default passwords, create new passwords, and initiate basic anti-malware, antivirus software;
- Segregate operating components, equipment, process applications and their networks from administrative/business-level networks and the Internet by using managed Ethernet switches as firewalls;
- Divide and segment operating areas, such as work cells, production lines and specialized applications, into functional sub-networks with more firewalls. This is especially important for mission-critical and safety processes;
- Enable read-only capabilities that allow production devices and operations to send data outwards, but prohibit any incoming instructions or commands. Publish-subscribe methods such as using hardware-based data diodes or software-based MQTT and AMQP can help as well;
- Implement and routinely update supplier and third-party software patching policies, even in a sequestering period is needed before deploying them on the plant floor;
- Continually train and retrain staff to comply with and practice their organization's settled cybersecurity policies and procedures. Simulations can help with comprehension and retention;
- Establish regularly scheduled network traffic evaluation using IT-based software tools that can identify, disallow and purge unauthorized probes and intrusions.
To make cloud computing, virtualized tools, and other digital formats more secure, Eliya reports there are several basic methods for monitoring them, including encrypted communications, managed virtual private network (VPN) tunnels, hosted servers or server clusters, and secure cloud services managed via web-based platforms.
"One of our clients is a specialty chemicals firm that makes resins, and they didn't allow outside access to the company network, but they did have customized logic with frequent bugs, and their local staff wasn't trained in PCS 7 controls," says Eliya. "We implemented mGuard switch from Phoenix Contact and used their servers to establish an encrypted, IPsec protocol, VPN tunnel between their control system network and our engineering workstation."
The chemical company and aeSolutions recently implemented a Siemens Virtualization as a Service (SiVaaS) system, which provides a virtual PCS 7 architecture using VMWare vSphere and ESXi. All control system terminals (OS servers/clients, ES, etc.) are running as virtual machines on ESXi host servers.
"Remote support engineers can employ mGuard to quickly navigate to any of the VMs to make program changes or install hotfixes," explains Eliya. "When connecting mGuard to SiVaaS, it’s important to define user permissions for each person who may need to remotely open certain VMs, reboot the host or an individual VM, and perform a full suite of monitoring and control tasks."
Eliya adds that signing up for mGuard secure cloud service also lets users access its VPN Builder software tool, which tells them what's happening with their connections and firewalls; allows them to set switches on mGuard devices; and provides the capability to ask their cloud service to build VPN data files. "Overall, the mGuard experience is good, and everyone saved time and money," he says. "However, there were some initial challenges with the VPN tools; activating the data plan took months; users must coordinate modifications with Phoenix Contact; and the PCs' secure cloud can be taken down for maintenance."
Meanwhile, another aeSolutions client has been using Siemens Totally Integrated Automation Portal (TIA Portal) to monitor the efficiency of cooling towers in Mexico, using a Siemens Simatic S7-1510 controller with six Profinet remote I/O drops, and recently decided to add Siemens Sinema Remote Connect (RC) server to its industrial Ethernet architecture, which is based on Siemens Scalence industrial communication products, to secure its wireless and company networks.
Eliya reported that an appliance is installed on an aeSolutions server and assigned an address for the Sinema RC server. Next, a profile is created on the server for the equipment or application being monitored, and that profile is automatically downloaded by the switch. Finally, client software is added to the user's workstations, a profile is created on the server, and a secure VPN tunnel is established.
"This is a very simple setup. You don't need to be an IT or cybersecurty expert to use these tools," adds Eliya. "We host it locally to prevent unplanned downtime.
"The case for adding secure remote access like this is pretty strong. Many facilities are constantly adding new users and locations, and this solution can do it more efficiently, prevents devices from ever being 100% isolated, helps pull IT and OT together, and helps users achieve better work/life balance."
Can't avoid culture
Even now, after all the recent improvements in cybersecurity software and devices, it's universally agreed that successful cybersecurity still depends on continual training to make people adopt and routinely practice all the tools available to them.
Despite its modernization gains, Franck explained that creating the plant-floor focused team was a big culture change for Campbell. "Some of the tasks that used to be owned at the engineering level are now owned at the IT level," adds Franck. "To do the plant assessments, we have a laundry list of items to look at for each plant. Rockwell Automation helps us develop the report, and we develop the remediation plan, including what we need to do now and what we can do further down the road."
Franck reports that Campbell was also aided by Rockwell Automation's longtime partnership with Cisco to develop their "Converged Plantwide Ethernet (CPwE) Deign and Implementation Guide" and its services for performing assessments, configuring Stratix switches, and upgrading applications such as its FactoryTalk AssetCentre and RSEnergy software.
"We're using AssetCentre to enhance our change management program on the plant floor. Having the proper change management program in place is another key component for us," adds Franck, who reports that other benefits include:
- More efficient network monitoring and troubleshooting;
- Using Stratix 5900 switches as zone-based firewalls;
- Allowing access to OEM black-box connectivity from equipment status and remote support needs;
- Possibility of using modern, network-monitoring tools like Claroty;
- Closer IT relationship allowing timely access to security SMEs for capital projects;
- Creating central governance model for proper planning and reduces risk of running unsupported, high-risk systems.
"For Campbell, this isn't about one-time results," concluded Franck. "It's a culture change on how IT and OT can work together. We already have them doing it, but we can do better. One of our senior engineers said it best that it's all about developing the trust needed to open that door, and let IT into the manufacturing space."
