You can't be everywhere at once—especially when it comes to managing widely scattered process units—but you can get close. But while a good remote support solution can provide access and interaction almost anywhere, the crucial trick is doing it securely, according to Arnold "Marty" Martin, control system manager at Air Liquide in Houston, which oversees hundreds of the firm's air-separation and other U.S. production plants.
"We were running into problems with aging assets, and needed to upgrade, but the remote support solution we were using wasn't migrating well," said Martin. "Besides having a lot of equipment that was reaching end of life, we also didn't have enough expertise and people. We needed to manage access to control systems on the industrial network to provide maximum security without impacting ease of access for people who need to use them; develop robust, multilayered access protocols that meet the needs of internal and external users; and incorporate strengths of the business IT world to improve how users access the industrial network environment. So, we pursued a privileged (remote) access management (PAM) solution."
Martin presented "Privileged Access Management (PAM) for Remote Support of Industrial Automation Systems—Evaluation, Selection and Implementation Overview" at the Yokogawa Users Group conference this week in Orlando. He defined PAM as enabling security professionals to control, monitor and manage privileged access to critical systems by authorized employees, contractors, and vendors.
Guided by business IT
Martin reported that he and Air Liquide sought out IT-based PAM because the business IT side has more experience in developing, implementing and maintaining secure solutions for remote support. He said the upgrade project began about a year and a half ago, included a six-month proof of concept, but only took four months to get up and running.
"Business IT and industrial IT have different methods and models, but business IT has more cool stuff for secure, remote support, and I can implement them without losing my mind," said Martin. "Frankly, I usually get lost within 15 minutes of talking to business IT people because they have so many unfamiliar acronyms, but PAM helped because we were also expanding facilities like crazy, and suddenly we might not have the remote access we needed.
"We're also dealing with a new security landscape. Everyone's throwing everything on their networks, and so there's increasing risk of control system cyber attacks and more stringent authentication requirements, but we still need anytime/anywhere remote support for better work/life balance. We also needed IT-based remote support for the industrial side because even though young staff don't want to go to bad locations, we still have to monitor them."
Arguing for remote access
To convince potential users to buy in and follow up on a secure, remote access upgrade, Martin reported that he and subsequent supporters must build a case for it. This included:
- Assess current methods for logging in and why they need to be changed, such as holding a "why consider a new system” brainstorming session, and examine if travel budgets get in the way of support;
- Identify requirements for remote access by reviewing remote access/control packages;
- Solicit industrial/business IT personnel for administrative requirements;
- Understand company's existing business and industrial IT network architectures, and secure or build network diagrams; and,
- Draft business rationale and risk analysis for remote support upgrade.
"You have to look at how fit your current remote support system is, determine if new requirements are being met and if you're adhering to enhanced authentication and auditing requirements," explained Martin. "We wanted to be virtually in front of the equipment and process applications at our plants—like we were physically there—even if we were seeing everything via cameras. I built the needs case for our remote support project, and it covered why the upgrade was so important and why Air Liquide should spend money on it. I also had to be careful about saying the word 'cloud' because the initial response was 'no.' Fortunately, it paid for itself in about one year."
To evaluate remote access requirements, Martin recommended:
- Address security requirements, such as access controls for authentication/authorization, auditing, logging policies and central management. Also, decide whether to run the new system in-house or on a cloud-based service;
- Determine connectivity for mobile devices such as smartphones and tablets PCs, pick an inside-company network such as a virtual private network (VPN) or an outside-company/non-VPN network, and settle on direct- or indirect-networked devices;
- Choose performance features, such as multi-operating system support, timed/escorted access, multiple sessions/users per session for collaboration, session-logging, bandwidth performance management, and remote sound and video and multi-monitoring for using remote HMIs.
"We identified our requirements for remote access, and knew we needed to segment our network with firewalls with rules," said Martin. "Filling out the IT sheets for risk analysis was difficult because a lot of the language seemed foreign at first and was hard to understand.”
Risk assessment and analysis
Martin reported that a thorough risk assessment (RA) will help remote support upgrade projects determine what kind and how much security to implement. "An RA is part of a sound digital security policy, and it's performed against the pre-qualified packages," explained Martin. "It helps ensure that all potential solutions are on a level playing field, and that easily overlooked security detail aren't missed. Its outcome includes a risk mitigation plan, controls risk reduction rating, and a cybersecurity system management document that is really nice to develop and have in place."
Martin added that subsequent risk-reduction controls may include procedural-only security policies that are the least desirable because users will likely neglect to follow them in the future, and system-enforced security polices that are the most desirable because the remote support system and network can't operate if they aren't followed. In addition, he advised that a complete security evaluation is crucial. "Risk mitigation is a major component, but consideration should also include the lifecycle costs, maintenance, supportability and essential product features," said Martin.
Multi-layered access, account management
To implement a robust, multi-layered access protocol for remote support, Martin also stressed using account management with well-defined account types and an administrator as the central hub; employing local authentication controls and integrating external authentication systems; establishing granular access control with access notifications, scheduled access times and whitelisted and blacklisted applications; deploying forensic functions like session monitoring and audit reports about video sessions and text logs; and using session protocols like Direct, RDP, SSH or Telnet.
"Account management, authentication and granular access control are what allow us to get in front of our processes virtually and remotely, but we can also avoid giving added rights when they're not needed because this isn't distributed control," said Martin. "This allows our plant managers to decide who gets in or not. Meanwhile, forensic functions let remote users see reports, but no one can delete or remove them, so they're secure. Also, RDP is cool because it shows everyone that's logged on, so no one can sneak in, and we can shut the RDP port if needed. Likewise, the industrial IT guys like SSH because there are no backdoors, everything is documented, and users are only granted access to do their specific job functions. We also have different levels of administrator and user accounts, and each type can monitor the level below, plus we can get digital logbooks to monitor the activity of everyone on our systems, document who's allowed in and who's been on, which we couldn't do before."
Martin added that the authorization functions in Air Liquide's new remote support solution lets it employ two-factor authentication, just like the company's business IT department. However, instead of going through the IT side and maybe enabling a vulnerability, users on the OT side can now go to a PAM redundant appliance/remote access server, which acts as a network demilitarized zone (DMZ) between two firewalls. "This is a much more narrowly defined pipe, and so it's a lot more secure, too," added Martin. "Now, we can identify and fix problems on a smart phone in two minutes, and do it securely. We're now in a virtual age, but we don't have to have less fidelity and support because now we can pull in anyone."