In this podcast, control system cybersecurity expert Joe Weiss sets Control executive editor Jim Montague straight on several major cyber security issues, especially sensor, device and Level 0 vulnerabilities, and the ongoing inability of network-focused efforts to acknowledge and address them. He also examines other parts of the current cybersecurity landscape, and provides several practical recommendations that users can employ to improve the cybersecurity of their process applications, facilities and organizations.
Transcript
Jim Montague: Hi, this is Jim Montague, executive editor of Control magazine and ControlGlobal.com, and this is the latest in our Control Amplified podcast series. In these recordings, we talk with expert sources about process control and automation topics, and try to get beyond our print and online coverage to explore some of the underlying issues impacting users, system integrators, suppliers and other people and organizations in these industries.
For our ninth podcast, we're talking to Joe Weiss, who is managing partner at Applied Control Solutions (ACS); producer of Controlglobal.com's popular Unfettered blog; ISA fellow, managing director of ISA99 ICS automation and control system security committee; and author of Protecting Industrial Control Systems from Electronic Threats.
Now, I don't want to call this podcast a showdown, but Joe and I have debated Control's cybersecurity coverage a few times in recent years, and I thought it might be useful to let our listeners in on our latest discussion. When I research Control's articles on cybersecurity, including this December's cover article, I ask as many sources I can find about practical cybersecurity methods. After that, Joe often informs me that I've painted too rosy a picture, which focuses to much on new security gizmos, and too little on sensor, device and Level 0 vulnerabilities, and the ongoing inability of network-focused efforts to acknowledge and address those gaps. Truthfully, I believe he's correct, so we're going to try and capture in this audio format some of the points he expresses online in his Unfettered blog.
Well, Joe, sorry for the usual preamble, and thanks for joining us today.
Joe Weiss: Well, thanks for asking. I appreciate the opportunity.
JM: Alright, let’s get started.
First off, what are sensor, device and Level 0 vulnerabilities, and why don't they get the attention they deserve?
JW: It’s kind of a, if you will, a historical thing. In fact, I want to go back a step. The thing that really makes control system cybersecurity are the control systems devices. Short of that, what you’re doing is talking about networks, and what happened was, prior to 9/11, the engineers owned everything about their equipment. So, the turbine engineer owned the turbine, but also the cybersecurity of the turbine. The manufacturing engineer owned that manufacturing line, including the cybersecurity of it.
Then, following 9/11, cyber was made national security, and it was taken from the engineering organizations and moved over to IT. By the way, this is irrelevant of IT or OT. But it was moved away from the domain engineering experts, and when that happened, two things occurred. One was the domain control system experts were basically severed from the world of cybersecurity of their own devices, their own equipment, which just makes no sense. The other was, and that’s where this is really coming from, to the OT or even IT/OT organizations. Cyber is the network, the IT, internet protocol ethernet network.
Well, Level 0, Level 1 devices use the Purdue Reference Model, in other words, process sensors, actuators, drives, analyzers, power supplies, are considered to be engineering devices, not network devices. And so, they basically go left off.
What’s happened is all of these devices, particularly the legacy devices, but not just legacy, have no cybersecurity, no authentication, no cyber logging, etc., and these are also the devices that are foreign to IT or even OT networking people, but this is what the engineers and technicians live with all the time. And that’s the gap here, is that the IT/OT or the networking community has basically taken over the world of cybersecurity, and to them, these Level 0/Level 1 devices are simply not what they do. It’s crazy because that’s what makes a control system different. It’s also what makes it useful and/or dangerous.
I was at an IoT conference yesterday in Santa Clara, and one of the things they brought up is these are the, if you will, dangerous but most important devices that are out there.
JM: So, everybody focuses on the network and network security, right, but these are historically devices that are in a support role, they’re just not as visible to the people talking about cybersecurity, right?
JW: Here’s the funny part of what you just said. The network is supposed to be a support to the production environment. What has happened is exactly what you said without necessarily meaning to. The tail is now wagging the dog. We’re more concerned about the network than we are about the devices and the processes we’re actually trying to monitor and control.
JM: Why can't network-centric cybersecurity efforts, IT departments and apparently many of the sources I cover see and acknowledge and address these security gaps at the lowest level? Why do they have the blinders on?
JW: Because it’s not what they know. What you’ve got is the old adage, if you’re a carpenter, and you have a hammer, everything looks like a nail, whether it’s smooth or has threads like a screw. What’s happening is the network people look at everything through network lenses, and these devices are simply not those. And a big part of this is senior management governance to essentially have allowed this type of situation to occur.
JM: Right, and when we’ve talked previously, some of the devices are sensors, but you’ve also talked about people will bring in laptops for configuration or they will bring in calibrators into the production situations and these are behind all the firewalls as well, right?
JW: Yes. What you’re talking about here also is, if you will, another difference between the control system world and the network world. In the network security world, they live on the principal of zero trust, in other words trust nobody, trust nothing. The process sensors, and for that matter actuators, drives, are based on 100% trust. You absolutely, positively trust what that sensor tells you, and you just assume, famous last words, that that sensor is uncompromised, authenticated, and the value is accurate, and all three of those assumptions may be wrong, certainly the first two are. None of our legacy sensors today, when I say legacy I’m saying anything being sold to date, none of them have gone through any cybersecurity certification testing – none.
JM: So, what's to be done? How can users and the rest of the process control and automation community begin to understand device-level vulnerabilities and plug those holes? I think you've mentioned going up from the inputs to the network, not from the top down, correct?
JW: Yeah, this is truly back to the future. What was done 20 years ago is done today, again if you will in the vibration monitoring or some of the other areas, but the way you know a sensor or a process, the health of it, is by looking at the noise or fluctuations in the sensor signals. Unfortunately, one of the things that’s happened is because the engineers were removed, when the if you will serial Ethernet converters or gateways devices came into play, in order to get those sensor signals to start with into an Ethernet packet, they had to do some conversion.
The problem was because the HMIs are slow, on the order of seconds, the gateway vendors, serial-Ethernet converter vendors if you will, decided to filter out the higher-frequency noise, in other words, the noise that tells you the health of the process. So, here we are with all of these wonderful IP networks, and we’re actually less knowledgeable today than we were 20 years ago, because the most important information has been filtered out.
So, what has to be done is we need to go back to the future to be able to look at these process fluctuations to understand the health of the signal, the health of the process, and oh by the way at that level, that’s physics, and you can’t hack physics.
The reason this all becomes so important from a cybersecurity perspective is what happened with Stuxnet and what happened with Triton. In both cases, these were cyber attacks meant to damage equipment, and in both cases, in order for them to be successful, they needed to be able to compromise the Windows-based operator displays so that the operator didn’t take an action that would have, if you will, overcome the attack mechanism, and so, the real message coming out of this is if all of our signals are only going to be on that Ethernet, LAN going to that Windows or other type of commercial-off-the-shelf operating system, we’ve really lost safety. We really, really have to have some type of external monitoring system. It doesn’t have to be you have 10,000 sensors in a plant, this is not 10,000 sensors. And for those, you really need to have an alternative system that is not connected to that Ethernet network.
JM: Right this is for monitoring the raw, real-time device-level data.
JW: Yes. For example, this would have been the only way to have detected Stuxnet, which would have been to be looking at the raw signals and be able to say, “Wait a minute, the centrifuge speeds are changing.” Whereas, the operator displays were compromised saying that weren’t.
With Triton, in order for Triton to be successful, this is the hack of a safety system, they had to make sure No. 1 that the operator displays were compromised so the operator wouldn’t know you’re getting to dangerous conditions, and No. 2 you would have had to suppress all the alarms, so the operator doesn’t manually shut the plant down. If he does either of those, he defeats that malware that was meant to prevent the safety system from operating. So, what has changed since, if you will, the late 90s early 2000s, is there was a tendency back at that time No.1, not to think that anybody would maliciously play with a system, and No. 2, to believe that you could trust what your HMI was telling you, and cyber is creating an entirely new paradigm we have to be able to address.
JM: As part of that paradigm, and getting beyond the sensor level and the device level, how can control engineers and other OT personnel understand IT's security priorities and policies, and how can IT appreciate process operations and safety priorities? And how can they learn to work together on cybersecurity? Better leadership by senior management?
JW: Here’s the point. The engineers are already trained, and when I say engineers I mean technicians, etc., they’ve been trained on how to operate, maintain and understand what’s going on with those systems. They just haven’t been trained to look at it and say “Wait a minute, there’s something not normal. Could it have been compromised?” The flip side of it is the networking people have been trained and understand what the network should do and are looking for anomalies in the network, but they’re not trained to look at the equipment. So, it goes back to governance. Senior management has to have both sides, each of which knows what they know best, working on this. And what has happened up until now is the engineers and technicians who understand the systems have been kept out of cybersecurity. Now, I say that as a general statement, not a 100%. But we’ve got to make it 100% that they start working together.
JM: I think you've also advocated for cybersecurity risk assessments (RA) and a separate, redundant network for sensor, device and Level 0 components. What to these involve and how can they help improve cybersecurity?
JW: Well, again here’s another point. Why did we get involved in cybersecurity for control systems to begin with? It was because we were worried lights would go off, water would stop flowing, or a pipeline would explode, etc. That’s the reason we did this.
So, what we need to be able to do is put things in their proper perspective, and for the control system world, the proper perspective is we need to make sure that the reliability, safety and resilience of the system is maintained and that cybersecurity can’t negatively affect it. That’s where cybersecurity of control systems lies. Cybersecurity of data is something very different, but we’ve got to understand why we’re doing this to begin with. It’s to keep processes up and maintain reliability and safety, and for that what you’re looking at cyber for is can it affect that? If it can, we’ve really go to address it, if not, it’s really not that big a deal. That’s not to say it’s not a big deal for data, but for the actual control systems, the real issue is what impact can it have? Not, what vulnerability does it have, what impact? Because you’re worried about risk and impact. There’s way too much of a focus on vulnerability of networks. For control systems, it’s not vulnerability of networks, it’s impact of processes.
That’s also why the, if you will, the credit rating agencies are looking at this is because they’re worried about the existential threat to an industrial or manufacturing facility or company, and that comes from the control systems. It does not come from IT.
JM: It was kind of very encouraging her hear though that some of the traditional control’s engineer attributes of being focused on process safety or being focused on the deviations in the operations, can then be used for cybersecurity.
When we talked earlier, there was a situation I think where if somebody sees some upset of hiccup in their operations, they can then go to their IT department and say, “Hey, could this be caused by an intrusion or something like that,” correct?
JW: Yes. Ironically, that’s where we were until after 9/11. Prior to 9/11, we weren’t worried about the networks, we were worried about the process. That’s what we’ve got to get back to.
JM: And then, one way to do this is to have a separate network?
JW: Yeah, it’s been demonstrated. It’s just that simple. A sophisticated hacker, and nowadays the sophistication is moving lower you don’t necessarily have to be a nation-state, but a sophisticated hacker is going to make a cyber attack look like a malfunction. We need to be able to understand when there is a malfunction, was it an unintentional or was it cyber-related?
JM: The collaboration between the controls guys and the IT guys could really be a huge benefit to preventing these kinds of things. It’s encouraging to hear that that’s possible.
JW: Well, I’ll take it a step further. I think it’s necessary. Not that it would be encouraging, but it’s necessary. The issue is we have to bring the engineers back into this. The people that actually understand the process have been removed. That’s just kind of a bizarre thing to say.
JM: Well technology has changed so fast it is hard for anybody to keep up, so it’s good.
JW: It’s much more a culture issue, because technology is changing on both sides. It’s changing on the control systems side. It’s changing on the cybersecurity side. It has nothing to do with technology. This is culture, and we have to be able to figure out how to overcome that culture gap.
JM: And culture is always tough to deal with for anybody in any industry or endeavor, correct?
JW: Yes, because until you solve the culture, technology is second.
JM: Cool. Alright, well we’ll work on it. Any final thoughts? This is a great interview again today.
JW: Well, again, thanks for having me on, and I’m hoping that this message will get out to both sides: the engineering side to say I need to be involved in this, and to the networking side whether you call it IT or OT to say you have to have the experts involved with you. It’s not to keep either out, but it’s to say both have to be in.
JM: Alright, well listen, we’ll keep covering it and keep doing more debates, it certainly doesn’t have to be the only podcast that we do.
Well, Joe, thanks for some great input, and thanks for cluing us in today.
JW: Jim, thanks and I will talk to you later.
JM: This has been another Control Amplified podcast. I’m Jim Montague. Thanks for listening. Oh, and please remember that Control Amplified podcasts are available on most podcasting apps, such as the iTunes store and Google Play, and on Control magazine's YouTube channel podcasts. Plus, you can also listen at controlglobal.com, of course.
