Cybersecurity disclosures-- the game everybody can play

Joe Weiss posted an extremely thought-provoking blog entry this morning on Unfettered. He questions the hacker/cracker cultural meme of disclosing cybersecurity vulnerabilities for the sheer pleasure of doing it. I think Joe's on to something here. We have a serious problem in cybersecurity in control systems...we don't have enough "cybersecurity experts" who know anything about process control or factory automation. We have a bunch of soi-disant experts who descended on control systems (remember, they're the guys who thought every control system was "SCADA"?) because they saw a big market, and have been spreading FUD ever since. Recently, a Wonderware vulnerability has been disclosed, and the disclosure is making the rounds. Several months ago, an ICONICS vulnerability was disclosed, causing ICONICS significant distress. Why? Well in both cases, the vulnerability was, although accurately described, not dangerous. In the Wonderware case, the vulnerability only applies to a very few customers who are still using a very old, outdated version of Wonderware's software that is so old that it will become "unsupported" at the end of the year. In the ICONICS case, the vulnerability, that generated a huge cyber alert both in Australia and the US, was only on the web demo on the ICONICS website. It would be a good thing if we all started thinking about these issues, and doing our best to discuss these types of vulnerabilities publicly with a clear eye to also disclosing the potential impact. Otherwise, we are reduced to a pack of former 13-year-olds giggling about scrawling metaphorical cyber graffiti, for the pleasure of the game. If we want to be taken seriously by policymakers, rulemakers, and politicians, we need to do better than that.

What are your comments?

You cannot post comments until you have logged in. Login Here.

Comments

  • Walt,

    Obviously I disagree completely about the need for disclosure. It is important to note:

    1) that Wonderware chose not to disclose this even through their secure, customer only sites. Even when a solution was available.

    2) until, that is, US-CERT got involved. Now customers can make an informed upgrade decision.

    3) disclosure was by US-CERT. Our policy is always to disclose to US-CERT and rely on them to make a determination if disclosure is warranted.

    4) many control systems do not upgrade unless there is a compelling reason, and Wondeware is a very popular product. I would be very surprised if the 8.0 community goes away soon.

    4) Ad hominem comment or attacks about peoples knowledge are hardly persuasive.

    Dale Peterson

    Reply

  • Dale, I was very tempted to not allow your post. It is my understanding that your point 1 is not correct. Your first point 4 is, according to my information from Wonderware somewhat misleading. The product in question is a very old version, which goes into "non support" in January.

    Finally, your second number 4...what can I say? I don't think telling the truth is ad hominem attack. And your name wasn't mentioned.

    Do you disagree with my premise: that in order to adequately advise people about cybersecurity in the process industries, significant familiarity with those industries and control systems is required?

    Walt

    Reply

  • Walt,

    We directly asked Wonderware if they had informed their customers on their private customer site or in any other way. They responded with a long white paper that did not address the issue and thought the issue was closed. We asked again in the most straigthtforward manner possible and got a non-answer and a request for more time. There was nothing in our discussions on this vulnerability that could remotely be construed as Wonderware notifying affected customers or even considering notifying affected customers. We wanted to hear that because they had a fix and customers should know about it. After five months of first Neutralbit contact and then Digital Bond contact, we decided in conjunction with Neutralbit to go forward and notify US CERT.

    This is all in our blog entry, which you and Joe should have linked to so people could draw their own opinion of what we wrote if you are referencing a blog.

    Regarding "very old", we regularly see devices and applications in our SCADA and DCS assessments that have not been supported for many years. Many obsolete for 5+ years. Some discontinued in the 90's. This market does not upgrade until it is absolutely necessary. The tales of spare boards being produced in garages to keep a system going are legion. Since there are recommendations for 8.0 and the vulnerability is not in 9.0 disclosure seems warranted.

    I don't know what else was implied when Joe wrote, "The way that the cybersecurity establishment has presented the Wonderware disclosure on the Digital Bond website clearly shows the lack of control system expertise in the cybersecurity “industry.� It IS an industry, and it is filled with people from IT security and cryptographic analysis backgrounds who have rarely, if ever, set foot in a control room for a process plant, refinery, or power plant." And I don't know who you are talking about in your blog. I haven't met those people you describe in any industry events.

    It's not a problem. You and Joe are great (love the passion), and my skin is thick. I just hate to see a legitimate issue degrade into a "these people are ignorant" argument.

    Disclosure is a tough, emotional issue. Always has been. The point of agreement should be that affected customers should be notified of vulnerabilities. This is not happening, and we have numerous examples of this as do others. Wonderware is just one, and as we said in our blog, they at least had a fix for the vulnerability which is better than many.

    Dale Peterson

    Reply

  • That's all well and good, Dale, but you didn't answer my question.

    It isn't about "those people are ignorant." It is about inexperience with the very real differences between cybersecurity as applied in the enterprise environment and in the industrial plant floor environment.

    I ask again: Do you disagree with my premise: that in order to adequately advise people about cybersecurity in the process industries, significant familiarity with those industries and control systems is required?

    Walt

    Reply

  • I agree that some level of control system knowledge is required, and add that IT and IT security knowledge is desperately needed because control systems developed this decade are complex networks and applications. On the security front we need people with expertise in database security, secure software development, application assessment, protocol analysis, AAA, security architecture, incident detection, ...

    Where we differ is I don't see a lot of pretenders coming from the IT side out there spreading FUD, and would rather encourage the people trying to cross over and help in control systems rather than say you don't get it. Stay out. We need their skills. Most engineers did not go into this field to do IT security so it will be hard to grow the numbers needed from the existing control system talent.

    Dale

    Reply

  • Nobody said that anybody should stay out. And I flatly disagree from my own experience. There are lots of people crossing over from IT security into plant cybersecurity-- and I've attended enough conferences and spent enough time talking to them and I can flatly tell you that most of them not only do not understand plant security BUT THEY ALSO DO NOT WANT TO LEARN. I for one am tired of talking to soi-disant experts who know far less than I do-- and I don't claim to be a cybersecurity expert.

    For many IT experts, this is at bottom a turf war, and we can't continue to do that.

    We ARE going to have a serious cyber incident that causes deaths and economic dislocation. It WILL happen. And when it happens, it may very likely be partially caused by cyber policies administered by "experts" who aren't.

    And your last point is flatly silly. There are plenty of control system engineers who DO understand the requirements of cybersecurity. In many companies they are being ignored by their IT expert brethren.

    It is obvious that we aren't going to agree on this, Dale, and this is becoming a religious argument. Let's agree to disagree.

    Walt

    Reply

  • I think that the issue posed in the original discussion above is only semi-valid.

    To be different, I'll use letters:

    A/ The fact that the Wonderware issue was on an old version that is destined for end-of-life status is *exactly* the point. Process control hardware and software "total systems in use" is going to be asymptotic. There will always be a slowly declining number of systems in use - regardless of the manufacturer's position on this. I recently encountered a process control system in use at a rural water management site that was running CP/M, using 8" floppies, and contained within a particle board desk. These systems aren't replaced until necessary, and in the case of older systems, necessary may be a very long time indeed.

    B/ Both sides of the argument have significant issues with hubris. As I'm originally from the IT side, I'll take that one directly on the chin - when I started in SCADA security, I had a (metric) butt-load to learn. I've done my best to ask the dumb questions, to agree to being made fun of, to set myself up as less the expert and more the student. I think I'm in the minority relative to some of the others scrambling for a chance to bite off some of the budgets being released for saving the planet (this time). Dale is also one who has put in the time. On the other side, I was informed (by a 35 year veteran of power systems design) that I should stick to my knitting as he's been a professional in the field longer than I've been alive. Gee, thanks Mister! By the way, your web-enablement of your power system is susceptible to more than half of the OWASP security flaws in web interface security. If he'd have stepped off of his high horse, we might've learned something from each other. Joe took the time to listen to me when I was describing a technical issue to him, considered what I had to say and took the advisement, rather than dismissing me out of hand because his hair is grayer than mine.

    C/ Of course people (media, governments) are going to misunderstand poor articulations of risk relative to control systems. The last 40-60 years of the "dumbing of the sheeple" has destroyed the general public's understanding of some basic science facts. If you connect real physical systems to logical controls, you *must* enable the logical controls with the same security that you would normally afford the physical systems. For a period of time, the administrative login to a large-area load shedding system was available on the internet to anyone who could figure out that a good password is not the word "password". By itself, this simply meant that you could forward a signal to all loads to shutdown (about 20 typed characters and about 10 clicks). By itself, this is just a configuration mistake and not really a big deal. Except that the load shed ability would be sufficient to destabilize the Erie Loop. How long do you think that took me to get the password changed? Would you be surprised if I said 2 months? I didn't release that one (in fact, this is the first time I've noted it publicly) and I know that the media would've had a bloody dance with it. Is it my duty as the "IT guy" side of SCADA security to keep prodding that vendor? Would you be surprised if I told you that there are at least two other ways to gain administrative control over those loads?

    D/ I'd love for it to be a requirement for all security teams to be comprised of *both* an IT expert (someone who understands the context of logical systems security in the wider "connected" world) and a Process Control expert (someone who understands what happens when you introduce an additional 200ms of latency on a safety system). Together, they're a powerful team. Alone... well, you can see where we are now based on the discourse above.

    E/ Both sides are responsible for education - Walt's got this one right, Process Control for a cookie factory is different from Process Control for a coal plant. The outcomes of the exposure of the risks are in different worlds. Do presentations such as the INL "Look - it blewed up!" movie help or hinder? I'd suggest hinder -- it's all about waving your arms in the air and screaming about the end of the world. Do presentations such as Joe's "The first rule of 'Bad Things Happen Club' is we don't talk about 'Bad Things Happen Club'" help or hinder? I'd suggest help -- practical demonstrations for an audience who can make the necessary changes. Is there something in-between? Probably - I'd ask you to look at the coverage of the Estonian bot-net attack in the most recent PBS "Wired Science" -- and what the impact of a same-scale event would be on North America -- it's not good people, not good at all.

    The question that I'll close with is the same question I always close with -- how can the various affected industries support pragmatic and rational change while simultaneously supporting those of us who are altruists as well as those of us who can't afford to "lose face"?

    Of all the credible people I know in Process Control/Plant Systems/SCADA security, I cannot think of any who are in it for fame and fortune. Joe's not wearing a Rolex, Dale's only tanned because he lives in Florida. Of those who lack the basic credentials (have actually worked in the right kind of environment and are not simply skills-transferral consultants), I cannot think of any who are willing to work the kind of hours and shed as much emotion as the other group.

    (By way of my own credentials, while I use a pseudonym, I know both Dale and Joe and I used to work in the power industry.)

    Reply

  • Oh - and as an aside Walt, IT guys can be helpful - you're running on a seriously flawed version of Wordpress - 7 revs behind current - each of which was released for both features and security fixes, please consider updating.

    Reply

  • Walt... it's 12 days later... still no update - this is an internet connected system. Are you starting to see why there is a certain level of frustration between IT folks and process control folks? You're vulnerable, anything on the same LAN as this server is vulnerable, and if you're like all too many organizations, you don't really have a concrete idea of what's connected to your LAN. This is the fundamental issue - a basic lack of understanding of your environment. And how we should be working together -- you teach me the right way to lock-out the system before I walk into the TIG welding robot booth, and I teach you how to avoid having your site defaced.

    Reply

  • In fact, we are working on a major upgrade that will include upgrading WordPress. Patiently waiting, I am.

    Reply

RSS feed for comments on this page | RSS feed for all comments