What Are the Metrics for Security?
Jeff Potter, director of security architecture for Emerson Process Management, says, "Security metrics are continuing to prove extremely hard to create, so a pure ROI calculation—I've spent N, and my security is X better—is not available. I'm not sure when it will be—we'd need many more incidents occurring to get a statistically valid sampling, and I hope that situation never occurs!"
Bodungen breaks it down very clearly: "Obviously, if you don't have a breach, then one way or another you have enough security—whether that means you've covered all of your bases, or you happen to have just the right mitigation for the specific attack being performed," he says. "Unfortunately, it's difficult to know the sophistication of the attacker and what attacks he will use, and covering all of your bases with the maximum level of comprehensive security is very resource-intensive. Therefore, at the most basic level, it boils down to this: You have to know what your vulnerabilities are. For each vulnerability, you have to have a good idea of the likelihood that it could be exploited. You have to have a good idea of the consequences or impact to your business should that exploit cause a breach. You must know the estimated cost to mitigate or reduce the risk. Implement mitigation where the consequence exceeds the cost to mitigate, beginning with those vulnerabilities that would have the highest impact to operations and with the greatest likelihood of exploitation."
But how do you measure the result of security improvements? Potter says, "The ISA99 Task Group working on this issue has struggled to create a useful product, and although there are some narrow metrics that have value, like patch frequency and training, I've not seen anything that is remotely comprehensive."
Our anonymous operator from Ontario, Canada, has a different point of view. "For high-severity incidents," he says, "security is always a subset of safety. Therefore, safety metrics apply. That is, you track near misses; track found issues over time; estimate remaining latent issues; and assume there are always latent issues."
"The security metric," says William Miller, president of systems integrator MaCT, should be considered from a perspective that with contemporary approaches using anti-virus and firewalls, the endpoints will be vulnerable to cyber attack. If you look at the problem, it can be seen that today's TCP-enabled systems are insufficient."
Branquinho explains TiSafe's approach. "We normally follow these steps," he says. "Conduct a quantitative risk analysis of the plant to be protected. With this analysis, we can have a precise vision of the value of the assets of the plant and of the total risk the plant suffers in case of security incidents. Based on the risk analysis, we do the summation of values in dollars of the risks classified as critical. We use this value as the basis for calculating the investment. We named it risk analysis value (RAV). Then we calculate 5% to 10% of the RAV as the acceptable range of investment in automation security. We developed this methodology just because there wasn't any metric available for this."
Guida concurs. "To paraphrase a famous person—I think it was Benjamin Disraeli—there are lies, damned lies, statistics and worst of all, ROI calculations related to security. It is simply not possible to come up with reasoned ROI calculations because every situation is unique, and there are no reliable data like those traditionally used for actuarial studies."
How Much Security Is Too Much?
Obviously, there is some contention between experts on how much security a plant should have. If security gets in the way of operating the plant, that may be too much.
Jeff Potter says, "How much security is enough is dependent upon a risk assessment, including the consequences (monetary, HSE, other) associated with a security incident. One can obviously put a dollar value to a production outage, but this is less clear-cut when one is dealing with an environmental release, where there are reputational issues as well as monetary fines, or injury or death of personnel."
Eric Byres says, "I think we make this too complicated. Start by looking at the potential consequences of a deliberate cyber attack. This is completely company/operation specific—if you operate an automatic car wash, then the consequences you face are low, and the money you spend should be low as well. However, if you operate an off-shore oil platform, the consequences are significant, and your security investment needs to be significant as well."
He adds, "You can also look at the probability of an attack, but that is hard and changing fast and for the worse. Therefore I would simplify things and assume that in the next few years, the probability for cyber events will approach that for physical events, such as theft, vandalism, sabotage or accidents. Of course, this is also company/operation specific—the carwash will not have the same enemies as the oil company, but for a specific company the adversaries will likely be similar."
"Now assuming your company has figured out a reasonable risk benefit equation for physical events, and assuming that the consequences and the adversaries will be roughly the same over the next few years, then the cybersecurity spend should be the same, too," Byres says. "Maybe not dollar for dollar, but as an executive at an oil company once said to me, 'If we spend $50 million for fire suppression on our offshore platforms, and we spend $50,000 for cybersecurity on those same platforms, and both types of incidents have the same consequences, then we have a problem. Either we are spending too much on fire suppression or too little for cybersecurity."