There have been more than 130 control system cyber incidents in water/wastewater utilities. Like Oldsmar and Discovery Bay, most of these incidents have occurred in small water utilities. Many of these incidents were not publicly disclosed, nor were the utilities required to disclose these incidents. Additionally, some of the real cases that were made public were later discounted such as the 2011 Illinois Water Hack where a small water utility had a water pump damaged from remote access into the SCADA system from Russia.
As Charles Dickens stated, “It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, …” This aptly describes the responses to the January 15, 2021, Discovery Bay, California, and February 5, 2021, Oldsmar, Florida, water system “hacks” to the OT cyber security community including the government.
It was the best of times for the OT cyber security community
The OT cyber security community was chomping at the bit to have a SCADA hack involving critical infrastructure made public so it wouldn’t be a “not if, but when, when being now” scenario. The Oldsmar “hack” was identified and made public through the Pinellas County sheriff’s news conference. The supposed “hack” involved the use of TeamViewer, a remote access tool which the OT cyber security community knew, even if they didn’t know how a water treatment facility worked – the epoch of incredulity. The major OT cyber security players provided their expertise on the “hack”. Examples included Eric Chein from Symantec, now Broadcom, was quoted as saying: “These are the targets we worry about. This is a small municipality that is likely small-budgeted and under-resourced, which purposely set up remote access so employees and outside contractors can remote in.” Dragos issued a report: “Recommendations Following the Oldsmar Water Treatment Facility Cyber Attack” including details of the event which were not correct. Even some of the key individuals making national cyber security policy got involved: “Frankly, they got very lucky,” retired Adm. Mark Montgomery, executive director of the federal Cyberspace Solarium Commission, told ProPublica. “They shouldn’t celebrate like Tom Brady winning the Super Bowl,” he says. “They didn’t win a game. They averted a disaster through a lot of good fortune.” Eventually EPA used the Oldsmar case for their water/wastewater cyber security requirements. The quick disclosure also met the need for expeditious cyberattack reporting (even though it was not a cyberattack). None of these organizations have addressed the fact that Oldsmar was not a cyberattack, but operator error.
It was the worst of times for the OT cyber security community
Obviously, expeditious reporting doesn’t count when it comes to the FBI. Moreover, the Discovery Bay hack was similar to the Australian water hack in 2000. In the Australian case, the hacker was caught following the 46th time he remotely opened the sewage discharge valves on a traffic stop. Truly, the age of foolishness.
Similarities in the two facilities
Both served small towns of almost the same size, used well water, had external SCADA support, used TeamViewer, used similar types of control systems, used similar chemicals, and could be operated manually.
Similarities in the two incidents
Both incidents involved incredible scenarios. For Oldsmar, a setting was changed that was beyond the capability of the control system. A properly designed system would not have allowed a value out-of-range to be set. The system would also have logged the user and time the value was input. For Discovery Bay, the SCADA software and associated displays were unavailable which shouldn’t happen with the back-up capabilities.
Differences in the two incidents
Oldsmar turned out not to have been a cyberattack, but caused when someone with remote access mistyped a value the program accepted even though it was egregiously out of range of the equipment. The operator was able to catch the mistake before anything further transpired. It is not clear how the operator error reached the Sheriff’s office, but it is a small town. As Oldsmar had local SCADA/I&C support, the county sheriff had precedence and could make a public announcement and did so without further detailed verification.
The other incident involved deliberate human misconduct. Discovery Bay contracted the operation of the water treatment facility to a private supplier of water services with headquarters in Boston, Massachusetts. For this reason, the FBI was involved, so there was no public announcement until the indictment of the Instrumentation and Control Systems (I&C) technician Rambler Gallo was issued June 27, 2023. The Discovery Bay hack can be viewed as “Living off the Land” by a possibly disgruntled I&C technician. As such, Gallo’s attack couldn’t be found from IT or OT network monitoring until the SCADA software was uninstalled. As Gallo’s core responsibilities were maintaining all field instrumentation and Programmable Logic Controllers (PLCs) used to control electromechanical processes, including instrumentation calibration, equipment upgrades, SCADA and SCADA upgrades, troubleshooting, and PLC improvement, he had full access to the field instrumentation and SCADA system. Specifically, Gallo had permission to install remote access software, TeamViewer, to his “corporate” laptop to remotely monitor and control the water treatment system. However, he also surreptitiously loaded the remote software onto his personal laptop. He then used his remote access to uninstall the commercial SCADA software, Ignition, running in the utility control room leaving the utility with no view or control of the process while at the same time maintaining view and control from his personal laptop. He was then able to remotely change or modify instrumentation (e.g., process sensors, actuators, valves, etc.) as well as control system configurations and logic. Gallo also demonstrated that engineers and technicians can compromise control systems without needing external network expertise. According to a confidential report compiled by the Northern California Regional Intelligence Center (NCRIC), the hack was not discovered until the following day, January 16, 2021. The facility subsequently changed its passwords and reinstalled the programs. “No failures were reported as a result of this incident and no individuals in the city reported illness from water-related failures,” the report noted. Yet, it took 2½ years for the information to become public.
No requirements available
Neither AWWA, EPA, or CISA cyber security guidance or requirements are designed to address Insider security threats. As mentioned, Gallo had access to all field instrumentation which has no cyber security and is not addressed by water cyber security guidance. The Oldsmar case was an operator mistake that was accepted by the locally designed SCADA system. This showed a significant design flaw in the SCADA system that could allow an egregiously large number to be accepted. It is unclear if the SCADA system had appropriate logging to identify who input the out-of-bounds value. It also exposes a question about “credible disclosure” to law enforcement. In both instances, if the SCADA and/or instrumentation were compromised in a manner that resulted in the systems “being in a credible range”, the impact may not have been identified by monitoring the OT networks and would have needed engineering input. Just like other infrastructures, the water/wastewater cyber security focus is on the Internet Protocol (IP) network issues and OT network personnel ignoring the other cyber vulnerable systems. The cyber incident reporting requirements addressed in the National Cyber Security Plan and other government and industry documents don’t address the FBI non-disclosure protocols and the recent Security and Exchange Commission (SEC) cyber disclosure requirements. From a personal perspective, I remember in 2001 when the Chinese cyber attacked the California Independent System Operator (CAL ISO), the FBI prevented CAL ISO from making any disclosure of the event. Another case that was similar to Oldsmar was the 2013 PG&E Metcalf substation attack on the transformers. Since it appeared to be a local event, the Santa Clara County sheriff went public with details on the event until the FBI got involved and stopped any further public discussions. What kind of calamity will it take for people to wake up and get the right people involved?
