When Florida law enforcement announced there had been a cyberattack on the Oldsmar water district, the news hit way too close to home. A few years ago, I lived and worked in and around the area. Today my family and I live close to Pinellas County, which manages the Oldsmar water district and whose Sheriff is currently investigating this incident alongside the FBI and United States Secret Service. My wife is in Pinellas County several times a week, and my 5-year-old son and I frequent the area. My family and many, many of our friends drink this water regularly.
That’s why it was professionally and personally alarming to learn that an attacker leveraged the TeamViewer app to remotely access the city’s water treatment system and dial up the sodium hydroxide levels by more than 100x. Thankfully, the issue was corrected before harm could be done, but the incident should be a wakeup call for water districts everywhere.
A Low Sophistication Cyberattack
An analysis done by the Nozomi Networks’ Labs team mentions the lack of sophistication of the attack, which raises a red flag. If it’s this easy to access Oldsmar, what about the other 50,000 (literally, there are 50k+ of them to consider) water production utilities in the United Sates? If a low-sophistication attacker can, with a few mouse clicks, begin the process of mass poisoning the population, what could a medium or highly skilled attacker do? When it comes to protecting my family, the bar can never be set high enough. However, in this case, the bar seems way too low. How did we get here? What went wrong?
Based on the information available at this moment, this attack seems to lack any sophistication that could trigger more profound reactions. The fact that the perpetrator didn’t conceal their visual presence to the personnel monitoring the water treatment operation is the first signal that suggests the relatively low complexity of the attack. Furthermore, according to the reports of the incident, the attacker increased the levels of sodium hydroxide by a significant amount, typically monitored by automated systems, which means it’s likely that the threat actor didn’t possess a specific background knowledge of the water treatment process.
Nevertheless, this incident is important because it reflects the status of too many ICS installations, especially those with smaller budgets and a smaller size where security is often overlooked. Remote access, in particular, when not designed with security in mind, is often the beachhead used by remote attackers to infiltrate an ICS network. In this very case, the water treatment plant of Oldsmar has been using a TeamViewer instance, which apparently was accessible from the internet. While it is not known at this stage how the attackers obtained the credentials required, this incident, like many that we’ve documented in recent years, didn’t seem to rely on sophisticated zero-day exploit for its execution.
Nozomi Networks Labs
TeamViewer and Other Remote Access Tools
COVID’s impact, and the resultant mass rollout of remote access technologies to the detriment of cybersecurity has been discussed in great detail in the cybersecurity space. In many water facilities – as well as other utilities and industries – the need to keep systems up and running in the midst of the viral pandemic prompted many operators to face harsh realities.
In order to work from home, the production control networks need to be accessible from home. Enter TeamViewer, and many other remote access applications like it. Prior to COVID-19 using these apps to access critical process controls might be described as negligence or used with disregard for security. However, today we have legitimate reasons for needing remote access and monitoring during the pandemic.
Lowering the Risk by Leading with Process
Understanding that we’re going to have to live with these types of remote access and supply chain risks for the unforeseen future, how do we reduce the impact of the risks, as much as possible?
First, we must remember the whole point of the (risk assessment) exercise. Why bother to protect one thing and not another? If the whole purpose of an enterprise is to produce water, then start there. How is the water produced? What specific OT processes are involved? And what are the risks to those processes? If we lead by the process, we can see some devices are less worthy of our attention than others. Do I need to secure my iPhone, or the digital control panel in the operations center?
How Could This Help Mitigate Attacks Like Oldsmar?
Water districts that invested in cybersecurity can answer many of these questions quite easily by looking into their state-of-the-art asset inventory software which tracks every facet of the critical equipment used in the process. Unfortunately, many municipalities aren’t budgeted for state-of-the-art cybersecurity.
Of the facilities with an asset inventory, many are using IT products that don’t do a good job of handling operations technology equipment, which is highly proprietary. In fact, many of these water districts are purpose-built specifically for their exact location.
Of the facilities monitoring their network, only a subset of them run network anomaly detection, technology that many rely on to hunt for previously unknown threats. However, in the Oldsmar attack, no network anomalies (from behind the firewall) would have been generated due to the attacker leveraging an officially sanctioned remote access pathway (using TeamViewer to set OT parameters on the PLC/DCS from the HMI).
There’s a chance the attacker was from out of the area, which may have shown an anomaly from the firewall standpoint, but if the attacker was in the neighborhood, the connection could have masqueraded as a county employee working at home. Or it’s possible the attacker breached a completely unrelated device, like a personal tablet, which was plugged into the same Wi-Fi network used by a remote-work engineer responsible for managing the plant. Then the attacker(s) stole the credentials to TeamViewer, and subsequently used the credentials to log into the facility and attempt to create damage.
Due to the Human-Machine Interface (HMI) functionality existing on the same machine as TeamViewer access, no network anomalies would have been present. In that scenario, the attacker(s) would have already had the necessary access to the process. There was no need for reconnaissance activities, or lateral movement. It’s as if the front door of the bank opened directly to the safe, with nothing or no one in between, including security. As unfortunate as it may be, network anomaly detection technologies aren’t a good fit to mitigate this specific use case.
Another common endeavor is to conduct system hardening by identifying and patching vulnerabilities. However, in the case of Oldsmar vulnerabilities weren’t needed (with the possible exception of TeamViewer itself, still to be determined). While deducing vulnerabilities is a crucial part of a solid cybersecurity program and the importance shouldn’t be diminished, it wouldn’t have prevented this attack.
The Crème de la Crème of Protection
A very small subset of water production and other critical infrastructure facilities that have a detailed asset inventory are monitoring their network for attacks, and have deployed and operationalized network anomaly detection solutions. This is the crème de la crème of protection. This approach leverages artificial intelligence to run anomaly detection against the actual parameters that are used to control the industrial process.
Take a pump for example. If it’s rated to spin at 100 rotations per minute (RPM), it wouldn’t be safe to run it higher. The engineers that programmed the HMI, shouldn’t allow dangerous conditions to be present, and in most cases would prevent operators from entering invalid inputs or introducing unsafe conditions like sending a parameter value to the pump to spin at 150 RPM.
Protecting the Process
The Oldsmar attack happened within the stream of data used to monitor and control the process. The attacker used a legitimate HMI to send a legitimate packet with a legitimate payload which increased the quantity of sodium hydroxide in the water. If anomaly detection was being applied to the OT parameters in the Oldsmar water facility, a successful attack elsewhere within the enterprise could have attempted to impact the industrial process using a variety of methods.
Maybe it was this particular HMI today, perhaps yesterday they injected packets directly on the network. For all we know, the attacker(s) could have moved laterally for weeks, and are still there today, maybe even using a different device in the industrial process to send commands to disable an automated safety system that is widely referenced as being there to prevent toxic water from reaching the population. The only way to know for sure is to monitor the actual industrial process tags and values to look for anomalies within. Not only would the Oldsmar attack be addressed, but many other risks as well, including negligent or compromised insiders.
Oldsmar and many other OT-specific attacks have something in common: they all impacted the process, and used the approved technology infrastructure to target the process. When the infrastructure is the target, attacks are detected more easily, but when the infrastructure is used to attack the process, things get more complicated, and more sophisticated tools are required to mitigate the threats.
Fortunately, many of our customers are very well equipped to deal with these scenarios. By leveraging Nozomi Networks solutions, they’re able to go from having almost no visibility straight to the crème de la crème of industrial cybersecurity monitoring. The entire journey can take place in as little as a few hours.
Nozomi Networks customers are leveraging the comprehensive asset inventory, network anomaly detection, vulnerability management, investigative graphing tools, forensics, reporting, real-time dashboards, and partner integrations to not only quickly identify when an attack like Oldsmar happens, but also automate and orchestrate the appropriate response to such an egregious intrusion of their facilities. In many cases, the value is immediate because these capabilities exist right out of the box, with little to no configuration required.
Today, it’s TeamViewer. Yesterday, it was SolarWinds. The week before, Ransomware #5,001. Let’s face it, the risks are pervasive and persistent. Full end-to-end monitoring of the critical industrial processes should be the goal, and the norm… and not the crème de la crème.
Related Links:
- Blog: (Likely) First Cyber intrusion Into An American Water Treatment System
- Blog: SolarWinds Cyberattack: Layered OT Security Creates Best Defense
- Webpage: Nozomi Networks Labs
