Tabletop scenarios, exercises, simulations, live cyber ranges, and preparing for failure modes and manual operations are imperative steps in planning for cyber disruptions to the electric grid. These events and learning measures often involve dozens of state, local, commercial, and federal teams, working together to prepare owners and operators of generation, transmission and distribution assets for disruptions, downtime, and emergency situations.
Conducted by the Department of Energy’s (DOE) Office of Cybersecurity, Energy Security, and Emergency Response (CESER), Liberty Eclipse is an exercise that convenes security operations center and operational technology (OT) experts from multiple utilities to test their cybersecurity plans in a state-of-the-art cyber-physical range. As part of its annual practice, the Electricity Information Sharing and Analysis Center (E-ISAC) coordinates the GridEx exercise. GridEx involves a scenario with physical, cyber, and operational events and injects.
Let’s take a closer look at what the Liberty Eclipse and GridEx exercises involve, and lessons learned about the sophisticated nature of modern electric grid infrastructure.
Liberty Eclipse (October 2023)
Liberty Eclipse is a Blue Team vs. Red Team live exercise with commercial grade operational systems from multiple vendors, not connected to the broader electric grid. Participants also bring their own hardware and software, security equipment and architectures to improve incident detection and response processes against realistic adversary activity.
“Following periods of adversary activity, the Red Team discussed their approach, actions, and what the utilities saw and how they responded in a discussion with the Blue Team.”
- Office of Cybersecurity, Energy Security, and Emergency Response, "Practicing Defense and Resilience at Liberty Eclipse"
Apart from this threat hunting and incident response capacity, participants historically have the opportunity to practice black start scenarios and islanding. Black start is a term for starting generation assets that cannot start on their own, where operators decide what gets energized when and how. Electrical islanding means only connecting infrastructure over a few points for operations, like interchange and synchronization points, prioritizing critical demand location like hospitals, military sites, etc.
In addition to utilities and asset owners, Liberty Eclipse and GridEx involve multiple volunteers and experts from the DOE, its National Laboratories, the SANS Institute, Federal agencies and the National Guard. Potential events and scenario playbooks analyzed in these exercises include:
- Disrupting generation, connection points or critical delivery infrastructure
- Malware targeting specific relays RTUs and SCADA systems
- Communications routers, switches, and firewalls
- Bricking device firmware
GridEx VII (November 2023)
The latest GridEx involved participants from across North America and beyond. In total, entities from 44 U.S. states, seven Canadian provinces, Mexico, and New Zealand were in attendance. The exercise unfolded as follows:
- Move 0: The System is Vulnerable. The focus is on investigating growing threats and categorizing needs and expectations.
- Move 1: Softening the Target. A series of cyberattacks made it harder for utilities to respond. A ransomware attack caused internal IT software and the third-party systems that operate the electric market to go out of service. An ICCP data link outage disrupted communications, and a natural gas pipeline flow disruption led to reduced generation capacity in the region.
- Move 2: The Coordinated Attack. A coordinated physical attack targeted multiple substations with assailants directing gunfire at critical transformer components. Several transformers, lines, and generators tripped automatically, causing outages across a large operating area. As utilities responded to these attacks, a distributed denial-of-service attack against the corporate virtual private network system rendered remote access intermittent or impossible. Meanwhile, misinformation and disinformation circulated on social media
- Move 3: Recovery Under Pressure. Transmission and distribution field breakers tripped due to unknown causes, interrupting grid supply to critical facilities. A vehicle-borne improvised explosive device detonated at a telecommunications facility, removing voice and data communications at an RC control center. The public became increasingly frustrated by power outages, and, as misinformation and disinformation continued to spread, protestors gathered and began to harass utility personnel. Explosives detonated at equipment storage and staging areas, damaging and destroying spare equipment needed to restore service. This compounded supply chain and market disruptions.
- Move 4: A Week Later. Players discussed and explored recovery and longer-term considerations. Global supply and diesel fuel shortages continued, delaying restoration and repair efforts. For the foreseeable future, no spare equipment was available, and entities had to rely on their current inventories.
Lessons Learned
These exercises reveal the sophisticated nature of modern electric grid infrastructure, as well as the multitude and magnitude of involved parties and personnel for incident response and emergency preparedness. Among the lessons learned from the most recent Liberty Eclipse exercise and GridEx VII, three major needs stand out for improving capacity for all invested participants.
1. There is a need for more state and local resources for emergency response planning and coordination, especially for events that may impact multiple towns and counties simultaneously.
Emergency planning and response has taught us that all emergencies happen locally. Internal stakeholders include network management teams, information security teams, OT system operators, and governance, risk management, and compliance (GRC) teams at minimum. External parties include local, state, and federal government agencies, citizens, the media, businesses, and corporations, and more.
Aligning these stakeholders in a coherent way for planning, exercising and preparing for cyber intrusions and disruptions to the electric grid is a constant resource-scarce initiative. Every organization has a role to play in recovering critical functions in an emergency impacting critical infrastructure, and more input and participation is needed across the board for incorporating state and local participants into cyber-physical exercises and simulations.
2. There is a need for alignment of cyber specialists and system operators to triage and respond to events, including the translation of technical details for non-technical audiences.
It is very difficult to relay vital information if you do not speak the same language. It further delays restoration of critical functions and recovery of grid operations to spend time building a common dictionary for siloed teams and specialists. Cyber incident responders leverage computer incident response tools and training to investigate and characterize the nature of incidents. Operators manage critical systems and have the tacit knowledge required to recover critical business functions in the event of disruption or damage.
Cyber specialists and system operators have to do more frequent and formal knowledge transfer in preparation for exercises and real-world events, to be able to triage in emergency situations. As nation-state capabilities continue to present themselves in critical infrastructure networks, foundational training and awareness for these teams as well as the stakeholders outlined above is the only path to creating a more secure and resilient future response team for defending the electric grid.
3. There is a need for more robust situational awareness.
Many organizations feel capable of calculating things like maximum tolerable downtime and mean time to recover from an incident. However, according to the GridEx VII report:
“Corporate functions such as communications, security, and customer care were not always comfortable with grid security incident response processes, making it difficult for them to support responders and the operational needs of the organization.”
- GridEx VII Lessons Learned Report
Operators have to know the current status and settings of infrastructure responsible for current and voltage on the grid. Cyber incident responders must determine the extent of infiltration and capabilities under an attack, and how to trust process and systems equipment after recovery. Better operational and communications data and visibility is key to accessing, utilizing, and assessing situational awareness in any potential attack or crisis scenario.
About Nozomi Networks
From day one, Nozomi Networks’ solutions have been deeply rooted in addressing the complex requirements of industrial and critical infrastructure environments. As OT converges with the vastly different worlds of IT and IoT, that experience has given us a unique understanding of the tools and processes associated with the largest networks in the world. We’ve earned a global reputation for unmatched service, superior cyber and physical system visibility, advanced OT and IoT threat detection, and scalability across distributed environments.
Nozomi Networks’ automated asset identification saves time and helps achieve a centralized view of your ICS and its related assets. Nozomi Networks’ comprehensive vulnerability analysis supports prioritized and efficient risk reduction efforts with actionable insights on remediation steps, patches, and upgrades. The platform provides visibility into unique protocols, like DNP3 and IEC 61850, monitors specific devices including RTUs, and simplifies NERC CIP compliance with automated asset inventory and vulnerability assessments.
Content Packs bundle queries, reports, Playbooks, and dashboards into a single file for easy distribution to teams. They can be edited and applied across various systems, which is particularly handy for intricate reporting needs like government compliance or targeted threat detection. Nozomi Networks has released several new Content Packs that will help administrators save time and address their compliance requirements with confidence and consistency.
