Operators in the midstream oil and gas industry know their main priorities are to keep product flowing through the pipeline — and making sure it’s done in a safe and secure manner.
With that laser-like focus, it’s easy to get tunnel vision and lose sight of the proper operating scope for the network and devices. After all, when things are running within scope, it’s natural to stay absorbed by the mission of keeping product moving through the pipeline from one end to the other. But, without proper operational visibility into what is really happening, a subtle change may end up causing a costly problem in the weeks ahead.
Rapid Expansion Technical Issues Make Pipeline Operational Visibility a Challenge
The U.S. midstream oil and gas equipment market is expected to grow from $697.17 billion in 2017 to $983.73 billion by 2026, according to a Polaris Market Research report1. At the same time, the industry is ramping up increased connectivity through digitization efforts aimed at improving efficiency and reliability.
On top of these changes, let’s face it, operators face a myriad of issues that challenge achieving full pipeline operational visibility or robust cyber security. These include:
- Extremely long pipelines that are open targets for physical or cyber attacks
- Limited visibility to the components that make up the pipeline system
- Poor communication practices regarding new components, such as new tie-ins (feeder pipelines)
- Inadequate ability to see or detect developing operations problems
- Multiple customers on various pipeline segments, all operating with different levels of security
- Naive reliance on customers for good security practices
- Hard-to-correlate data from different customers along the pipeline
- Impractical nature of manual audits
Understanding what is on the network, and filtering through all the data to make smart decisions to protect against any kinds of anomalies or cyberattacks are top issues. Network and asset visibility are a must.
“Ghost Drift” PLC Failure Costs $1.9 Million
An example of the challenges of visualizing a potential problem is the case of a pipeline organization that had a truck offload-onload skid. This facility pulls oil off a pipeline and hauls it away in tanker trucks.
When a PLC went down, the truck onload and offload terminal backed up to the point where it cost the company $1.9 million in lost revenue and downtime. In the midstream market, time is money. If you suffer unscheduled downtime, it’s unlikely that you’ll ever make up the lost time /revenue, because you’re always moving product at high capacity.
However, if the pipeline operator had an industrial network monitoring solution in place, they would have been able to predict a potential outage when the PLC started to behave abnormally. And, if the operator knew the type of PLC , the type of cards it was running, the firmware and the serial number, it would have been possible to quickly diagnose the problem.
On top of that, by adding an enterprise-wide centralized network monitoring solution, it would then be possible to look at all similar devices within the eco-system. If one PLC behaved strangely and it cost $1.9 million, it’s worthwhile to flag the 25 others and watch to see if they start behaving like the problem PLC. If they do, the issue can be mitigated before it causes a bigger problem.
In triaging this problem, one of the pipeline company workers said they noticed some “weird” operational values, but since no problems were trigered, they assumed it was normal behavior. It never really fell out of scope.
When asked how long he trended the data, the worker said, “Oh, just a few months.” If it was already failing when he started observing the trend, the abnormal behavior looked like part of normal functioning, because the change was slow and gradual.
The operator didn’t trend the PLC back to when it was operating well. And, the operator didn’t compare its behavior trend to similar devices to see how the others were operating. He could have checked whether all devices with the same load were behaving similarly, or not.
In this case, the problem with the PLC was that it suffered from “ghost drift”. This is when a device slowly and quietly slips out of scope over such a long period time that no one ever notices.
It is kind of like watching your son grow through his teen years and not noticing he shot up five inches over the last six months — until you suddenly have to get him a completely new wardrobe.
With this PLC, when something started to fail, it skewed the numbers ever so slightly that it was not noticeable.
Automated Monitoring of Industrial Networks Identifies Operational Anomalies
In this scenario, pipeline operational visibility comes into play. Today’s ICS network monitoring solutions can detect when devices are starting to drift. They alert the operator that it’s time to take a closer look before another unplanned downtime incident wracks up a $1.9 million loss.
If you not familiar with passive network monitoring, here’s how it works. Typically, an appliance is attached to a SPAN or mirror port of a switch or router on the pipeline. The application on the appliance observes network traffic and builds a model of the pipeline’s network and operational behavior, employing machine learning and artificial intelligence (AI) to deal with today’s complex systems.
There are two phases to the implementation of the network monitoring application, the learning phase, and then an operational protect mode. After installation, the application quickly learns the system, and then it can start detecting operational changes.
From a cybersecurity perspective, a potential problem is that when you first plug in the appliance, if you have a malicious malware beaconing out to an external server, the application learns that as normal behavior. To deal with this problem, the best passive monitoring solutions use a technique called Dynamic Learning, where they go through the learning phase and then conduct a statistical process control analysis. If the system’s behavior is within one standard deviation, it will tell you it has learned it, and will start monitoring based on that behavior.
Operationally, if ghost drift has been occurring over a period of time, that behavior will hide in the one standard deviation, and you have to go through the results with the operator and do the due diligence needed to eradicate the problem. One way to do it is to compare the operational behavior of similar devices across the pipeline system and see if the trend for one of the devices is different from the others.
This exercise truly educates the operator about their process inside and out!
When that happens, good cybersecurity becomes a byproduct, and the operator can flip the monitoring application into protect mode. Going forward, the operator will immediately know if there is any kind of drift or any kind of malicious attack, because the system will generate alerts with its laser-like focus.
Avoid Downtime with Pipeline Operational Visibility – and Improve Cyber Security Too
Accurately documenting the network and asset infrastructure of a SCADA system like a long-distance pipeline used to be virtually impossible to do, especially in terms of keeping it up-to-date. It was also next to impossible to monitor all of the types of equipment involved. Now, thanks to technology advances, it’s easy to implement passive industrial network monitoring that automatically provides real-time network visualization and asset discovery.
And, the great thing is that the same solution can be used to provide early detection of both operational problems as well as cyber security incidents. In the truck offload-onload skid scenario, the Realized ROI of a visibility solution is at least $1.9 million. That’s a significant return based on improved reliability, with added, unquantified cyber security benefits.
If you’re a pipeline operator I urge you to check out industrial network visibility, monitoring and cyber security solutions.
If you would like to find out how the Nozomi Networks solution can help your organization better monitor its pipeline or any other industrial process, please request a demo.
