This week’s Pipeliners Podcast episode features cybersecurity expert Pascal Ackerman of ThreatGEN discussing security monitoring and incident response relevant to the pipeline industry.

In this episode, you will learn about the WannaCry ransomware attack, how to use tools to help prevent viruses in your network, the importance of getting the architecture right and maintaining that architecture over time, and more topics.

Cyber Security Monitoring: Show Notes, Links, and Insider Terms

• Pascal Ackerman is a Principal Analyst in Industrial Threat Intelligence & Forensics and the author of Industrial Cybersecurity. Pascal is also part of the ThreatGEN team. Connect with Pascal on LinkedIn.
  • ThreatGEN is a virtual reality (VR) industrial cyber-physical range for physical threat response training, process improvement, and team events.
• ICS (Industrial Control Systems) encompass the control systems and instrumentation used for industrial automation and process control. These systems are used in oil & gas and other key industries.
• Network Traps use Simple Network Management Protocol (SNMP) on a system network to duplicate traffic packets.
• SIEM (Security Incident and Event Management System) provides real-time analysis of security alerts generated by applications and network hardware
• I/O (Input/Output) Network is the communication between an information processing system, such as a computer, and the outside world, possibly a human or another information processing system.
• SCADA (Supervisory Control and Data Acquisition) is a system of software and technology that allows pipeliners to control processes locally or at remote location. SCADA breaks down into two key functions: supervisory control and data acquisition. Included is managing the field, communication, and control room technology components that send and receive valuable data, allowing users to respond to the data.
• WannaCry ransomware attack was a May 2017 worldwide cyberattack by the WannaCry ransomware cryptoworm, which targeted computers running the Microsoft Windows operating system by encrypting data and demanding ransom payments in the Bitcoin cryptocurrency. It propagated through EternalBlue, an exploit discovered by the United States National Security Agency (NSA) for older Windows systems.
  • EternalBlue is a cyberattack exploit developed by the U.S. National Security Agency (NSA) that has now been used by hackers to target U.S. cities.
• CD (Continuous Delivery) automates the delivery of applications to selected infrastructure environments. Most teams work with multiple environments other than the production, such as development and testing environments, and CD ensures there is an automated way to push code changes to them.
• ISO image is a disk image of an optical disc. In other words, it is an archive file that contains everything that would be written to an optical disc, sector by sector, including the optical disc file system.
• APT (Advanced Persistent Threats) is a stealthy computer network threat actor, typically a nation-state or state-sponsored group, which gains unauthorized access to a computer network and remains undetected for an extended period.
• Ransomware is a type of malware from cryptovirology that threatens to publish the victim’s data or perpetually block access to it unless a ransom is paid.
• VirusTotal inspects items with over 70 antivirus scanners and URL/domain blacklisting services, in addition to a myriad of tools to extract signals from the studied content.
• AlienVaults OTX systems is a crowd-sourced computer-security platform. It has more than 80,000 participants in 140 countries who share more than 19 million potential threats daily. It is free to use.

Cyber Security Monitoring: Full Episode Transcript

Russel Treat:  Welcome to the Pipeliners Podcast, episode 106, sponsored by EnerSys Corporation, providers of POEMS, the Pipeline Operations Excellence Management System, SCADA compliance, and operations software for the pipeline control center. Find out more about POEMS at enersyscorp.com.

[music]

Announcer:  The Pipeliners Podcast, where professionals, Bubba geeks, and industry insiders share their knowledge and experience about technology, projects, and pipeline operations. And now your host, Russel Treat.

Russel:  Thanks for listening to the Pipeliners Podcast. I appreciate you taking the time, and to show that appreciation, we give away a customized YETI tumbler to one listener each episode. This week, our winner is Michael Burton with BP Pipelines. Congratulations, Michael, your YETI is on its way. To learn how you can win this signature prize pack, stick around ‘til the end of the episode.

This week, Pascal Ackerman is returning to the Pipeliners Podcast to talk to us about cyber security monitoring and incident response. Pascal, welcome back to the Pipeliners Podcast.

Pascal Ackerman:  Thank you. Thank you for having me again.

Russel:  I wanted to follow up on our conversation last time, and in particular, I wanted to talk about security monitoring and incident response. Just to tee us off, I want to ask you the question, what is security monitoring?

Pascal:  Security monitoring is basically all of the technologies and all of the efforts you put into keeping an eye out on your systems, specifically for something going wrong. Security monitoring could be a firewall that’s reporting something going wrong.

It could be an intrusion detection system, and ICS, that says, “Hey, I’m seeing bad traffic.” But it could also be a person sitting out on the production floor or in the operator station and telling you, “Hey, my computer is doing weird stuff. Can you take a look at it?”

All of that combined gives you an overview of the security posture, the stance of your security, the way it’s implemented, and how well it works.

Russel:  One of the things that you had mentioned in our previous podcast is that you believe that the monitoring is probably the most important part of an overall cyber security implementation. I wonder if you might tell us why you believe that to be so.

Pascal:  From experience, I’ve noticed that the business we operate in, the technology we implement by itself often doesn’t apply for security features or security extras very well.

These devices were designed with controls and operations in mind, and they don’t have the overhead and resources to do the extra computation for authentication, for example, as well as the environment they’re in. They don’t lend themselves too well to be more secure in traditional ways.

A lot of times, it’s also resources from a personnel perspective. You don’t have the downtime, you don’t have the people to implement patches to do all this stuff, so the next best thing you can do is keep an eye on stuff.

For example, you have a system that’s been neglected for a while, but you just don’t have the time or the money to fix it. Put a monitor on it, so if something does try to compromise it, if something does weird happens from it, you at least know this and then you can take action before it becomes a big problem.

Russel:  I always try to come up with physical analogies for cyber security because I think it makes it easier for people that are not familiar with all of the electronics and the buzzwords and the software magic, if you will, that occurs.

The way I view this is doing an assessment is just understanding where I’m keeping all my valuables and then another portion of that would be doing security, which is kind of like putting locks on all the doors, if you will.

All of that’s good, but if I’ve got something highly valuable, even if it’s well locked inside a door and somebody wants it, if I’m not keeping an eye on it, somebody could come crack those locks, get in, and run off with my stuff.

I think the monitoring piece is really critical if you really want to keep things secure. Otherwise, how do you know until after the damage has been done? And then I’m just doing clean-up. I’m not actually preventing the bad outcome.

Pascal:  That’s so true. I’m taking note of your analogies because I’m going to use them later on. I really like them.

Russel:  Good. Glad to help.

Pascal:  That’s exactly the point. How do you know if something is secure or not secure if you don’t keep your eyes on it? We do the same with flows. We do the same with temperatures. We don’t just rely on the control to make sure that our temperature is stable. We keep our eyes on it.

We put trending charts on it. We put some person on who’s dedicated to watching those screens to make sure that our processes are functioning properly.

Russel:  How do I keep my eyes on a computer software system or a computer network? What does that look like? What are the things I do to keep my eyes on things?

Pascal:  The most effective way of doing that nowadays is to put an agent on either the network or an agent on the client itself. So you have a computer system, the agent will look at entries for files, see if there’s new files or modified files coming, keep an eye on network connections that might be coming in.

Apart from that, if we cannot touch the end device directly, we look at traffic on the network and this can be passive. We can install Network Traps. That’s basically an area of the network where you duplicate every single traffic packet that comes by.

You start inspecting it with specialized software and then the software can tell us the inventory, everything that it sees on the network, but it can also tell us anomalies it sees.

For example, somebody is trying to use a password over and over again on different machines or on the same machine, it’s going to pop up with an alarm because it sees it in network traffic packets.

Russel:  Right. To me, that’s a really fascinating technology, and I know that there’s people that are using analytical approaches to this. But the idea that through software I can monitor and see things that are unusual or unexpected — those kinds of things can be extraordinarily helpful. But that stuff has got to go someplace, right?

Pascal:  Yeah. Like with everything, just because you now have something in place that monitors it that doesn’t mean that you’re safe because somebody has to look at it. It’s the same with everything we do. If somebody isn’t responsible for following up on findings then it’s useless technology.

Russel:  Yeah, to me, again, using a physical analogy, if I put cameras around my house and I capture lots of footage about everything that goes on around my house that in and of itself is not of any value.

Likewise, if I have to sit and watch all that footage that’s kind of problematic too because I actually have a life to live and a job to go to, so I need something that helps me.

What a lot of people do or what a lot of systems do now is they incorporate motion detection with the cameras, and with the motion detection, they send a notification.

People have probably seen the commercials where somebody rings the doorbell, and then via a phone application, they’re answering the door without actually opening the door as if they were home.

That’s the kind of thing that we’re talking about here where somebody gets notified when something weird is going on and then they look into it to figure out what’s going on.

Pascal:  In the cyber realm, there’s a product for that. It’s called a SIEM, or a Security Incident and Event Management System. It basically sits on an I/O network and it collects all of these logs, all of these events, and it goes through and then it puts analytics on there.

If it sees something weird, it sends out an email, so you no longer need somebody to constantly look at the events. But somebody will receive an alert by email or by text message that something’s going on and then you have the means in place to go back and look at that incident.

Yeah, that’s a new technology for the ICS environment. It’s been used for a while now on IT. It’s invaluable because once you start looking at stuff, you’ll notice really quick there’s a lot. You have events coming from everywhere.

Russel:  Trying to manually monitor things like that, I view that as virtually impossible. It’s just too much data. There’s no way to go through it all.

Pascal:  Yeah, absolutely.

Russel:  Obviously, monitoring is important and there’s tools and things out there that I can use. What happens when the monitor says, “Hey, you need to look at this?”

Pascal:  That depends on, of course, what it is monitoring and what the reports are about. Most of the time, there’s going to have to be some research behind what did we just see to rule out that it was a false positive or a false negative in the first place.

So you make a strategy on what you’re going to do next and then you’re going to have to go out there and actually address the finding.

For example, a vulnerability that’s been picked up, it could be that it’s on a system that you cannot touch for the next year because you’re running a certain process, but you have to put a plan in place to eventually address that.

Or, as people might do, you mitigate the risk by accepting it. We know that that happens, and you look at the surrounding risk involved with it.

Let’s say the vulnerability is that a certain port is accessible from the network. You might decide not to address it, but to put an extra monitor in place that says if anything tries to connect to through that port then I get another alert in my inbox.

Russel:  Yeah, exactly. I guess what I’m trying to lead to and I’m thinking through as we’re talking is at one level, I get alerts and I analyze alerts and then I take appropriate action about those alerts, which I would see as mitigation activity.

Pascal:  It depends on the alert. Sometimes it’s mitigation, or sometimes it’s just mopping up because you’re already too late because the attack happened.

And even the monitoring system informed you, but you’re already too late because the system was compromised and you’re going to have to do an incident response at that point.

Russel:  Yeah, and that’s really the thing I wanted to talk about next. I happen to be aware of a couple of oil and gas operators that within the last six months or so lost complete use of their SCADA systems because of a ransomware attack. They didn’t know they had a problem until control systems quit working.

I just wanted to ask you, do you see those kinds of things escalating?

Pascal:  Yes. Unfortunately, probably 90 percent of it, the vector that the attack occurred was in such a way it could have been detected and prevented before it hit the ICS or the control system. A lot of these will start in the enterprise, in the business systems.

I’ve seen many of them start on the active directory, on the domain side of an enterprise, and then eventually the operators or the attackers make it into what they call the domain controller.

They manipulate that in such a way that it starts spreading ransomware. Because all these systems are connected, it spreads down to the ICS system and you get a total compromise.

Russel:  My experience with ransomware is more the spearfishing stuff where you get the email that looks like it’s legitimate and it asks you to click on something, and you click on something and that opens up a vector to take an attack.

But what you’re talking about is not that type of thing. It’s more of a hacker gaining access at one point and then using that to contaminate a broader network. Am I understanding that correctly?

Pascal:  Absolutely. You’re putting the two types of attacks right in place. So the one is the drive-by. Somebody picks up a piece of malware or ransomware from the Internet and it spreads around on the network.

The other one we call a targeted attack where somehow your systems were compromised to begin with, and they use that leverage to pivot into your environment.

They’ve been sitting there for an X amount of time. The attack by itself is not geared towards putting ransomware out there, but the ransomware is basically a goodbye note.

They make sure by putting it on your systems, they wipe out any trace because what better way to hide evidence than to destroy it.

Russel:  That’s interesting.

Pascal:  Those kinds of attacks are the most devastating, in my opinion. Sure, if you get a particularly well-written piece of malware like WannaCry a couple of years ago, it’s very devastating.

But knowing that these operators have been in your system, have poked around for who knows how long, and then finally they decided to leave you with a big mess, I think that’s more devious than anything else.

Russel:  I’m not aware of anybody actually paying the ransom.

Pascal:  I hope not because we’re just helping them at that point.

Russel:  Yeah. Generally, it’s pull out all the infected equipment and either scrub it and rebuild it or buy new equipment and rebuild it. But in either case, it’s a rebuild everything from scratch kind of thing.

Pascal:  I agree. That should always be your way to go. Because even if you paid the ransom, what’s preventing them from coming back and using the same infection over again and coming back for more ransom money?

Russel:  Yeah. So what do you do when you discover something like that? We’re talking about incident response, but really, what do you do?

Pascal:  The most important thing, and that’s what most people fail at, is don’t panic. Don’t start pulling plugs, shutting stuff down, rebuilding it. Make sure that you keep enough evidence to do forensics afterward.

Leave systems turned on, but unplug them from the network. If you have the means, start rebuilding your system from fresh hardware. So get new servers in. Install operating systems from the original install disk and start from scratch.

By all means, try not to get rid of the evidence. That’s going to be very important later on if we decide we want to do forensics on this stuff and find out maybe even the key.

Maybe we can find the decryption key and get some data back. But as soon as you start rebuilding systems and reformatting hard drives, you’re losing very critical evidence.

Russel:  Yeah, well, that’s certainly true. I guess the thing, as you’re talking, that I’m thinking about is a lot these, particularly the smaller operators, they don’t have the resources or the wherewithal really to do the forensics.

They’ve just got to get their operations back under control, up and operating and being monitored versus trying to protect machines and do forensics and all that. Those activities are complicated and can be expensive is certainly my experience.

Pascal:  Yes, most smaller companies will either just rebuild and cut their losses, or they bring in an external company to help them with the forensics.

If anything, if you do nothing else, start from scratch. That’s always my advice because you don’t know what else was on that system. There could be any type of malware on there.

So always make sure you format your hard drive, you start from a known, secure medium. Even backups, depending on how long an attacker’s been on your network, your backups can be compromised as well.

Russel:  Exactly.

Pascal:  You’ve got to assess what happened, what kind of attacker did we have on there, and then go back to a known state that you know you weren’t infected yet, if you can completely prove that, or start from scratch. Get new install CDs, get new ISO images from the Internet that have your operating system, install your control software — everything from scratch; do it that way.

It’s not a pleasant operation. I’ve done a few of them, and it’s cost me a lot of hours to do it, but it is the only guaranteed way to make sure that you’re completely clean again.

Russel:  Yeah, I think you’re making a couple of points that I kind of notionally understood but you’re clarifying them for me. One, if I discover an attack, the bad actors have probably been poking around in my network for a while.

They probably didn’t deploy an attack immediately after they gained access. They probably poked around for a while, which means anything that I restore as a backup or anything else I’m at risk of just giving them access again.

Pascal:  Correct, yes, again, depending…

Russel:  That’s not a pleasant thought, Pascal.

Pascal:  [laughs] And that’s where forensics, or at least high-level forensics comes into play. So prove out what kind of attack you had. When WannaCry came out, it was quickly really clear what kind of attack it was.

It wasn’t a targeted hacker who’s been on your systems for a while. It was a piece of malware that used any kinds of means to spread through your network and cause mayhem.

So that’s what you do. You look at the attack, you decide, okay, this was a drive-by, I’m not too concerned, or this was something targeted and I have to be very concerned.

Russel:  Again, every time I talk to Pascal, yourself, or someone else I know that’s a cyber security guy, I’m always learning stuff.

Pascal:  I’m always learning stuff, too. That’s just the nature of the beast.

Russel:  But the idea that part of your immediate incident response should be some kind of high-level forensics and preliminary analysis and assessment to determine if you need to do something more detailed, I hadn’t thought about it that way, but that certainly makes sense to me versus not doing anything and just rebuilding.

Pascal:  Exactly.

Russel:  Even if I rebuild a system, and I install fresh software — or restore my app, or I restore backed up data — I’m running the risk of putting the infection back into the system.

Pascal:  And not only that, you’re probably alarmed whoever was attacking you that you’re on to them, unless they’re already gone. But then they know, “Hey, we need to change our tactics,” and it might come back in a different way and attack you again.

Russel:  What’s the motivation behind these attackers?

Pascal:  It depends on who’s attacking you. You have opportunity attackers who put in ransomware just to get your money out. Then you have state-sponsored attackers from China or Russia or whoever the other popular flavor is this month. They will go in and they will look for proprietary information for secret recipes.

There are known attackers, they call them APTs, or advanced persistent threats, who infiltrate systems just to be able to sit there and disrupt operations at some point for some reason.

Russel:  Yeah. I guess if I’m competing with somebody, and I can steal their technology and then slow their ability to get a release out, I can beat them to market with their product.

Pascal:  There you go. In all honesty, those are the kind of stories that people tell you to buy their products and to make you scared. Most of these attacks, they start with an opportunistic attack — somebody going to the wrong website and picking up a piece of malware that opens up a port that somebody picks up on.

They’re curious to see what’s going around. They poke and probe to the point that they break something and that’s when you notice this.

From all of these attacks, from all of these ransomware outbreaks, there’s only 10 percent that are really targeted attacks.

Russel:  Right. I guess the summary here is monitor, make sure you’ve got some kind of mechanism to proactively alert people that can respond and mitigate and head things off.

And when you do have an incident, disconnect whatever you know that’s infected from the network and do some level of forensic analysis to determine where you need to go from there.

Pascal:  Yes, and that can be as simple as looking at the ransomware itself. So if you take the executable and you upload it to something called VirusTotal —  it’s virustotal.com — it will tell you what kind of ransomware it is and how it propagates and that’s important.

For example, like I said before, WannaCry has the mechanisms to self propagate, so it can jump from system to system. But an incident response I was performing awhile back, they had a piece of malware that didn’t auto propagate. It had to be installed and started by a person.

That difference tells a lot of details because if you have something that has to be installed by a person, that means that there was an attacker actually actively starting that on your system.

If you have something that’s self-propagating, it could have come into your system in any way and just jump from system to system.

Russel:  Yeah, and that goes to why the forensics are important even if you’re doing something fairly fundamental because there are entities out there that full time are categorizing and logging and analyzing all these various types of tools and bots and stuff that these nefarious actors use.

Pascal:  Yeah, and I can give two good resources in the show notes for the listeners. They are VirusTotal and AlienVaults OTX systems.

You basically go there with either the executable itself or a hash value, which is a computational value on the executable, and you can find all kinds of resources, all kinds of details about that. That’s always the very first step I do when I’m involved with a new piece of malware.

Russel:  Right. Interesting. If we’re going to wrap this up, I guess there’s a lot of people out there, probably more than we realize, they’re running these automation and control systems and really don’t have a lot of exposure to the details of cyber security and the threat and the mechanisms that are available.

If you were going to say here’s two or three things that I’d tell you that everybody needs to know about cyber security, what would that be?

Pascal:  Make sure your architecture is up to date. If nothing else, your architecture allows for looking at network packets, doing packet inspection. Make sure you have a comfortable setup to look at alarms and make sure you actually look at these things. Those are my three most critical points to point out.

Russel:  Well, you know what, that sounds really easy, Pascal, but I know enough to know that’s not easy to do.

Pascal:  [laughs] No, that is not. Getting to the point where you can actually look at these events is probably fairly complicated depending on how your architecture is laid out right now. A lot of the companies I’ve been involved with, they have pockets of automation or pockets of network systems where they have a switch, they go, “Oh, that’s handy to put here. And now, I can hook up all of these things.”

But that doesn’t give you the visibility for that particular area, so getting your architecture to actually report on all of these findings is probably the hardest part.

Russel:  Yeah. I know too that particularly when you go look at the smaller operators and those that are quickly building new stuff, or quickly acquiring stuff, the level of complexity around doing this and doing it well grows exponentially.

Pascal:  Yes, absolutely.

Russel:  So getting the architecture right and maintaining that architecture over time, very, very important.

Pascal:  Yes.

Russel:  All right. Well, look, Pascal, this has been helpful. I’ve certainly learned some things today, and I very much appreciate you coming back on the podcast. We’ll certainly have you back again as time moves forward.

Pascal:  I’m looking forward to it, thank you.

Russel:  I hope you enjoyed this week’s episode of the Pipeliners Podcast and our conversation with Pascal. Just a reminder before you go, you should register to win our customized Pipeliners Podcast YETI tumbler. Simply visit pipelinepodcastnetwork.com/win to enter yourself in the drawing.

If you would like to support the podcast, the best way you can do that is to leave us a review. You can do that on iTunes, or whatever smart device app you use to listen to podcasts on your phone. You can find instructions on our website at pipelinepodcastnetwork.com.

If you have ideas, questions, or topics you’d like to hear about, please let me know on the Contact Us page at PipelinersPodcast.com, or reach out to me on LinkedIn. Thanks for listening. I’ll talk to you next week.

[music]

Transcription by CastingWords