This week’s Pipeliners Podcast episode features IT and cybersecurity subject matter expert Clint Bodungen discussing the use of gamification for conducting exercises in pipeline operations.

In this episode, you will learn about how ThreatGEN is being used to support cybersecurity efforts in industrial settings, including oil and gas pipelines, how pipeline operators can benefit from implementing this tool in their team training, and how this tool can be used to support PHMSA Control Room Management requirements.

Gamification: Show Notes, Links, and Insider Terms

• Clint Bodungen is an ICS cybersecurity guru, the author of “Hacking Exposed: Industrial Control Systems,” and he teaches at the Gas Certification Institute (GCI). Connect with Clint on LinkedIn.
• Gamification is a method of using video game environments or gaming principles to simulate real-life events for training or education purposes.
• ThreatGEN® Red vs. Blue is a cybersecurity gamification platform, which uses game-based learning and adversary simulation for cybersecurity education, awareness, and incident response preparedness training.
• Colonial Pipeline Cybersecurity Incident: In May 2021, a cyber attack was launched against Colonial Pipeline that eventually resulted in the payment of $4.4 million to resolve the attack. The attackers gained access to Colonial’s systems and data through an unmonitored VPN by stealing a single password.
  • Listen to Pipeline Technology Podcast #11 for Niyo “Little Thunder” Pearson discussing why the Colonial attack raises alarms for midstream companies.
• Stuxnet is a type of virus known as a computer worm that targets SCADA systems. It is commonly referred to as the first cyber weapon.
• BEER-ISAC is a subset of the annual S4 ICS Security conference that covers ICS and OT topics.
• ICS (Industrial Control Systems) captures the control systems and instrumentation used for industrial process control. These systems are used in oil & gas and other key industries.
• OT (Operations Technology or Operational Technology) refers to the hardware and software systems that perform critical functions — such as monitoring and controlling equipment and processes — to support pipeline operations.
• Computer-Based Training (CBT) is a method of training that uses a computer or computer software to train a large group of individuals on a specific task.
• Incident Response (IR) is the plan used to prepare for, detect, contain, and recover from a data breach.
• PLCs (Programmable Logic Controllers) are programmable devices placed in the field that take action when certain conditions are met in a pipeline program.
• Software-as-a-Service (SaaS) companies create and deliver special-purpose applications that perform a business function for companies.
• PHMSA (Pipeline and Hazardous Materials Safety Administration) is responsible for providing pipeline safety oversight through regulatory rulemaking, NTSB recommendations, and other important functions to protect people and the environment through the safe transportation of energy and other hazardous materials.
  • CRM (Control Room Management) or CRM Rule (Control Room Management Rule as defined by 49 CFR Parts 192 for natural gas and 195 for liquids) was introduced by PHMSA to provide regulations and guidelines for control room managers to safely operate a pipeline. PHMSA’s pipeline safety regulations prescribe safety requirements for controllers, control rooms, and SCADA systems used to remotely monitor and control pipeline operations.
    • Access PHMSA FAQs.
• Alarm management is the process of managing the alarming system in a pipeline operation by documenting the alarm rationalization process, assisting controller alarm response, and generating alarm reports that comply with the CRM Rule for control room management.
• High-Performance HMI extends the capabilities of SCADA in pipeline operations and complies with the ISA 101 requirement by providing an HMI philosophy, style guide, and design guide.
• API 1165 refers to the Recommended Practice for Pipeline SCADA Displays. This standard outlines the best practices for designing and implementing displays that are used by controllers to evaluate information available in all operating conditions.
• GRC (Governance Risk Management and Compliance) is a set of processes and procedures that help organizations achieve business objectives while acting with integrity.

Gamification: Full Episode Transcript

Russel Treat:  Welcome to the Pipeliners Podcast, episode 220, sponsored by EnerACT Energy Services, supporting pipeline operators to achieve Natural Compliance through plans, procedures, and tools implemented to automatically create and retain required records as work is performed. Find out more at EnerACTEnergyServices.com.

[background music]

Announcer:  The Pipeliners Podcast, where professionals, Bubba geeks, and industry insiders share their knowledge and experience about technology, projects, and pipeline operations. Now, your host, Russel Treat.

[background music]

Russel:  Thanks for listening to the Pipeliners Podcast. I appreciate you taking the time. To show the appreciation, we give away a customized YETI tumbler to one listener each episode. This week, our winner is Tanner Hunt with Marathon Pipe Line. Congratulations, Tanner. Your YETI is on its way. To learn how you can win this signature prize, stick around till the end of the episode.

This week, Clint Bodungen returns to talk about using gamification to conduct exercises.

Hey, Clint. Welcome back to the Pipeliners Podcast. It’s been a long time.

Clint Bodungen:  Thanks for having me. Good to be back.

Russel:  Haven’t had you since we last talked about Colonial. We don’t have any crises to talk about this time. We’ll talk about something fun.

Clint:  Stop using drinking words.

Russel:  [laughs] Drinking words?

Clint:  Yeah. Colonial is the new Stuxnet.

Russel:  Oh my gosh.

Clint:  At BEERISAC, or all the trade show conferences, when you’re with your pals, having a happy hour drink, and somebody mentions Stuxnet or Colonial, you’ve got to take a drink.

Russel:  I wish I had known.  I’ve missed many opportunities.

Clint:  Indeed.

Russel:  I asked you to come back and talk about gamification for conducting exercises. As a tee up to that conversation, why don’t you catch us up? What have you been doing with ThreatGEN? For those who haven’t heard before, what is ThreatGEN? How is it used?

Clint:  ThreatGEN originally comes from and, all the founders, we come from the industrial control systems space and have been doing services and consulting for decades, but no longer because now we focus on our product, which is ThreatGEN Red vs. Blue.

ThreatGEN Red vs. Blue is a gamified learning and simulation platform. Many people may know it as gamification. Ultimately, what it is that we use gaming technology to create cybersecurity simulations using what we call active adversary simulation technology, meaning that’s where the red and the blue, the red versus blue, the defenders versus the attackers.

Our philosophy is that when you’re doing whether it’s awareness training, whether you’re doing cybersecurity skills training, whether you’re doing team exercises, team training, like incident response training, tabletop exercises, those sort of things, most classical training, traditional training, is one-sided. As we know, cybersecurity is not one-sided. It’s not set it and forget it.

In order to get the most realistic real-world training, you need…Back in the old martial arts days, we would say, “You need to fight like you train and train like you fight.” I guess that’s the same in the military.

Instead of just doing your training or taking your CBTs and saying, “Okay, got it. I learned the knowledge,” you need to put it into practice against an active adversary, somebody working against you to subvert what you’re trying to do, just like you would in real life. That’s the long-winded version of my premise.

Russel:  We’ve talked about this before, although it’s been quite some time. How is that being accepted and adopted? What kind of feedback are you guys getting about this as an approach for cybersecurity training and cybersecurity readiness?

Clint:  [laughs] I guess my first answer would be “This started as a part-time hobby, and now we’re doing it full time as a revenue-generating company.” [laughs] I would have to say that that’s the best feedback, in terms of people are actually paying for this and buying it.

Ultimately, where it really stands out is in exercises. A lot of team-based exercises, tabletop exercises – I’ve been through a lot of them, and quite frankly, they’re dry. They’re based off of PowerPoint and Excel spreadsheets or sometimes paper. They’re a tabletop or a roundtable-style.

People are looking for something different, more effective, more engaging, more immersive, more interactive. That’s what gamification does. That’s what Red vs. Blue does. Instead of staring at PowerPoints and Excel spreadsheets, you’re staring at an actual interactive network environment, like an animated Visio diagram of something that is representative of your network.

Then, on top of that, like I said before, you have this active adversary component, to where somebody’s working against you. It’s just more engaging. It’s more interactive. People are really responding to that.

Russel:  I know when I looked at it, one of the things that was big for me is I’m like a lot of people in that I have a pretty good notional understanding of cybersecurity. I know what it is. I know the buzzwords. I know how to think about it, but I’ve never really thought about what is it to do cybersecurity in practice.

One of the things I found about ThreatGEN is I very quickly started getting sensitized to, “Oh, so this is what you’re actually doing,” because you are actually doing it in the game.

Clint:  That’s an excellent point. In fact, that’s what people tell us. When people come to us and tell us that “Hey, this really helps me illustrate and reinforce the concepts that I’m learning in other cybersecurity courses,” that became kind of one of our taglines. [laughs] It really does help you reinforce those concepts, but also it helps you see the big picture.

Like you just said, it helps people put it all together, see the big picture. A lot of people are visual learners. A lot of people are hands-on learners. What Red vs. Blue allows you to do is get a hands-on application for even strategic concepts.

How do you do hands-on practice for building a cybersecurity program or doing GRC and that kind of stuff? That’s what this interactive gamified simulation allows you to do it.

Russel:  When we talked, queueing this conversation up, one of the things you said that you had recently added that’s new is you’re starting to use this tool to do incident response exercises.

Clint:  That’s exactly right.

Russel:  Talk to me about how that works. That, to me, starts having applications well outside of cybersecurity and around some of the things we need to be there in pipelining.

Clint:  As you mentioned earlier, Red vs. Blue has you doing it. It has you doing the notion of cybersecurity and doing these things. Red vs. Blue, the entire simulation, is a start to finish thing, from building your cybersecurity program out through the Red team now attacking your environment, and then, as the Blue team, you respond. There’s a huge incident response component to that.

Before we even thought about using it as an IR tabletop exercise platform, we had people coming in to us, telling us that they were using this for IR tabletop exercises and wanted us to modify it and provide specific tabletop exercise functionality.

What we did was that we added your common scenarios so that you can actually point and click a scenario that you want to run against. It could be maybe ransomware in your IT, ransomware in your OT. Maybe a bad actor gets in your network and starts messing with PLCs and things like that.

We have pre-built scenarios, and we have pre-built environments. You essentially go into the game configuration. You click on, “Well, I want to work on a manufacturing plant.” You click on manufacturing plant.

Then you click on, “Well, I would like my team to respond to a ransomware incident that originates in the IT side of the enterprise network and then shuts down scheduling,” speaking of Colonial, “shuts down scheduling servers, and then figure out where they go from there.”

You click on that scenario. Then the scenario starts. Then you, as the Blue team, do what you do. The facilitator no longer has to focus on figuring out the scenario. They no longer have to figure out injects. They no longer have to run the facilitation.

They facilitate dialogue. They facilitate what’s important in the IR tabletop exercises, which is discussion and dialogue. The game facilitates all of that. You get the most important parts of the IR tabletop exercise, which is figuring out where you have deficiencies in your IR plan, figuring out where you have deficiencies in your communication, and all of that good stuff.

At the end of the day, you can record all this. The game will still give you your stats, your results, and everything like that, but then you as a team will fill out the endgame questionnaire and results, the observations.

Russel:  The observations, the lessons learned, all that kind of stuff.

Clint, to me, this is what’s really compelling about this as a concept or as an idea, is that if you’ve ever been a facilitator for any tabletop exercise, or anything like that, you always find you spend more time actually getting people to understand what they’re doing than you do facilitating learning and improving.

Having a game that’s self-explanatory really helps with not only helps with just the learning and the process for learning, but it helps with getting much better feedback because you’re actually interacting with things that are semi-real.

Clint:  What we like to say is the game takes the minutiae and the logistics. It takes it all out of your way. It lets you focus on what’s important.

Russel:  ThreatGEN is built on a platform, correct? It’s not an application.

Clint:  We built it from scratch from the ground up using a gaming engine, but it’s web delivered. You don’t install it. It’s not an application. It’s like a SaaS, but it’s subscription-based and all that. I wouldn’t say it’s true SaaS.

Russel:  It’s more like an online game.

Clint:  It’s a portal. You have an entire educational portal, which has documents. It has courses. It has supporting material. We divide it into two sections. There’s the gamification platform, the gamified learning platform, which is the game, and there’s the education portal, which is where the game is delivered through. It’s all online.

Russel:  You have a cool video on your homepage and all that kind of stuff.

I’m a pipeliner. All pipeliners have certain requirements where they have to exercise their emergency response plans and they have to do other kinds of things that are not unlike. The context and specifics are different, but the process is similar in terms of doing an emergency response plan or an emergency response exercise versus doing a cyber response exercise. How big a leap is that to make?

Clint:  First of all, let me say that while I’m not a pipeliner by trade, I would like to say that I was raised by pipeliners, because the very first scenario that we built in Red vs. Blue was a pipeline scenario. As you know, I’ve worked in the pipeline industry for really the majority of my clientele.

Someone very wise with a mustache from A&M once told me, it’s a requirements engine. If you have a requirements engine, if you have something that can measure and evaluate requirements, then it’s easy to adapt that. That was you, by the way, in case you’re wondering.

Russel:  I know who you meant, but I appreciate the referral for everybody else.

Clint:  There’s not a lot of difference. This is the interesting part of my job now. The fact that I went from being a cybersecurity professional to now also a game developer, I learned a lot of things about how game development and how cybersecurity application development work. The underlying engine, maybe it’s just good software engineering. I don’t know, I’m not a software engineer.

The underlying mechanics in the engine that make the thing go, that evaluate and make things happen, event-driven things, it’s all the same underneath. At the end of the day, it doesn’t matter if you’re kicking over a lemonade stand, hacking a server, or cracking a pipeline, whatever.

At the end of the day is when you have an event. That event has consequences and cascading events. There are mitigations for those events. What we’ve done with the backend of Red vs. Blue is we’ve made it so that we can change the requirements. We can change the events. We can change everything and it doesn’t require new code.

We can just go into our level builder on the backend, swap things out, change the names of things, change the order of things, and now you’ve got a new requirements evaluation game.

Russel:  I get that part. When one of your customers is using your tools to do a cyber incident response, are they pretty much able to do that themselves or do they bring one of your people in to help with the facilitation? How does that typically work?

Clint:  What we’ve learned in the early stages of running pilots and development was that we needed one of our people to help facilitate because we found that the game mechanics got in the way sometimes, especially people that aren’t familiar with games like this, gamers, and things like that.

We’ve made corrections. When we do our full launch in March, we’ve made those corrections to help make that easier. In our learning portal, we’ve also provided a video-based course that actually guides you through these exercises as well. There’s a virtual version of me guiding you through the exercise.

Ultimately, we’re trying to make it so that Clint or Clint’s colleagues don’t have to be there. We want this to be an IR tabletop exercise in a box.

Russel:  I want to flip the script here if I might. I want to tee up a conversation. You might want to ask me some questions about this because one of the things that a lot of pipeline operators are really struggling with right now is the whole topic of team training. Team trainings are relatively new requirements. About two years old, I guess, in the code, maybe a tad longer.

Team training is, “I’ve got to do training that supports and orients all the people who might reasonably be expected to communicate with or collaborate with the pipeline control center.” For a lot of operators, that’s a large number of people. For a lot of operators, those people are very highly geographically distributed.

We have done some of these trainings for some pipeline operators. We designed the training and helped them facilitate it. More helping them build capability to do it themselves. Not so much running it for them. One of the things that we found is there’s three parts to this. One part is there’s some soft skills training I’ve got to do around teamwork and communications and those kinds of things. There is the exercise itself. Then there’s the debrief.

Generally, PHMSA said you can’t do all this with CBT, which I think means I’ve got to have people interact and learn from the interaction. Doesn’t necessarily mean that I’ve got to have them all in the same physical room to interact.

I don’t know. That may be getting a little further than what PHMSA is saying or contemplating, but I think when PHMSA was writing the FAQs, they weren’t really thinking about games as a mechanism to facilitate training and collaboration and lessons learned development.

Clint:  The government very rarely leaves things up for interpretation, I understand that. That was a joke.

Russel:  [laughs]

Clint:  I think you would have to figure out what they mean by CBT, computer-based training, because by nomenclature, this would be computer-based training, but by concept, this is not computer-based training. This is interactive simulation training.

Also, that being said, while we do provide the corresponding e-learning course that does those things, if you want a facilitator, there is virtual Clint facilitating it, and there is virtual Clint helping with the debrief, at least guiding you on how to come up with your own debrief.

However, we’ve also designed it to where a facilitator, such as you or whomever would be facilitating this, can use this as a more interactive tool to help their own exercises. If PHMSA did have the requirement of “Sorry, you have to have a live person doing this,” then that live person would use this as a much more engaging alternative to PowerPoints and Excel spreadsheets or forbid the paper. [laughs]

Russel:  When I think of CBT, what I think of is a series of charts that I’m reading and then questions I’m answering, I’m thinking like your defensive driving thing that you have to do upon occasion.

Clint:  I think of CBT is, what most people think of CBT, which is you watch a silly video. You go through some little question and answer session or a quiz or something like that. That’s CBT.

Russel:  Where this is more like multiplayer gameplay. If you’re going to say, “Hey, Russel, here’s what it would take to build a team training framework that was supported with a gaming engine,” where would I need to start?

Clint:  Are you referring to if you wanted to do something outside of the built-in stuff that we already had the built-in scenarios and requirements we already had?

Russel:  Great question. Let me try to ask it a bit more specifically. The scenario I might be doing is we get a leak alarm on this pipeline segment, or I’ve got a phone call. Somebody is calling in from the field, and they’re saying I’m smelling hydrocarbons.

Or I have a first responder contact me and say, “Hey, somebody has broken into your facility at this location,” but real-world things that come into the control room from the field and would require interaction or collaboration with what I call dudes and trucks, not necessarily just guys, but the people in the trucks that are operating the pipeline and doing all the field response. That’s the kind of scenario that you would be building.

Clint:  Fundamentally, like I mentioned before, is that the functionality to build those sorts of things already exists. That’s already how the game works is that you may have something go down. You have a pipeline go down. The pipeline turns red. You get a notification. That tells you what’s going on. Whatever actions that you take the game responds.

By the way, it’s a turn-based game. It’s not a real-time game. Otherwise, you can’t simulate the passage of time very easily and things like that for things that take weeks. It’s a turn-based game.

Like you pointed out, the game, the simulation is geared towards cybersecurity. In order to take the context and put a new context, such as pipeline incidents, things like that, into the game mechanics, usually takes about two weeks to a month of interacting with the other, the third party, such as yourself, that would want to create new scenarios that are fundamentally mechanically different, just like fundamentally different from cybersecurity, but it’s still based on an incident happen.

We have to respond, the game responds to stimuli or to interact with the game usually takes about two weeks to a month of working with whatever third party wants to create new scenarios, new content, possibly, maybe a couple of weeks and that based on the fact that it’s not cybersecurity related. It’s still event-driven.

I would say anywhere from two weeks to six weeks of collaborative development to create an entirely new scenario of different contexts like pipeline incidents.

Russel:  Then the follow-up question would be, once you had that in a generic way, how much effort would it be to tailor to specific pipeline operator differences?

Clint:  That’s what I’m talking about. Creating the new scenario and the new context, when I say that, completely outside the norm of cybersecurity and moving specifically into pipeline operator context.

Russel:  There’s some things that would be normal. A leak alarm in the control room, a phone call into the control room – those would be normal across multiple operators. Each operator might want to try and make that a little bit more customized need for their particular operation.

Clint:  It stands to reason that once you have a core framework in place with a specific context and scenarios, then it would always take a couple of weeks to add in a few more different contextual notifications or actions or things like that. Once the framework is in place, then it’s always going to take a little bit of time to add more customization, but not nearly as much as the initial kind of configuration as a configuration because it’s really not any new code. It’s just modifying…

Russel:  It’s building scenarios. It’s not writing code to enable scenarios.

Clint:  For the gamers out there, we literally just have a level builder internally that we’re literally just building a new level.

Russel:  It’s like Minecraft.

Clint:  It really is [laughs] except it’s prettier than Minecraft, I think.

Russel:  Of course, you do. You’re the guy’s building. Of course, you think it’s prettier.

Anyways, this conversation has been a little bit unique from what I’ve often done, but I just think that what you’re doing with ThreatGEN and what some of the needs are in the pipeline space, it’s a pretty compelling potential connection.

If somebody wants to ask me or wants to ask Clint questions, just go to the Pipeline Podcast Network site. Fill out the contact form. We’ll set something up to have a conversation if you find this conversation intriguing. You’d like to know more.

Clint, what would be some of those things you think might make sense for the use of this platform as it relates to pipeline operators?

Clint:  You’ve covered it in terms of the typical pipeline incident, control room operator, context, notifications, response, things like that. I think that there’s just a few things that carry over, regardless of whether you’re in pipeline or not, but, obviously, overlap with any industry vertical, which would be there’s a component of physical security IQ and just general cyber IQ.

By the way, I am calling this cyber IQ as opposed to cybersecurity awareness or security awareness. You’ll know why when my article drops this week.

Anyway, just general, I think one of the things that we try to do by leveraging this interactive cybersecurity simulation platform is what it is really, is we try to get people involved, like you said before, to see it, to do it, to understand the big picture, as opposed to going through CBT is like a zombie is, “Okay, I’ve done this.” Click, click, click.

We want to arm people with actionable knowledge, not just check the boxes. That’s physical security, cybersecurity, incident response, pipeline or cyber. Those are the big things.

There’s also an element that we’re working on, which we’ll have in a couple months, which is GRC, which in GRC, any type of regulatory compliance, that doesn’t have to be just cybersecurity. You’ve got API 1165, which is not cybersecurity. I believe that’s CRM, right?

Russel:  Yeah. More specifically 1165 is HMI but a CRM program, yes.

Clint:  We’ve got a program that we’re putting in place to where everything that you do in the simulation as you go through it. It’s checking off regulatory requirements and saying, “Okay, you’ve done that or you’ve complied with that.”

At the end of the simulation, it tells you, these are the things that you’ve done. While you’re doing all of this, here’s where you’ve complied with the following requirements of the following standards, and here’s where you missed it. GRC is a big thing that we’re we’ll be working on there.

Russel:  Give me the decode for that acronym, GRC.

Clint:  Governance risk management and compliance. It doesn’t necessarily GRC may not pertain to pipeline, per se, but regulatory compliance, definitely not.

Russel:  I can see that. I was going more for the things that you’re required to do as part of a pipeline safety program by PHMSA, your incident responses, your emergency response drills, those kinds of things that you’re required to do from a regulatory perspective as a mechanism for doing them and doing them more effectively.

Clint:  Like you said, it all goes back to requirements management and having a requirements management engine. You and I had a conversation a long time ago. We were driving back from San Antonio, I believe it was.

Russel:  My gosh.

Clint:  We were driving back and we were in your truck and we’re sitting there talking about having a requirements, evaluation engine, and that kind of stuff. I’m sorry, I’m not going to give any royalties.

Basically, I took that concept. In all fairness, you built a requirements evaluation engine to do so. I basically took that concept, and instead of point and click, I put it in my world. I grew up as a gamer, and I’m a gaming geek.

I said being able to do this, but in a simulation-type manner, is going to be much more interactive, much more interesting. At the end of the day, the underlying engine is a requirements evaluation and feedback mechanism.

Russel:  I think you said a mouthful there, but the bottom line is, if you make the training more interactive, more interesting, more fun, then people are going to be more willing to do it. I think the other thing because it’s all web-based, and it can be multiplayer, while you don’t have to travel to attend, you just figure out how to manage those logistics and you can still have the same impact.

Clint:  Nowadays, when we’re forced to work remotely, a lot of the times – I think that because of the changing times and the circumstances – I think working remotely and working in geographically dispersed locations is going to become more of the norm. So, yes, having something that is more accessible via the web is definitely an advantage.

Russel:  I absolutely agree. We will certainly be doing more of this. Anyways, I hope that all the listeners found this interesting. This is my opportunity to talk to Clint and geek out. I hope the listeners enjoyed listening to this because these are the kinds of conversations that I enjoy.

Clint, great to catch up. Thanks for coming on. We need to do it more often.

Clint:  Like you said, when I come on the show, that’s what we do is geek out. I hope everybody enjoyed it. Russel, thanks for having me back on. Great conversation. This is a topic that’s near and dear to my heart, obviously, and made a career of it.

Russel:  Clint, take care. It’s good to talk to you.

Clint:  Likewise. Thank you. Take care.

Russel:  I hope you enjoyed this week’s episode of the Pipeliners Podcast our conversation with Clint. Just a reminder before you go. You should register to win our customized Pipeliners Podcast YETI tumbler. Simply visit pipelinepodcastnetwork.com/win and enter yourself in the drawing.

If you’d like to support the podcast, the best way to do that is to leave us a review. You can do that on Apple Podcast, Google Play, Stitcher. You can find instructions at PipelinePodcastNetwork.com.

[background music]

Russel:  If you have ideas, questions, or topics you’d be interested in, please let me know either on the Contact Us page or reach out to me on LinkedIn. Thanks for listening. I’ll talk to you next week.

[music]

Transcription by CastingWords