In this edition of the Pipeliners Podcast, we are airing an interview with host Russel Treat that originally aired on the Crux OCM Podcast. The podcast is hosted by Vicki Knott and Rebecca Greenan of Crux OCM.

In this episode, you will learn about industrial automation, how the pipeline control room has advanced through automation (e.g. SCADA systems), the future of Control Room Management, and more topics.

Crux OCM Podcast: Video Interview

Watch this interview with Russel Treat and Vicki Knott from the 2022 API Pipeline & Cybernetics Conference in May 2022.

Crux OCM Podcast: Show Notes, Links, and Insider Terms

• Vicki Knott is the CEO and co-founder of Crux OCM. Connect with Vicki on LinkedIn.
• Rebecca Greenan is the SVP of Finance and Operations for Crux OCM. Connect with Rebecca on LinkedIn.
  • Crux OCM enables the autonomous control room of tomorrow, operating within the safety constraints of today. Combining advanced physics-based methodologies with machine learning, CRUX software helps clients increase throughput production and energy efficiency (up to 10%), improve safety, and ensure operators stay safe while contributing to a seamless, continuous operation. Find out more about their technology at CruxOCM.com.
  • Getting to the Crux of It Podcast discusses Robotic Industrial Process Automation and everything about it, featuring guests from the energy industry. Listen to the podcast here. 
• PHMSA (Pipeline and Hazardous Materials Safety Administration) is responsible for providing pipeline safety oversight through regulatory rule-making, NTSB recommendations, and other important functions to protect people and the environment through the safe transportation of energy and other hazardous materials.
• NTSB (National Transportation Safety Board) is an independent Federal agency charged by Congress with investigating every civil aviation accident in the United States and significant accidents in other modes of transportation – railroad, highway, marine, and pipeline.
• HMI (Human Machine Interface) is the user interface that connects an operator to the controller in pipeline operations.
  • High-Performance HMI is an advanced HMI system that improves a pipeline operator’s situational awareness during normal and abnormal situations. High-Performance HMI takes available data and presents it as information that is helpful to the controller to understand the present and future activity in the pipelines. (Listen to Pipeliners Podcast Episode #5 covering the topic of High-Performance HMI.)
• PLCs (Programmable Logic Controllers) are programmable devices placed in the field that take action when certain conditions are met in a pipeline program.
• SCADA (Supervisory Control and Data Acquisition) is a system of software and technology that allows pipeliners to control processes locally or at remote locations. SCADA breaks down into two key functions: supervisory control and data acquisition. Included is managing the field, communication, and control room technology components that send and receive valuable data, allowing users to respond to the data.
• GIS (Geographic Information System) is a system designed to capture, store, manipulate, analyze, manage, and present spatial or geographic data.
• ICS (Industrial Control Systems) encompass the control systems and instrumentation used for industrial automation and process control. These systems are used in oil & gas and other key industries.
• ESG (Environmental, Social, and Governance) refers to the sustainability movement in oil and gas to continue operating safely, in compliance, and in a responsible manner to do no harm while achieving business objectives.

Crux OCM Podcast: Full Episode Transcript

Russel Treat:  Welcome to the Pipeliners Podcast, episode 234, sponsored by the American Petroleum Institute, driving safety, environmental protection, and sustainability across the natural gas and oil industry through world class standards and safety programs. Since its formation as a standard setting organization in 1919, API has developed more than 700 standards to enhance industry operations worldwide. Find out more about API at API.org.

[background music]

Announcer:  The Pipeliners Podcast, where professionals, bubba geeks, and industry insiders share their knowledge and experience about technology projects and pipeline operations. Now, your host, Russel Treat.

Russel:  Thanks for listening to the Pipeliners Podcast. I appreciate you taking the time. To show that appreciation, we give away a customized YETI Tumbler to one listener every episode. This week our winner is Ian Ellington with Hilcorp. Congratulations, Ian. Your YETI is on its way.

To learn how you can win this prize. Stick around until the end of the episode. This week, I am sharing my appearance on the Crux OCM Podcast, where I appeared as a guest to talk with Vicki Knott and Rebecca Greenan about my experience in the control room.

Vicki Knott:  Welcome to Getting to the Crux of It Podcast, where we discuss robotic industrial process automation and everything about it. Each episode, we have candid conversations with our guests who are industry leaders in their areas and bring enticing stories and insights. Thank you for joining us today. I’m your host Vicki Knott, and I’m joined by my cohost Rebecca Greenan.

Rebecca Greenan:  Thanks, Vicki. On today’s episode, we are talking about matching systems to human capabilities to control room management. Today’s guest is Russel Treat, CEO of EnerACT Energy Services. Russel is an industry leader, software entrepreneur, and trusted subject matter expert specializing in oil and gas pipeline operations, leak detection, and automation.

He’s also a veteran podcaster and is the creator and host of the Pipeliners Podcast. Welcome, Russel. Thank you for taking some time today to join us.

Russel:  It is so good to be on a podcast as a guest.  I’m just so excited that I don’t have to think so hard. I can just chat about. It’ll be great. I’m looking forward to this.

Rebecca:  Russel, when we talked before, essentially, you’re an engineer that’s been doing business process automation for 30 years. You’ve been a disruptor, an entrepreneur, a technical software builder, and that’s the high level, that’s what I walked away with. Let’s do a bit of a deeper dive. Can you tell us a little bit more about yourself and about EnerACT?

Russel:  I’m an engineer by education. I’m a civil structural engineer. 

Vicki:  We don’t hold it against you.

Russel:  [laughs] I spent the first part of my career in the Air Force. I thought I was going to build steel structures, and I ended up building concrete runways and parking pads for aircraft and a little bit of steel structures. I did that. I got out of the Air Force.

I worked in cryogenics for a while, which is liquid air products, liquid oxygen, liquid nitrogen, liquid CO2, that sort of thing. The different applications for those products. Then, I left that company and started a company very quickly. I was a partner in a group called Software Marketing. What we did is we would find devices, typically things built by engineers, and we’d commercialize them. You find a piece of software, and you figure out what are all the things you need to do around that software to have it deliver value to the customers, the training program, the marketing and sales program, the product roadmap, the technology plan, all of the, “How do we deliver it? How do you support it?” All those kinds of things. Did that for a bunch of years, found out I liked oil and gas, and then I started with an oil and gas software company that did back-office measurement accounting. The bridge between all the field metering and into all the accounting systems.

I did that for a bunch of years and then left and started a company called EnerSys, which is one of the operating companies within EnerACT, and did measurement, consulting, and then telecoms and then some HMI, and then some control rooms, and then some leak detection.

Then, in 2007, the control room management rule started making its way out of PHMSA, and I got all excited about the business again because it started becoming about operation effectiveness rather than just being a technical geek. That’s my story.

Vicki:  Very cool. We’d love to just dive into some questions then. Something that comes up a lot in the good old pipeline world, but lots of folks probably haven’t heard of it, so we’d love to dig in a bit. What is control room management?

Russel:  There’s two parts to the answer. I want to give the regulatory answer first and keep that brief. Control room management is an area of the pipeline safety code that sets a bunch of requirements on pipeline operators and their business practices inside the control room. When most people hear control room management, they’re thinking about regulatory policy and things I have to do.

What control room management is, if you peel back the regulatory layer and you look at the pipeline incidents that led to the creation of the rule and you read the NTSB reports, what control room management is about is making sure that the systems that are in the control room and used to operate the pipeline match the human beings and their capabilities.

One of the best examples of that is prior to the implementation of control room management, most pipeline control rooms existed in a perpetual alarm flood. An alarm is something that comes in and it flashes at you and maybe makes some noise and it says, “Hey, you need to pay attention to me.”

That’s effective if you get a handful of those over the course of a shift and completely ineffective, you get one every minute or two because you just ignore them all. When we first started doing control room management and working with control rooms, it was not uncommon for a control room to have 25 alarms an hour. An alarm flood would probably be more than two in an hour. Two in 10 minutes or six in an hour is the number that people typically use, but then more than that’s too many. If you think about it, if I get 25 an hour, that’s one every 30 seconds. That’s barely enough time to find it, acknowledge it, and move on.

Vicki:  I can imagine people are getting that many notifications on their phones. They go nuts. That’s a big job. [laughs]

Russel:  Yeah, right. Exactly. That’s just one aspect. One of the other aspects is called adequate information. That has to do with having the controllers understand the system and a lot about high-performance HMI and how you build screens in the control room. Again, historically, pipeline screens were basically black. They had some colored lines and some numbers on them. Now, we’re doing all kinds of advanced graphics and animations and putting together things that tell me, “Am I normal or not, or am I abnormal? If I’m abnormal, how abnormal am I?” Which is much different than knowing what the numbers are. We used to require the controllers to have all those numbers in their head.

Controllers do that when they’re operating a pipeline. They build this mental framework in their mind, but that mental framework is generally built based on the system they’re given to operate, so it’s constrained by and limited by the system. Control room management is all about helping controllers become more effective in their real job. Their real job, very simply, is deliver products on schedule without incident.

Vicki:  It’s much more complex than the average folk tend to know. We love digging in and giving some insights to it.

Russel:  Everything is easy until you know enough about it.

Vicki:  I know, right?

[laughter]

Rebecca:  Russel, I just want to dig in a little bit more. Alarm management still seems to be the solution for all the abnormalities that are happening in the control room. I’m wondering if we can dig into that a little bit more and your thoughts on the evolution of the control room.

Is alarm management the solution, or are there other ones that you’ve been exposed to that would help out on that?

Russel:  Rebecca, that’s a good question. When I talk about a high-performance control room and what’s necessary and what is the perfect human machine interface and what’s the perfect alarm management? Those are two different sides of the same coin. They’re just different views. What we have done as we’ve begun to implement alarm management, and we began to use alarming as the way to tell controllers there’s something abnormal going on. That’s getting better, but we’ve got a long way to go, in my opinion.

I’ll tell you about what I tell people the perfect system is. The perfect HMI shows you you’re heading to abnormal before you get to an alarm. The perfect alarm is one and only one alarm for each abnormal condition, and it only comes in for that abnormal condition. Neither of those exists in practice.

Vicki:  I’ve never seen that. [laughs]

Russel:  No. They neither exist in practice, but it’s important. We tend to think in our world of control room and pipeline operations, we tend to think in specification. Like, “This is the standard, this is what I got to do.” Operational excellence is about aspirational goals, so zero incidents. Everybody agrees that that’s the right goal. That’s an aspirational goal.

We may never ever get there. Now, we might be able to have zero incidents for an hour or for a day or for a week or so forth, but ultimately, we’re never going to get to zero incidence, but that’s the aspirational goal. What I laid out with the HMI and the alarm system is that’s the aspirational goal. We’re not even to the place yet as an industry where we’re recognizing that as the aspirational goal.

Vicki:  We need to get there for sure. With that as the aspirational goal, in knowing us and the automation work we’re up to, how do you think automation plays into the evolution of the control room and specifically getting to that zero incident aspirational goal?

Russel:  Certainly, automation is critical. There’s some problems with the language we use in the control room. When I say alarm, for everybody listening to this, that word means something very specific to them, and they can visualize it in their mind. Probably most of them have a different idea of what that is.

Vicki:  Different ideas of what alarms are?

Russel:  Yeah. If you talk to somebody in the field that’s running a compressor station, their idea of what an alarm is very different from somebody in a control room when you ask them what an alarm is.

Vicki:  Correct. Yeah, and probably that with automation as well.

Russel:  Exactly. That’s the point I was trying to make is that automation is a word we use, but it can mean a lot of different things to a lot of different people. Most of us are familiar with the automation we do in the PLCs and in the flow computers and all that sort of thing.

The idea of automation and the SCADA system, we’ve avoided that. There’s a lot of focus and intentionality on putting the automation in the field near the process we’re automating for failsafe and redundancy and safety and all that. That’s very distinct from process automation in the control room. What you guys are doing is process automation in the control room. It’s a bit evangelical. There’s a whole lot of convincing people that’s even a real thing. [laughs]

Rebecca:  Glad we got you on the podcast, Russel. Help us convince the masses. [laughs]

Russel:  The masses. [laughs] Automation is going to be critical. We have to think of automation in terms of user experience. Alarm management is about doing alarming focused on user experience. High-performance HMI is about building a human machine interface focused on user experience. What you guys are doing in process automation is focused on user experience. I want to take the mundane, repetitive work off the controller and allow them to do the more important analytical work.

Vicki:  To date, most of the more advanced automation technologies are primarily only focused on equipment, and that’s where the industry has been failing. It’s like, “Hey, let’s optimize the absolute crap out of this pipeline or this pump from the equipment standpoint, and then hand that over to the control room operators, and they’re going to write procedures and figure it out.”

That’s not the approach that is the most effective. A positive user experience leads to a safe experience and then, therefore, efficiencies and all that great stuff. We’ve learned that in so many other sectors.

Russel:  The challenge is automation in the pipeline space is very cross disciplinary. You’ve got your PLC automation in the field that’s automating the remote processes. You’ve got your telecommunications, that infrastructure you’ve got to build, and you’ve got your SCADA systems and your HMIs.

Then, if you’re doing leak detection, you’ve got all that stuff that you’re doing to feed all that data there. All those disciplines have their own piece of the automation puzzle, and we don’t typically engineer those things as systems well.

Some people do. There are examples out there of that, but a lot of times it’s, “Well, we have these pieces, and we just plug them together, and we don’t think about them holistically.” We need to get to the point we’re thinking about this stuff holistically.

Rebecca:  You touched on a little bit, Russel, but just like the human factors, and we’re trying to connect why the human factors are so important, and maybe just touch on the synergies between the control room automation and control room management.

Russel:  They’re both the same thing. The control room management is establishing best practices and regulatory requirements around things like how you should be managing your alarms, how you should be managing your HMIs, how you should be managing workload. One of the aspects of workload, if you do workload analysis, is the complexity of the task is as important, if not more important than the quantity of the tasks because that goes to fatigue. There’s that aspect.

Automation is a mechanism to make humans more effective. To do that requires some deliberate design understanding what humans do well and what automation does well. Automation does the repetitive task well. Automation takes data and turns it into actionable information and does that well. What it doesn’t do is see things like weak signals. Weak signals will be those things that are subtle indications that something’s going wrong but not a strong indication that something’s going wrong. A lot of the more complex things we do particularly like leak alarm response is an exercise in working your way through weak signals.

Vicki:  That’s a great way to describe leak detection response. Yeah, I’ve had that experience exactly.

Russel:  I was just having a conversation at the API Conference yesterday with somebody about what we need in leak detection is not better mechanisms for leak detection. We need better mechanisms to take these weak signals and turn them into strong signals.

Vicki:  Yes, that’s an excellent way to frame it. I completely agree because when you’re sitting there, and you see the leak alarm, you’re just like, “OK, great. It’s a leak alarm, but there’s so many other things that are going into it.”

The leak alarm cannot actually be correct. It’s a warning, and so then you’re looking at everything and you’re synthesizing it, and a lot of times it’s nothing. It’s just a transient hydraulic situation.

Russel:  That’s right. Then, what happens is you start to train yourself in this mental model that a leak alarm, anytime there’s a transient hydraulic, is not a real alarm. When do most leaks actually occur?

Vicki:  I don’t know that data.

Russel:  During a transient.

Vicki:  End of the shift, but yeah during a transient.

Russel:  Often, they occur during a startup or a shutdown or a slackline or something like that. Not only do those things mask your ability to see a leak. This is a good example in terms of where process automation, HMI, and alarm management can come together to make a job easier to do.

The more I can put information into the console that’s processed, that gives me indications of not only do I have a leak alarm, but here’s all the things that are happening that are contributing to it so that I can look and see the pattern versus having to analyze the numbers. That makes a huge difference.

If you see I’ve got a spike in my imbalance in a leak system, but it’s coming back down. That’s very different from when it came up and it’s continuing to go up. That’s hard to see in numbers. It’s easy to see if you build the right graphics, and that’s what high-performance HMI is all about.

Vicki:  You’re already touching into your next question here, which is the integration of these systems being so important and vital for safety and efficiency. These things need to be able to tie together in an easy way, something that’s visual because when you’re in that hot seat, there’s so much going on.

Like you said, for folks not having to just analyze the start numbers and then when they’re tired analyzing certain numbers becomes very difficult. Do you have anything you want to add to that with respect to the integration of these systems and its importance?

Russel:  It’s a interesting question because there are certain situations where you don’t want to integrate. Things like maps, GIS, logbooks, procedures and policies, and all of that. You don’t want any of that type of thing in your automation system.

On the other hand, in terms of the presentation, I want all of the alarming and all of the operational data and all of my analytical tools to resolve alarms and all of that. I want alarming and HMI and process automation and all the information about the automation in the field and all the information that I can about what’s happening with that. I want all that in my SCADA system.

Vicki:  The stuff that you say you shouldn’t be integrated, for our listeners, could you tell them a bit more why you think that they should not be integrated, your logs and your GIS?

Russel:  This starts to get technical, but I’ll use the map example a lot. I’ll use the map example as a starting point. Anybody who’s traveled in a city with a subway system, they have this map, but the map is not a geographic reference. It’s more of a relationship reference. This station is here in relation to this other station.

In a control room, I don’t care if the product is running north or south or east or west. I don’t care if these two sites are 15 miles away versus 100 miles away from each other, so much as I know where they are in relation to one another. Putting mapping in provides a whole bunch of data and context that can confuse the issue.

Vicki:  It’s not important, right?

Russel:  Right. What I’m doing is giving you a whole bunch of interesting stuff to look at and think about, but it’s not important to the job I’m here to do.

Vicki:  You’ve just been distracted.

Russel:  Exactly. “Oh, look at this little cool thing I can do with the map, which has nothing to do with what I’m here to do.” Now, on the other hand, if I need the map, give them access to the GIS.

Vicki:  Pull it up on a separate screen or whatever you need.

Russel:  There’s all kinds of cybersecurity issues around the things that are in the business network versus what’s in the ICS, the Industrial Control System. That’s one example. There’s a lot of others like that where there’s information that I’d like to have, but I don’t need it in my SCADA system.

The information I need to do the core job is a subset. Most of what’s in this SCADA system, particularly if they’re very elaborate, most of that’s there for diagnostics. It’s not there for operations. Those are two very different kinds of tasks. When you’re building an HMI, you’re trying to build the screens for the tasks.

A good example in aviation is the cockpit. I have a set of signals that I used to know where the airplane is, and that is consistent regardless of what airplane I’m in, Piper Cub, the fighter jet, the airliner, or the space shuttle. They all have that same basic instrument display to know where the aircraft is. They have a whole bunch of other stuff related to operating and flying the aircraft, and that’s all unique to the aircraft. What I’m trying to do is I’m trying to paint a picture of this whole issue of what am I integrating and how am I integrating it, and how am I presenting it?

You want to normalize the things that matter, but provide the detailed context that’s specific, and then you want to segregate that information in a way that’s meaningful.

Rebecca:  I like that. It’s a good way to think about what truly is important. Just pivoting a little bit from that, but there is a level of concern and reality around industrial assets and safety, and that brings ESG into our value conversation all the time. In your opinion, how important do you think it is that we look at ESG solutions from a control room perspective?

Russel:  I haven’t given a lot of thought to ESG. If we’re going to change the mission of the control room, and the mission of the control room is going to include limiting emissions, then that’s a different conversation. We don’t have cathartic protection coming in at the control room. We don’t have integrity management information coming into the control room.

Do we need to bring ESG information into the control room? The answer to that is probably yes, but this gets into safety management systems and safety management systems that are cutting across disciplines because integrity management cares about the number of pressure cycles on the pipe.

Control room and normal operations don’t care about the number of pressure cycles in the pipe. One of the things we can do with this industrial automation that you guys are doing, is we can lower pressure cycles while being able to be more efficient and effective from an operating standpoint.

That’s where the real value proposition comes in. Will we bring ESG into the control room? I don’t know. Is there an opportunity to understand that how we operate affects our ESG goals? Yes. The question is going to be how do we actually do that? Right now, even in things that matter to us, we don’t do that very well.

Vicki:  You’re correct. 

Rebecca: I was just going to say yeah. Just figuring out what those metrics are, and it’s almost like the control room is the data point, and putting those data points out to the people that are looking at the metrics will be important.

Russel:  That gets into a whole other conversation. SCADA stands for Supervisory Control and Data Acquisition. One of the things that I tell people is I think SCADA is dead.

We’re going to continue to have SC, Supervisory Control, and we’re going to continue to have DA, Data Acquisition. They’re going to split because there’s a whole lot of other people other than just the control room that need the data to do things with it that are meaningful.

That is a whole rethink of our OT infrastructure and how it ought to be built, which goes to the question you’re teeing up about: How do I do this? Where is this headed?

One of the places it’s headed is that these things that we’re talking about, about process automation and high-performance HMI  and alarming, which has historically been SCADA conversations, they’re going to begin to become operations affecting these conversations. The technology that we use to do them is going to be less important than what we’re able to do with the technology.

Vicki:  Being goal focused.

Russel:  Exactly. That will not be an easy transition to make.

Vicki:  No, that one folks struggle with is we get so many amazingly super smart engineers in the industry, and super smart engineers love technology, so it’s a hard shift.

Russel:  If you spent your life being a carpenter and somebody brings you a brick, you try to put a hammer to it.

Rebecca:  Absolutely.

[laughter]

Russel:  I’ll leave this conversation right there.

[laughter]

Rebecca:  Yes, that is perfect. Russel, can you give us your socials and where folks can find you?

Russel:  Yes, I can. You can find me on LinkedIn. It’s Russel Treat on LinkedIn. That is the best way to find me. You can also go to eneractenergyservices.com, or if you like, and I recommend you do, go to the Pipeline Podcast Network, go to the contact us page, drop me a note there. Lots of ways to find me.

Vicki:  Check out the Pipeliners Podcast because it’s great.

Russel:  Thank you. I was looking for that plug before we got off of here.

Vicki:  No worries. I got you. [laughs] Amazing. Everyone, thank you for tuning in to the Crux Podcast, and please do subscribe to the show. Sign up for the newsletter or give us a follow on social media: YouTube, LinkedIn, and Twitter. We’re all over the place.

Russel:  I hope you enjoyed this week’s episode of the Pipeline Podcast and our conversation with Vicki and Rebecca. Just a reminder before you go, you should register to win our customized Pipeliners Podcast YETI Tumbler. Simply visit pipelinerspodcastnetwork.com/win and enter yourself in the drawing.

If you’d like to support the podcast, please leave us a review on Apple Podcast, Google Play, or whatever smart device you happen to use to listen. You can find instructions at pipelinerspodcastnetwork.com.

[background music]

Russel:  If you have ideas, questions, or topics you’d be interested in, please let me know either on the Contact Us page at Pipeliners Podcast Network, or you can reach out to me on LinkedIn. Thanks for listening. I’ll talk to you next week.

Transcription by CastingWords