In this episode of the Pipeline Technology Podcast, the tables are turned as Jim Watkins, director of podcasts at Gulf Energy Information, interviews Russel Treat, the regular host, about his recent article in the January 2024 edition of Pipeline & Gas Journal titled “Optimizing Emergency Response Through Alarm Management.” 

They discuss the challenges and importance of effective alarm management in pipeline operations, emphasizing the need for clear prioritization of alarms to ensure swift and appropriate responses to potential incidents.

The conversation also delves into the intersection of alarm management with emergency response planning, highlighting the potential benefits of integrating these processes more seamlessly within pipeline operations.

Optimizing Emergency Response through Alarm Management Show Notes, Links and Insider Terms

• Jim Watkins is the Director of Podcasts at Gulf Energy Information. Connect with Jim on LinkedIn.
  • Gulf Energy Information—a leading provider of media, marketing and market intelligence services—provides in depth insights, technical content and strategic direction to the international energy industry. Gulf’s market-leading brands—Petroleum Economist, World Oil, Pipeline & Gas Journal, Hydrocarbon Processing, Gas Processing & LNG, Carbon Economist, Hydrogen Economist, H2Tech and Underground Infrastructure—serve their markets with digital media and events that leverage large audiences. Gulf also provides market intelligence solutions to the international energy industry through the Global Energy Infrastructure Database.  The company produces 13 conferences and technical events for the global energy industry.
• Pipeline & Gas Journal is the essential resource for technology, industry information, and analytical trends in the midstream oil and gas industry.  For more information on how to become a subscriber, visit pgjonline.com/subscribe.
• Read Optimizing Emergency Response through Alarm Management
• The CRM Rule (Control Room Management Rule as defined by 49 CFR Parts 192 and 195) introduced by PHMSA provides regulations and guidelines for control room managers to safely operate a pipeline. PHMSA’s pipeline safety regulations prescribe safety requirements for controllers, control rooms, and SCADA systems used to remotely monitor and control pipeline operations.
  • Control Room Management is regulated by PHMSA under 49 CFR Parts 192 and 195 for the transport of gas and hazardous liquid pipelines, respectively. PHMSA’s pipeline safety regulations prescribe safety requirements for controllers, control rooms, and SCADA systems used to remotely monitor and control pipeline operations.
• Alarm management is the process of managing the alarming system in a pipeline operation by documenting the alarm rationalization process, assisting controller alarm response, and generating alarm reports that comply with the CRM Rule for control room management.
• AOC (Abnormal Operating Condition) is defined by the 49 CFR Subpart 195.503 and 192.803 as a condition identified by a pipeline operator that may indicate a malfunction of a component or deviation from normal operations that may indicate a condition exceeding design limits or result in a hazard(s) to persons, property, or the environment.
• ISA-18.2 defines an alarm as “an audible and/or visible means of indicating to the operator an equipment malfunction, process deviation, or abnormal condition requiring a response.
• Alarm rationalization is a component of the Alarm Management process of analyzing configured alarms to determine causes and consequences so that alarm priorities can be determined to adhere to API 1167. Additionally, this information is documented and made available to the controller to improve responses to uncommon alarm conditions.
  • API 1167 provides operators with recommended industry practices in the development, implementation, and maintenance of an Alarm Management program. The implementation of API 1167 is required by reference in the CRM Rule.
• Alarm Response: Specified actions for pipeline controllers when alarms occur.
• Emergency Response: Actions taken when an emergency is identified, aiming to mitigate consequences.
• Operating Condition: Pipeline system state categorized as normal, abnormal (AOC), or emergency.
• ESD (Emergency Shut Down Systems) are high-powered control systems designed to protect people, pipelines, and the environment in the event of a pipeline operating beyond set control limits.
• Situational Awareness: Understanding of current operating conditions and potential risks.
• HCA (High-Consequence Areas) are defined by PHMSA as a potential impact zone that contains 20 or more structures intended for human occupancy or an identified site. PHMSA identifies how pipeline operators must identify, prioritize, assess, evaluate, repair, and validate the integrity of gas transmission pipelines that could, in the event of a leak or failure, affect HCAs.
• Roles and Responsibilities: Designation of individuals with authority to direct actions or supersede controller authority.
• GIS Mapping: Identifying geographic information, such as proximity to valves or HCAs, for emergency planning.
• Alarm Analysis: Reviewing alarm responses and emergency situations to identify areas for improvement.
• Continual Improvement: Ongoing efforts to enhance alarm management and emergency response programs.

Optimizing Emergency Response through Alarm Management Full Episode Transcript

Announcer: The “Pipeline Technology Podcast,” brought to you by “Pipeline & Gas Journal,” the decision-making resource for pipeline and midstream professionals. Now, your host, Russel Treat.

Russel Treat: Welcome to the Pipeline Technology Podcast, “Episode 42.” On this month’s episode, we’re going to flip the script. Jim Watkins, director of Podcasts at Gulf Energy Information, is going to interview me. He’s subbing for me as host and I’m going to be the guest.

I’ll be talking about my recent article in the January 2024 edition of Pipeline & Gas Journal titled Optimizing Emergency Response Through Alarm Management.

Jim Watkins: Hey, Russel, welcome to the show. Normally, you’re the host, but this time, you’re the guest.

Russel: Yes, and it’s always a pleasure to be the guest. I find that easier to do than to be the host. Appreciate the opportunity, and I appreciate you coming on and filling in as the host this time, Jim. That’s good stuff.

Jim: I’m Jim Watkins, by the way. I’m the podcast director for Gulf Energy Information. I’ll give a crack at hosting, see if I can fill your shoes there, Russel. We’re going to be talking today about an article that you wrote for the magazine in January. Before we jump into that, a lot of people listen to this podcast.

They probably hear you all the time, but I don’t know if they know about your background. I always like to get a little background on guests of shows that I’m talking on. Tell us a little bit about what you do, how you got to where you’re at, all of that.

Russel: Sure. It’s an interesting journey. I’m a civil engineer by education. I went to Texas A&M. I spent some time in the military doing engineering, the funnest part of which was blowing the things up that I built. I also learned that I didn’t want to be doing construction the rest of my life and I liked computers.

Got out of the military. I worked in cryogenics for a while. After that, I joined a company called Software Marketing, and this was back in the late ’80s before software was even a business. We would go out and identify software technology that we thought had real product commercial opportunity and we would commercialize it.

I did that with about 20 different products over about six years, really liked it. I had an opportunity to take a job running the US division of a Canadian company, running their software, the software part of the business, and doing back-office measurement accounting. That’s where I started.

From there, it led to doing SCADA stuff, doing the control room stuff, doing leak detection stuff. Now, I run several companies that do various things around control room management, and pipeline safety. Really, really liked the pipeline business. I found myself my home. Took me a while to get here, but I found my home.

Jim: That’s great. That’s why you’re a guest on your own show today, because this article falls right into your area of expertise. For everybody following along at home, this is Pipeline & Gas Journal, the January issue, Vol. 251, No. 1. The article is called Optimizing Emergency Room Response Through Alarm Management.

Let’s get into it, Russel. Tell us a little bit about why you even wrote this article. You have written articles for the magazine in the past, but why this article at this time?

Russel: Let me talk a little bit about the control room management. Control room management was a PHMSA rulemaking, came out of recommendations following Bellingham, because some of the things that contributed to the severity of the Bellingham incident were factors occurring in the control room.

That rule was published in 2010. It went fully into effect in 2012. One of the pieces of that rule is alarm management. Almost everybody has something that they’re doing. These control room management programs, they’ve been around over 10 years now.

They’re maturing in the pipeline operators, but everybody is struggling with getting efficiency and effectiveness in their alarm management programs. I say, everybody, I’m using air quotes, but that’s one of the more challenging aspects of the control room management rule is to get really good alarm management practice in place.

If people understand what the real opportunity is with alarm management, alarm management is kind of your operational insurance. It’s a way to think about it. It’s where I’m defining those things where I need to take action, or something bad is going to happen and I’m giving myself room to take that action before that bad thing happens.

That’s what alarm management is all about. That’s maybe easy to say, but it is not easy to do.

Jim: Just from a regular Joe’s standpoint, on my phone, I must get a thousand notifications a day, and maybe a handful of those are actionable. I try to keep that down to the things that I need to do because otherwise, you become desensitized. Alarms are going off all over the place and you’re just like, “Whatever, it’s another alarm. We don’t care.”

Russel: That’s called an alarm flood. That’s one of the things that was going on during the Bellingham incident. There were so many alarms coming in that the things that really mattered got missed.

The whole idea, and I talk about this–this is one of the tricks in alarm management–is everything that’s a notification in the SCADA system is called an alarm. I call that a little A versus a capital A alarm.

The capital A alarm is the one that’s more the API 1167 definition, which is an alarm indicating an abnormal operating condition and requiring action by the pipeline controller.

Russel: The key is part of that which requires action. When we’re doing it, we add a couple of things to that. It requires action in a time frame or something bad is going to happen.

Jim: That’s got to be the key. Ideally, those capital A alarms wouldn’t be that frequent, but would be very obvious to somebody who’s paying attention to what’s going on. That’s the whole rationalization idea, right?

Russel: Exactly. One of the things you do, if you come across a SCADA system that the alarms have never been rationalized, the first thing you’re going to do is say, “What of these are alarms, and what of these are alerts?”

The alarms are the ones that require action. The alerts are other things that are provided to the controller for information or analysis. They’re not requiring action. One of the first things you do is you make a very clear cut about those things and you present them differently.

Generally, that will reduce your alarm count by 80 percent. Pareto principle, 80-20. Right out of the gate, you can get an 80 percent reduction in things they got to look at. Once you’re there, it’s like, “OK.”

What I say, in a perfect rationalization, and a perfect rationalization doesn’t exist, it’s an aspirational thing, in a perfect rationalization, there is one and only one alarm for every abnormal operating condition. If you think about pipeline operations, a lot of our alarms are pressure-related, but pressure is not always the same thing.

Suction pressure to a pump is different than discharge pressure from a pump is different than line pressure going into the pipeline. They indicate different things operationally as an abnormal operating condition. Getting really clear about that, it can be tough.

Knowing I have a high pressure on the line is different from knowing that, “I have this high pressure because I have a PD pump and that PD pump is over-pressuring the line because it’s running too hard.”

That’s a different thing than, “I’ve got a low suction pressure because we have some kind of leak and we’re not getting good feed or we have a low tank level and we’re not getting good feed to the pump.” Knowing not just that I have a low pressure, but what’s the abnormal operating condition that’s driving that, that’s where you ideally would be when you’re doing alarming.

Jim: Now, I think, and I don’t know if this is covered in the article in detail, but the advent of sensors and being able to put sensors on anything, that has to help this situation. Now, it’s not just a sensor, at the pump or before the pump, a flow meter or something.

You can have it everywhere. You can have audio sensors. You can have all kinds of things going on.

Russel: The short answer is, yes, you’re right, and certainly, there’s a lot of opportunity as we move forward with automation to be more deliberate about how we’re doing alarming. The practice in the industry is they got an alarm rationalization process in place, but they’re not at that level of maturity yet.

The reality is rationalization is a very tedious, very painful process. It’s very similar to doing a process hazardous analysis. Very similar in terms of the work, and all the people we have to bring together, and the expertise you need in the room to do it well.

Jim: Costly as well?

Russel: Yes.

Short answer, yes. When you think about the resource cost of doing it. What a lot of pipeline operators have done is because they had to do something to get into compliance with the rule is they set some kind of general guidelines around how they did rationalization.

They said, “If we have a mainline pump, then this is the rationalization for our mainline pump,” and that’s done pretty consistently throughout their operation. That’s where we are. Now, people are starting to look and ask some questions about, is there a better, more effective, more efficient way to do this process?

Jim: Interesting. If it is that costly, and difficult, and time-consuming to get that done, it would seem like in this day of AI and all these wonderful things we have, that we could figure out a way to do that more accurately, better, using machine learning or something.

Russel: That’s certainly true, but the reality is there’s a level of understanding the operations that’s required before you can get to that level. Right?

Jim: Yeah.

Russel: From a maturity standpoint, we’re just not there. The other thing I would say, you asked me earlier, and I don’t think I fully answered the question, but why write the article?

If you think of alarming as an insurance program, and if you think about, “What an alarm is telling me is I have an abnormal operating condition that requires management, like I have a potential leak and I need to make sure I don’t have an actual leak,” that sort of thing.

That steps immediately into mitigation. If I actually have a leak, the quicker I can know that I have a leak, the quicker I can know the size of the leak, the quicker I can know the location of the leak, the more effective I can mitigate the leak.

Certainly, particularly with liquid pipelining, leak detection, and leak response, leak mitigation are very important. We do all these things in integrity management that are leak prevention, and then we do things in SCADA, which are leak detection.

If we’re going to look at this system holistically, the other thing we got to look at is, “How quickly and effectively can I mitigate a leak, should I have one?”

Jim: We see how important that is all the time. If you don’t get on top of a leak right away, that could be an environmental disaster. Maybe the leak’s in a place that’s not so easy to get to, like in the bottom of the ocean. You got to have a plan, right?

Russel: Exactly. To that point, the NTSB today published an article on the San Pedro Bay leak that happened off the coast of California off of Huntington Beach. That was related to an anchor drag, or at least it’s believed it was related to an anchor drag. I don’t know if that’s been fully settled.

I haven’t had a chance to read the report yet. There’s a lot of learning around what we’re talking about here. They had a situation where there was a lot going on in the platform when they were getting their leak alarms. What they were having is some operational upsets that masked the other problems.

If you have a leak, and you identify it quickly, and you respond quickly, it’s a much different process than if you have a leak, but you don’t identify it quickly or you don’t respond quickly. Quickly is different depending on who you are.

Jim: Exactly. That’s interesting, because if that’s the case in that report, that they had this problem of too many alarms going off all over the place, and so they couldn’t see the really important ones, like we were talking about earlier, it’s a rationalization problem.

They just didn’t have things set up properly to make them focus on the real problems, the big A alarms, right?

Russel: That’s right. Once you actually have an alarm, and you have confidence that that’s really an alarm, and you know you have a problem, then…The purpose of the article is to say, look, if we think about situational awareness is the ability to see the system, and alarm management is the ability to raise upsets to the top of my attention.

Then, emergency response is, when something actually goes wrong, I’ve connected the dots between the alarm and the adverse outcome. Now, I can use that same rationalization process to pre-plan my mitigation. There’s a lot of value, a lot of opportunity in that for operators, I believe.

That was what I was doing is I was going to…What’s interesting about this conversation, Jim, you were talking about the opportunity to use artificial intelligence, or other kinds of algorithms, or other kinds of sensors to get a better indication of what the process upset is.

There’s probably more opportunity for operators in the short term to take what they’re doing with alarm management and use it to inform their emergency response process.

Jim: Interesting. To make that link completely so that there’s not extra time between, “Hey, this could be a critical situation, and now we got somebody out there.” That varies on the size of the operator and those types of things. I imagine that they get out there pretty quick.

For example, not what we’re talking about here, but one time, I punched a hole in a gas line at a house that I owned and I could smell the gas. I shut everything off. I called, and those guys were there in two minutes. Literally, they were there immediately.

Russel: They’ll get there before the fire department sometimes.

Jim: Absolutely. They were definitely there before the fire department. I was shocked at how quickly they got there. I was like, “Wow, were you guys around the corner or something waiting for this to happen? I don’t know.” For pipelines, what’s the deal?

Russel: For the utility operators, they have mechanisms to take those things and move them right to the top. It’s the same kind of thing in a control room when you get a call off of a line marker. Those things go right to the top.

They get treated like a critical alarm. It’s interesting when you look at this. I want to come back to talk about this a little bit more. We said that this is costly, right?

Jim: Right.

Russel: It’s costly because of the time it requires to build the context. When I do a process hazard analysis, I have to put a team together, I’ve got to go through a structured process.

I’m walking through process diagrams, and cause-effect diagrams, and other things, and maybe even looking at the automation to understand how a process operates. Out of that, I determine, “What are the physical things I need to do so that if I have an upset, the system will shut down safely?”

Alarm management does all the same process building, but it asks a different question. It is, “What alarms do I need so that I don’t have the upset?”

Jim: The alarms? Because it’s preventive. Hopefully, you’re getting an alarm before it actually happens.

Russel: Right. All the context building is the same. If you think about emergency response, it’s the same thing. It’s all the context building is the same. I just need to say, “If I have this adverse outcome, what is the mitigation?”

Jim: Next step.

Russel: Right. What I’m advocating to industry, and I don’t know that the industry is ready for this ship, but this is what I’m advocating in the article, is I ought to take my PHA, my alarm management, and my emergency response, build that context once, and answer all those questions with the same group of people.

What we typically do is we do the PHA, and then we come back later and we do the alarm management, and then another group does the emergency response. You’re building that context multiple times with multiple groups of people.

Jim: That’s interesting because anytime you develop similar systems in parallel, I’m sure there are some inputs and tweaks that maybe emergency management has done that would be useful to the alarm system earlier on or some other part of it, instead of them recreating the wheel each time and just tacking on their specialty on the end of it basically, right?

Russel: Exactly.

Jim: What do you think the barrier is? Is it just because operating companies are siloed and those are different projects, so they get different project managers and…?

Russel: That’s part of it. One of the good things that’s going on in our business is most operators are beginning to lean into API 1173 pipeline safety management systems. That’s requiring pipeliners to start to look at cross-functional types of teams to focus on specific things. Historically, that’s been hard to do.

Pipelining is very technical. It’s got a lot of very technical, very deep disciplines, and there’s a lot of work required in those disciplines. Getting those disciplines to work across one another is challenging, particularly when they exist in different places or I’ve got to pull people out of the field to participate in these processes. All those things, they’re challenging, to say the least.

Jim: If there’s anything that our business in particular is good at is recognizing where they can increase efficiencies. In the end, if they were doing it the way you’re advocating, it would be quicker, and more complete, and more integrated, and save them a lot of pain on the back end of each one of those systems, right?

Russel: That’s right.

Jim: It just makes sense. All you pipeline operators listening out there, go to your bosses, tell them, “Hey, this is what we got to do. We need to integrate these systems.” I don’t know, maybe that’s a new pipeline specialty, is just being able to work across disciplines.

In the business, I’ve met some guys who seem pretty adept at that. It’s rare. It’s like a unicorn or something, but there are people out there who can do that. It seems like if I was an operator, I’d want to tap those guys to run these teams.

Russel: Absolutely. I do want to talk about one of the things that’s in the article. There’s a chart that comes from API and AOPL, now LEPA, Liquid Energy Pipeline Association. It’s talking about the priorities for managing a pipeline incident.

Jim: I see that.

Russel: I want to connect this with how this relates to alarm rationalization. The first thing, safety first. That relates to alarm rationalization, because when I’m rationalizing, I’m quantifying, risk, I’m looking at, what are the risks, given this particular alarm?

That helps you understand what the safety issues are. The next thing is isolate the area and size up the incident. If I actually know where those alarms are located, that helps inform that. Identify the operation and the product. I’m going to know that with the alarm. I’m going to know what pipeline it is.

If I’m running batches, I’m going to know what’s in the batch, that sort of thing. Then, I can build and work my plan. The same information you’re capturing in alarm rationalization is very similar to the information you need for incident response.

Jim: Of course, because you need all of those. You need all of those information, those data points right there.

Russel: The other thing that’s interesting about this whole process, I cannot control my alarm activations. Whatever’s going on in the process is going to drive what alarms I get. I can control my alarm management plan. Likewise, I can’t control what emergencies occur. Things happen that are outside of my control.

There’s dig incidents, people break lines at their house when they’re putting in their irrigation system. Things happen. What we can do is we can plan. We can look at, “Where do we set our alarms?” We can look at how we do our rationalization. We can look at how we’re doing our emergency planning.

If we do all that well and we use the right kind of tools, then I can make that process more holistic, and I can reduce the amount of context building required to be able to get there effectively.

Jim: Then, decrease the response time if that’s what it needs to be. That’s everything that every operator would hope for, because if you do have something bad going on, being able to get out there and have a plan to stop it as quickly as possible, that’s the best.

Russel: Getting things turned off as quickly as possible and getting a rope around it as quickly as possible is really important.

Jim: It’s strange, because I think, rightfully so, that pipelines are the safest way to transport anything, safer than rail cars, safer than whatever, but anytime there’s any kind of incident, it really makes the news. It really makes the news. Most of the time, it’s not a small thing. It’s not a train overturned with some chemicals. It’s like…

Russel: Pipelines, it’s not like a chemical plant, or a refining plant, or that kind of stuff where it’s all inside the fence, right?

Jim: Right, exactly.

Russel: It’s across my back fence.

Jim: Exactly.

Russel: Those kinds of things tend to cause pipelines to get more visibility than they might otherwise.

Jim: All right, Russel. Thanks for giving us an overview. Again, I’m going to remind everybody, that’s the January issue, 2024, Vol. 251, No. 1, Optimizing Emergency Response Through Alarm Management. Russel, before we go, any last words on the article or words of wisdom for our audience?

Russel: I would like to say, Jim, thanks for letting me be a guest on the Pipeline Technology Podcast and coming on and being the host. You’re awesome. I appreciate it. For those that don’t know, Jim’s a very important part of the things that go on around the Pipeline Technology Podcast.

He’s worked a lot in the background on the things we’ve done over the last few years. Really important piece of what we’re doing. You ought to check him out on LinkedIn and see some of the other things he’s got his fingers in.

Jim: Thanks for that endorsement, Russel. Everybody who listens to this podcast, they know where to find you, so we don’t even have to tell them.

Russel: That’s right.

Jim: Talk to you later.

Russel: I hope you enjoyed this month’s episode of the Pipeline Technology Podcast and our conversation with myself and Jim. If you’d like to support the podcast, please leave us a review and you can do that on Apple Podcasts, Google Play, wherever you happen to listen.

If there’s a Pipeline & Gas Journal article where you’d like to hear from the author, please let me know on the Contact Us page at PipelinePodcastNetwork.com or reach out to me on LinkedIn. Thanks for listening. I’ll talk to you next month.