This week’s Pipeliners Podcast episode features host Russel Treat, Pipeline Podcast Network Founder, discussing previously aired episodes over alarm management and how to incorporate those lessons into your own alarm management plan.

In this episode, you will learn about the importance of alarm management and the ways it has evolved as it has become more widespread throughout the industry, as well as what to include in your own program.

Jump To:

Alarm Management Show Notes, Links, and Insider Terms:

• Russel Treat is the CEO of EnerACT Energy Services, as well as the host of the Pipeliners Podcast and the founder of the Pipeline Podcast Network. Connect with Russel on LinkedIn.
• EnerACT Energy Services is a holding company of oil and gas technology and service companies. EnerSys Corporation is a frequent sponsor of the Pipeliners Podcast. Find out more about how EnerSys supports the pipeline control room through compliance, audit readiness, and control room management through the POEMS Control Room Management (CRM Suite) software suite.
• Control Room Management is regulated by PHMSA under 49 CFR Parts 192 and 195 for the transport of gas and hazardous liquid pipelines, respectively. PHMSA’s pipeline safety regulations prescribe safety requirements for controllers, control rooms, and SCADA systems used to remotely monitor and control pipeline operations.
• Alarm management is the process of managing the alarming system in a pipeline operation by documenting the alarm rationalization process, assisting controller alarm response, and generating alarm reports that comply with the CRM Rule for control room management.
• SCADA (Supervisory Control and Data Acquisition) is a system of software and technology that allows pipeliners to control processes locally or at remote locations.
• Valve and Rupture Rule is a newly updated PHMSA regulation. This rule establishes requirements for rupture-mitigation valves, such as spacing, maintenance and inspection, and risk analysis. The final rule also requires operators of gas and hazardous liquid pipelines to contact 9-1-1 emergency call centers immediately upon notification of a potential rupture and conduct post-rupture investigations and reviews.
• Process Safety Management (PSM) refers to a set of interrelated approaches to managing hazards associated with the process industries and is intended to reduce the frequency and severity of incidents resulting from releases of chemicals and other energy sources.
• Process Hazard Analysis (PHA) analyzes the potential causes and consequences of an incident involving natural gas, hazardous liquids, or other product in a pipeline system.
• AOC (Abnormal Operating Condition) is defined by the 49 CFR Subpart 195.503 and 192.803 as a condition identified by a pipeline operator that may indicate a malfunction of a component or deviation from normal operations that may indicate a condition exceeding design limits or result in a hazard(s) to persons, property, or the environment.
• Slack line is a condition when both liquid and vapor exist in a liquid pipeline at the same time. A similar term is column separation.
• API Conference will take place in Nashville, Tennessee, May 1-3, 2023 where the industry is able to connect, collaborate, learn, and continue leading the energy evolution.
• SCADA (Supervisory Control and Data Acquisition) is a system of software and technology that allows pipeliners to control processes locally or at remote location. SCADA breaks down into two key functions: supervisory control and data acquisition. Included is managing the field, communication, and control room technology components that send and receive valuable data, allowing users to respond to the data.
  • HMI (Human Machine Interface) is the user interface that connects an operator to the controller in pipeline operations. High-performance HMI is the next level of taking available data and presenting it as information that is helpful to the controller in understanding the present and future activity in the pipeline.
• PLCs (Programmable Logic Controllers) are programmable devices placed in the field that take action when certain conditions are met in a pipeline program.
• Plan Do Check Act (Deming Method) is a continuous quality improvement model consisting of a logical sequence of four repetitive steps for continuous improvement and learning: Plan, Do, Check (Study), and Act.
• Alarm rationalization is a component of the Alarm Management process of analyzing configured alarms to determine causes and consequences so that alarm priorities can be determined to adhere to API 1167. Additionally, this information is documented and made available to the controller to improve responses to uncommon alarm conditions.
  • API 1167 provides operators with recommended industry practices in the development, implementation, and maintenance of an Alarm Management program. The implementation of API 1167 is required by reference in the CRM Rule.
• ISA-18.2 defines an alarm as “an audible and/or visible means of indicating to the operator an equipment malfunction, process deviation, or abnormal condition requiring a response.
• Episode 51 featuring Jerry Coleman discusses the challenges of Alarm Management.
• Episode 165 and Episode 166 with Doug Rothenberg goes over what alarm management is and its history.
• Episode 186 with Ross Adams talks over alarm management from a compliance standpoint, and Episode 163 discusses overcoming challenges in alarm management. 
• Episode 96 with Stephen Swanson talks about process safety management and combines that with alarm management.
• Episode 28 featuring Giancarlo Milano discusses leak alarm response.

Alarm Management Full Episode Transcript:

Russel Treat: Welcome to “The Pipeliner’s Podcast”, episode 282, sponsored by the American Petroleum Institute, driving safety, environmental protection, and sustainability across the natural gas and oil industry through world class standards and safety programs.

Since its formation as a standard setting organization in 1919, API has developed more than 800 standards to enhance industry operations worldwide. Find out more about API at API.org.

Announcer: The Pipeliner’s Podcast, where professionals, Bubba geeks, and industry insiders share their knowledge and experience about technology, projects, and pipeline operations.

Now, your host, Russel Treat.

Russel: Thanks for listening to The Pipeliner’s Podcast. I appreciate that you’re taking the time. To show that appreciation, we give away a cool customized YETI tumbler to one listener every episode.

This week our winner is Amanda Farrel at Chesapeake Energy. To learn how you can win this signature prize, stick around until the end of the episode.

Fundamentals of Alarm Management

This week we’re going to do something a little different. It’s just me. There’s no guest. I want to talk about alarm management.

One of the things that we have certainly learned in our businesses as we work with pipeline operators to implement control room management tools and programs in their companies is one of the most challenging aspects of doing control room management are the alarm management requirements.

It’s very difficult to do it both efficiently and effectively.

We’ve done a number of podcasts over the years on this subject. I thought it’d be a good idea to give you guys a walkthrough and offer this up as, “Hey, this is the listening list or the reading list, if you will, to learn about alarm management for pipeliners.”

I wouldn’t recommend, necessarily, listening to the episodes in the order they were recorded because sometimes the way this works is we’re doing the podcast is it’s, who are we talking to? What do we think would be of interest? That’s not necessarily the best sequence.

Here’s what I think I would recommend that you start with. We did an episode 51 with Jeremy Coleman where we talked about the challenges of alarm management. That’s a good place to start. There’s a lot of challenges with alarm management. We probably have a better understanding of that now than we did…Gosh, that would have been five years ago when we recorded that episode, now.

Pipeliners Podcast Episode 51:

Some of the challenges were these. One, alarm management is one of the aspects of control room management that cuts across multiple organizations. It probably impacts engineering. It will impact automation. It will impact SCADA. It could impact operations.

If you’re looking at data in the control room from manned facilities, then there’s what those folks are doing to run the facility versus what you’re doing to run the pipeline. How does that get arbitrated?

There’s a lot of details.

Likewise, alarm management has to be implemented consistently across the entirety of the automation. It’s got to be done correctly. However you implement your program, you’ve got to get clear about that implementation as it relates to the field automation, as it relates to any local automation, or local auto HMI as it relates to what’s going on in the gas control center.

Those things can all be fairly significant challenges. That’s where I think you ought to start. I think for anybody who’s trying to rethink a program…Everybody who’s a pipeliner’s got something in place for this, but if you’re trying to rethink it and figure out how to do it in a more streamlined fashion, that’s a really good place to start.

The next thing I think I would listen to is episode 165 and episode 166. Give you a little bit of history from my standpoint. In 2007 was the first time we and our company actually started hearing about control room management and looking at it, trying to figure out what is this going to mean for our business? What is this going to mean for our customers?

Pipeliners Podcast Episode 165:

At that time, we had no knowledge of what alarm management was. Why wasn’t there alarm management? Our knowledge about that was pretty much zero.

We were referred to Doug Rothenberg. Doug Rothenberg literally wrote the textbook on alarm management. He’s a PhD engineer. His PhD is in process management, alarm management. He wrote the book on alarm management for the process industries.

We contracted with Doug. We started offering alarm management training. Basically, we were offering his training. I attended his training…Oh, there was probably a course of about 18 months. I went to his training probably a half a dozen times.

I have to say, I wasn’t getting it. I could pass his exam, but I wasn’t really getting it, if that makes sense.

Then, I think it was probably the fourth or fifth time I was in the class, I had an “aha” moment and understood what he was actually trying to teach about what alarm management is.

Alarm management, fundamentally, is how do I use alarming in a way that I’m giving one and only one alarm for an abnormal operating condition? When I alarm that abnormal operating condition, I am providing that alarming with instructions around how to process the alarm.

If you think about the new valve and rupture mitigation rule and the things that are in that code around rupture, what I think the regulators are asking for is I want an alarm that’s clear and evident and is going to require action.

That is fundamentally the challenge of alarm management. Oftentimes, we don’t get an alarm on the abnormal operating condition. What we get is an alarm on a higher pressure without knowing what is the operating condition that’s driving the higher pressure. Then, the controller, the person in the chair, has to figure that out.

The idea of alarm management is to simplify that process. It’s easy to say, but very difficult to do.Doug Rothenberg’s the guru. Those two episodes we did with Doug, 165 and 166, are a good, fundamental place to start in terms of learning about what alarm management is and how you do it.

Pipeliners Podcast Episode 166:

Compliance & Alarm Management

The next episode I’d recommend you listen to is episode 186. This is an episode I did with Ross Adams. Ross Adams is general manager for EnerSys Corporation. EnerSys specializes in doing work in the control room.

The whole idea was, let’s talk about alarm management from a compliance standpoint. How do I make compliance as straightforward and efficient as possible? I was going to say how to make compliance easy. I don’t know that compliance is ever easy, so that’s probably not the right way to say it.

We’re talking about alarm management from a compliance perspective. What are the records I need to create? How do I keep those records and make sure that I’m keeping those records in a consistent way over time?

That sets a fundamental basis for what it is, and how to do it, and what am I trying to accomplish.

Pipeliners Podcast Episode 186:

Process Safety Management & Alarm Management

The next that I would recommend is looking at episode 96. Episode 96 talks about process safety management and combines that with alarm management. We were working on a project with a customer. There was a really sharp process safety management engineer by the name of Stephen Swanson. He agreed to come on.

We talked about what we were learning. I’ve done a little bit of PSM. I’ve done some PHAs. I’ve been in those with customers.

Pipeliners Podcast Episode 96:

Stephen’s really an expert at it. We were talking about what is the same and what is different between process safety management and process hazard analysis, and what you’re doing in alarm management and alarm rationalization.

Fundamentally, they’re very similar kinds of processes. Anybody who’s been involved in a PHA knows they’re very tedious. You often bring in a facilitator or a consultant to help walk you through things.

That facilitator or consultant’s going to make sure you’re dotting every I and crossing every T and looking at everything you need to look at.

A big part of the process is context building. You bring a team together. You’re probably going to have some automation people, some engineers, some process folks that know how to operate the facility that you’re doing the PHA for, and so forth.

A big part of what you’re doing is you’re getting consensus in that team around how the process works. You’re going through the process flow diagrams. You’re going through PNIDs. You’re asking questions about the automation, what drives shutdowns. You’re going through your cause/effect diagrams, all of that stuff to make sure you have it right.

All of that context building is the same, regardless of whether you’re doing process safety management or a PHA or you’re doing alarm management and alarm rationalization. The fundamental difference between those two processes is this. In a PHA, I’m trying to make sure that if I have an upset, the facility fails, shuts down, shuts in safely.

That can often be done with mechanical reliefs, and automated shutdowns, and so forth.

The difference is in alarm management what I’m asking is for each one of those AOCs that can lead to an upset or lead to a shutdown, how would I alarm it so that the controller sees it in advance. Then, what automation would I put in place so that the controller could take a corrective action from their console such that we don’t have the upset result in a shutdown?

Good process safety management makes sure that you never have an incident due to a process upset. Good alarm management makes sure you never have a shutdown as a result of a process upset.

Really good episode. For those trying to build programs and looking for ways to be more efficient, that’s a way to get there.

Leak Alarm Response

The next thing I would recommend you listen to is episode 28. Episode 28 is something I did really early. We did a whole series on leak detection with Giancarlo Milano from Atmos International. We talk about all the different kinds of leak detection technologies.

Pipeliners Podcast Episode 28:

One of the last episodes we did was on leak alarm response. Leak alarm response is a really fascinating subject because oftentimes leak alarm response is a process of working through what Rothenberg would call weak signals, meaning I know something’s not quite right, but I’m not sure what it is.

We talked primarily about leak alarm response. One of the things I feel kind of obligated to add that’s not in the episode, but for anybody listening and trying to build a program. If you’re doing something to respond to the rupture rule and you’re looking at, how do I respond to a leak alarm? How do I respond to a rupture alarm? Are there differences or not? How does all that work? You’re designing that.

That’s all well and good, but I would tell you that from an operator risk perspective, the riskiest thing you do is restart the pipeline after a shutdown based on a leak alarm. In other words, I’ve got a leak alarm. I shut the pipeline down. I believe the leak alarm to be false, so I’m going to restart.

That’s the riskiest thing you do. There have certainly been pipeline incidents where, because the operator, due to other operating conditions like slack line or production upsets or non-standard operations, mask the problem that the leak alarm was actually real.

Anyways, not in the episode but if you’re listening to this, you’re going to listen to the episode, or you’re designing a program. Pay very particular attention to that idea about what is the appropriate mechanism for doing a restart to make sure I do not restart into a leak.

Alarm Management Challenges

Then, there’s episode 163 on overcoming challenges in alarm management. We start off with here are the challenges. Then, we’re going to build context around, here’s how you would build a program, what you would do in a program. Then, here’s some of the challenges you’re going to come up against as you do that. How do you overcome them?

Pipeliners Podcast Episode 163:

Alarm Management & Emergency Response

Lastly, I’d recommend you listen to “Alarm Management and Emergency Response,” which is episode 63, also with Ross Adams. The idea of this episode is to correlate what we’re doing as we respond to alarms with what happens in the emergency response process.

Pipeliners Podcast Episode 63:

If I were going into a new control room and we’re talking about how we’re going to build a program, we’re going to talk about normal operations. We’re going to talk about abnormal operations. We’re going to talk about emergency operations.

The idea of abnormal in alarm management is I use alarms to draw my attention to an abnormal condition and start a countdown on a clock. I’ve got this much time to manage this abnormal condition before it becomes an upset. That’s the alarm management perspective.

That transitions, if I’m unsuccessful and it ends up in an upset or if it moves so quickly because of something that happened in the field that causes an upset or an incident, then I’m transitioning to emergency response.

This whole episode is, what is all that? How does that dynamic work? How should I design a program around it?

Simplifying Alarm Management

This episode drops on Tuesday, May 2nd. May 2nd, I will be a the API Pipeline Conference in Nashville. At 4:00 in the afternoon, I’m going to be presenting a paper on simplifying alarm management.

I’m going to cover this at a high level. I certainly don’t want to cover all the details because if you’re listening to this and you happen to be at API, I’d like for you to come and see the presentation this afternoon.

But, I think it’s helpful, because this podcast will be around a lot longer than I’ll be in the presentation in Nashville, to cover some of the high points. I’ve already covered some of the things that make alarm management complicated.

One of the other things I haven’t talked about and I’m going to talk about in some detail during the conference is the whole question about where do I manage the alarms.

This is actually one of those things that you need to make these decisions up front. They need to be made at an enterprise level. They need to be made across SCADA, field automation, field operations, control room, so that everybody’s on the same page as to where is the governance? Where are the alarms implemented? How are they activated?

I’m going to talk about that in more detail in the presentation, but that’s a very fundamental kind of conversation. I have a philosophy around that that’s probably more advanced than what most people would want to take on.

Simply stated, I would say that alarms and safety should be managed in the PLCs, but alarms for heading off problems should be managed in the SCADA HMI. That’s a lot of what I’ll be talking about in the conference.

Also, one of the key things that needs to happen in any alarm management program is you’ve got to get really clear about roles and responsibilities. Who’s doing what, particularly if I’m using alarms on the field devices, at facilities, and in the control room so that those definitions are clear across that spectrum.

Because we have a tendency in our industry and our history has been we call everything an alarm, it’s a definition between a little A alarm, meaning anything that comes out SCADA or automation as a notification, and a big A alarm, meaning somebody’s got to do something or we’re going to have a problem.

Getting clear about those kind of definitions for your programs is really critical.

Then, I’ve alluded to this. I want to elaborate on it a little bit more. When I have a manned facility and the pipeline has a visibility to that manned facility, there is an opportunity for there to be a disconnect in the alarm management between those two things.

It’s very important that that does not happen, that those two things are maybe not managed the same, but they’re managed in alignment and proper communication is happening amongst all the parties.

When you don’t have your alarm management well defined, you get problems like floods, spurts of alarms that are more than I can actually effectively answer or respond to. I’ll have problems like lots of bad actors. Chattering alarms, fleeting alarms, sympathetic alarms, meaning whenever I get this one alarm, I get these other three every single time.

All of those kinds of things make it more difficult for a person sitting in the console to understand what’s going on with the process and know what to do when they’re getting those alarms in.

A perfect alarm management system – This is an aspirational go. I don’t know that you can accomplish this in reality – would give you one and only one alarm for every abnormal operating condition.

For example, rather than getting a low suction pressure, I would get an alarm that says valve X closed upstream of compressor Y. That would be a good example of an abnormal operating condition being alarmed versus the pressure being alarmed.

The reality is, given how we do things currently, generally, I have to alarm on the process value. I can’t actually alarm on the process upset or the abnormal condition. That would be a perfect alarm system.

The other thing, particularly with leak alarms, is your false alarm rate. The higher your false alarm rate, the more likely you’re going to miss a real alarm when and if it comes.

Given that leaks, particularly for the liquid pipeliners, are the thing that can cause us the most grief and aggravation with the communities we serve and the communities we operate in, not to mention the regulators and others, that’s a big deal.

Really understanding what my leak alarm response is and how I work through a leak alarm to determine whether it’s real or not is very critical.

If I’m going to simplify my alarm management, you really need to do the whole plan, do, check, adjust thing that is talked about in pipeline safety management. Your plan is going to be your alarm philosophy and your alarm management plan.

Many people out there, pretty much everybody’s going to have an alarm philosophy and/or an alarm management plan. Let me tell you what we’ve found out.

The philosophy is very important because it says to the enterprise, “This is how we divine things. This is how we do things.” The alarm management plans are very important because they lay out the specific implementations based on the SCADA platform or HMI toolset or operating system.

I might have a single alarm philosophy for a pipeline. I might have an alarm management plan for the gas pipeline, and an alarm management plan for the liquid pipeline, and an alarm management plan for my gas compressor stations. Those would all the philosophy and implement them, given the operating realities and the HMI limitations that exist in those various places.

That’s the planning phase.

Then, you have to do, which is rationalize all those alarms. Then, you have to check, which is to run reports to look at my alarm activity, my bad actors, and to perform audits to make sure that what’s configured in my alarming in automation and SCADA matches what my master alarm database says it should be.

Then, finally, I have the adjust, which is I’ve got to review how I’m doing. Am I getting floods or not? Am I having bad actors or not? If so, what do I need to change in order to make sure that that’s not a persistent recurring issue?

From a best practice standpoint, the best place to go for a pipeliner is to get a copy of API 1167 and read through it. There’s other documents out there, if you want other resources that are cited in the 1167. A couple of good ones are Doug Rothenberg’s book, “Alarm Management for Process Safety Systems.”

Another good resource is the ISA standard, ISA 18.2.

The other thing to keep being mindful of… One of the unique things about pipelining versus other kinds of processes is that pipelining has a lot of reuse. To a large extent, if I have a pipeline and that pipeline has mainline valves, every mainline valve is probably instrumented the same way and would need to be rationalized the same way, or at least largely the same way.

One of the things you can do is you can define what are my common areas of operation. I can then specify the alarming, and instrumentation, and so forth that I want for that standard area of operation. Then, I can implement that over and over again.

Of course, certainly, I’m well aware that there’s always differences between what your standard is and what you actually have implemented in the field, but having a standard and knowing how to implement the standard will streamline the field implementation and improve it.

That’s basically where I’m going to wrap this conversation up. If you’re interested in learning more, you can reach out to me on LinkedIn or you can reach out to me through the Pipeliner’s Podcast website. I’d be glad to talk to you if you’ve got specific questions.

I hope you found this helpful.

Thanks for listening to this episode of The Pipeliner’s Podcast. If you’d like to support us, the best way to do that’s to leave a review. You can do that wherever you happen to listen. Apple Podcasts is probably one of the best places because that’s where most of the listeners come from.

Likewise, if you want to win this customized Pipeliner’s Podcast YETI tumbler, simply visit PipelinePodcastNetwork.com/Win, go to the form, and enter yourself in the drawing. They’re really nice. They’re really cool. I want to give you one, but you can’t get one unless you register to win one.

Lastly, if you have ideas, questions, or topics you’d be interested in, please let me know either on my Contact Us page at PipelinePodcastNetwork.com or reach out to me directly on LinkedIn.

Thanks for listening. I’ll talk to you next week.

Transcription by CastingWords