This week’s Pipeliners Podcast episode features Ross Adams of EnerSys Corporation discussing the 2021 PHMSA Enforcement Actions pertaining to the pipeline control room and how operators can address team training requirements in the PHMSA CRM Rule.

In this episode, you will learn about the lessons learned for control room managers and other control room personnel from the PHMSA Enforcement Actions, some of the key findings that led to proposed fines, how operators can advance team training to satisfy the CRM Rule, and other key Control Room Management topics.

2021 PHMSA Enforcement Actions: Show Notes, Links, and Insider Terms

• Ross Adams is the General Manager of EnerSys Corporation. Connect with Ross on LinkedIn.
  • Listen to Ross’ previous appearances on the Pipeliners Podcast.
  • EnerSys Corporation is a frequent sponsor of the Pipeliners Podcast. Find out more about how EnerSys supports the pipeline control room through compliance, audit readiness, and control room management through the POEMS Control Room Management (CRM Suite) software suite. 
    • Access Ross’ blog recapping the 2021 PHMSA Enforcement Actions pertaining to the CRM Rule.
• PHMSA (Pipeline and Hazardous Materials Safety Administration) is responsible for providing pipeline safety oversight through regulatory rulemaking, NTSB recommendations, and other important functions to protect people and the environment through the safe transportation of energy and other hazardous materials.
  • PHMSA’s inspection and enforcement department promulgates safety and training standards by working to ensure that PHMSA-regulated operators comply with regulations and meet the expectations for safe, reliable, and environmentally sound operation of their facilities.
    • Access PHMSA FAQs.
    • Access PHMSA Inspection Protocols.
• The CRM Rule (Control Room Management Rule as defined by 49 CFR Parts 192 and 195) introduced by PHMSA provides regulations and guidelines for control room managers to safely operate a pipeline. PHMSA’s pipeline safety regulations prescribe safety requirements for controllers, control rooms, and SCADA systems used to remotely monitor and control pipeline operations.
  • Control Room Management is regulated by PHMSA under 49 CFR Parts 192 and 195 for the transport of gas and hazardous liquid pipelines, respectively. PHMSA’s pipeline safety regulations prescribe safety requirements for controllers, control rooms, and SCADA systems used to remotely monitor and control pipeline operations.
• Alarm management is the process of managing the alarming system in a pipeline operation by documenting the alarm rationalization process, assisting controller alarm response, and generating alarm reports that comply with the CRM Rule for control room management.
  • Alarm rationalization is a component of the Alarm Management process of analyzing configured alarms to determine causes and consequences so that alarm priorities can be determined to adhere to API 1167. Additionally, this information is documented and made available to the controller to improve responses to uncommon alarm conditions.
    • An alarm management program is a method to manage the alarm rationalization process in a pipeline control room.
• Crew Resource Management Whitepaper is a document created to share recommended practices and information on Crew Resource Management (CRM) and promote the development of CRM training for both Air Operators having CRM training responsibilities and Competent Authorities having oversight responsibilities.
• Clint Bodungen is an ICS cybersecurity guru, the author of “Hacking Exposed: Industrial Control Systems,” and he teaches at the Gas Certification Institute (GCI). Connect with Clint on LinkedIn.
  • ThreatGEN is for organizations, professionals, students, and enthusiasts to learn, interact and share industrial cybersecurity knowledge.
• Doug Rothenberg is the President and Principal Consultant of D-RoTH, Inc., a technology consulting company providing innovative technology and services for industry. Doug’s specialty is control alarm management training and consulting for the industrial process industries. Find and connect with Doug on LinkedIn.
• SCADA (Supervisory Control and Data Acquisition) is a system of software and technology that allows pipeliners to control processes locally or at remote locations.

2021 PHMSA Enforcement Actions: Full Episode Transcript

Russel Treat:  Welcome to the Pipeliners Podcast, episode 218, sponsored by EnerACT Energy Services, supporting pipeline operators to achieve natural compliance through plans, procedures, and tools implemented to automatically create and retain records as the work is performed. Find out more about EnerACT Energy Services at EnerACTEnergyServices.com.

[background music]

Announcer:  The Pipeliners Podcast, where professionals, Bubba geeks, and industry insiders share their knowledge and experience about technology, projects, and pipeline operations. Now, your host, Russel Treat.

Russel:  Thanks for listening to Pipeliners Podcast. I appreciate you taking the time. To show the appreciation, we give away a customized YETI tumbler for one listener every episode. This week, our winner is Sheila Putman with ONEOK. Congratulations, Sheila, your YETI is on its way. To learn how you can win this signature prize, stick around till the end of the episode.

This week, Ross Adams is returning to talk to us about PHMSA’s 2021 Enforcement Actions for control room management and team training. Hey, Ross, welcome back to the Pipeliners Podcast.

Ross Adams:  Hey, Russel. Thanks for having me. I’m actually sitting here drinking out of my Pipeliners Podcast YETI trying to get my voice ready for today’s session. I’m ready to go. It’s good to be with you.

Russel:  There’s no better way than to have a nice beverage from a Pipeliners Podcast YETI. Thanks for the shameless plug. I appreciate it.

Ross:  There you go.

Russel:  This is for all of our control room brethren out there. I asked you to come and talk about the recent blog post that you wrote on the 2021 PHMSA Enforcement Actions around the control room. Before we dive into the details, I think what I’d like to ask is what did you notice just in your review of that? What came up for you as new or different or something to pay attention to?

Ross:  I think that we’ve been doing this exercise for several years now. It’s always really helpful for us and the work that we do. There’s a story to be told. I think our conversation today is going to be very much around tying a bow around much of what we’ve been talking about recently. I did a podcast recently pertinent to a particular case study.

We’re going through PHMSA and the CRM inspection. The data that we’re looking at coming out of PHMSA’s 2021 year pertain to CRM is in alignment with our experience. What I mean by that is that PHMSA’s in Year 2 plus of their three-year audit initiative. They have made good progress and continue to plan for more good progress moving forward. In fact, we’ve got several customers that have had audits scheduled here in the first half of the year already. We will be supporting them.

This whole story within that three-year initiative of them going deeper, being more rigorous, asking tougher questions, digging into more of a procedure orientated mindset–all of those things are wrapped up into not only the story that we’re trying to tell from our experience but also from the data that we’re seeing looking back at 2021.

Russel:  Reading your blog, one of the things that popped out at me is there were 163 individual findings in violations related to Control Room Management. That to me seems like a much larger number than what we have seen in previous years.

I can’t recall off the top of my head what those numbers were. It seems like there’s certainly an escalation there. Is that what you’re seeing as well?

Ross:  I think it’s right. Looking at the data, we think about it not just in terms of individual findings, but in terms of total cases, the number that we wrote up is the total cases against pipeline operators is up 22 percent compared to the prior year. 

Russel:  Is that just related to CRM or is that the entirety of pipeline safety?

Ross:  My understanding is that that’s the entirety of pipeline safety. I don’t have a number for you, necessarily, to say that CRM is up or down, but CRM is a significant portion of those total cases–163 out of 236, I think, if we’re looking at the right information.

What that means is that as PHMSA’s trending up in their activity levels, and they’ve been pretty consistent in the CRM space for the last couple of years.

One of the really helpful documents that’s out there for public use is Byron Coy’s PHMSA PowerPoint presentation. If you go to the PHMSA’s CRM web page, it’s right over there in the bottom left, there’s a link to it. Their 2020 and 2021 planned enforcement, assuming audit count was consistent, we would expect the same for 2022 as well.

Russel:  In the presentation, we both saw of Byron at the AGA conference back in October that they said they expected to complete their three-year plan and be on target. Certainly, if you look at the level of activity, that’s reasonable to expect to be true.

Ross:  That’s right.

Russel:  The next thing that I saw, as I was going through this, is that the propensity of these issues, just where they’re landing, is scattered out around, but there’s a lot in alarm management and a lot in training. Again, at least in my experience, that’s a little different than in previous years. Which would you concur with that?

Ross:  No, that’s right. I think that in terms of the percentage of unsatisfactories from within the audits is up in those areas. There is, at present, a focus or has been a focus in those areas. I think that you dig into that, and you see that part of the area of focus around training correlates with the more recent regulations that incorporate team training as a requirement.

You and I, of course, have had conversations about that and about some of the helpful information that’s out there and trying to paint a rudimentary framework at least of understanding for how the implementation should occur for team training, but because that’s newer, I think that there’s been a focus and increased level of focus from PHMSA to ensure that operators are taking that new requirement seriously.

Alarm Management is difficult. There’s a lot of work tied up into an effective Alarm Management program, and you and I have also had conversations on this podcast about Alarm Management, and there’s a good reason for that.

It is difficult. There’s a lot to it. There are a lot of moving parts. There are a lot of silos that have to be broken down to be effective. It doesn’t surprise me that PHMSA is focusing on those areas and has been focusing on those areas. We’re hearing the same story from folks that we work with and interact within different agency bodies.

That’s not to say that in 2022 and then moving forward from there that PHMSA will continue to focus there as heavily, but it’s certainly, as recently as November, been an area that they have particular attention to.

Russel:  Let’s take a little bit to dig into this whole team training conversation for a minute. You said that a lot of the findings in the training area or team training findings. What percentage of those are just not keeping the records versus what percentage of those were something material you’re not doing what the rule requires?

Ross:  It’s hard to know. I’ve done this work. Actually, in the work that we do in supporting our operators, we want to dive into the details of each finding/case, which means that I look at and review throughout the year that information.

We try to learn from industry and learn from others who’s doing what, and where are the traps that one might fall into as we help folks manage CRM programs. I would say that, whether it’s team training specifically or across the board, that there are a couple of major issues.

There always is… Like failure to have proper policy and procedure, establishing that framework. You and I have talked about, as we talk about natural compliance rate, that foundation and how important it is. There are operators that are inadequate in that space. That is a huge portion looking overall at where there are unsatisfactories that PHMSA’s found.

Documentation is a big part of it from a policy and procedural standpoint. And then inadequate process as well. Maybe you’ve got the policy in place, but the way that you’re going about it may not be as detailed as it needs to be.

I have often talked about how procedures need to be written in such a way that if you take someone from off the street and you have them execute the procedure, it should be executed at the same standard as your most senior controller.

In preaching that message, we’ve gotten eye rolls over the years because there is a level of education that your controllers have and contexts that they have that someone off the street might not. What we heard from PHMSA recently is that that philosophy in terms of processes and processes being adequate is really what they’re looking for as well.

We want to be leaning into that as well. Policies, processes, and then implementation. Actually, implementation or lack of implementation was one of the least common issues. Certainly, it’s still statistically valid and it’s something to pay attention to.

Russel:  Just to piggyback on what you’re saying here. In my experience, as PHMSA rolls out a new rule and in the overall scale of things, team training is still pretty dadgum new, the first thing they do is they come through and they check and say, “Hey, does your policy conform to what the rule is requiring?”

They come back and they’ll say, “Well, do your procedures demonstrate an effective implementation of your policy?” Then they’ll come back and they’ll say, “Are you keeping records evidencing that you’re following your procedures?” Lastly, they’ll come back and they’ll start really digging in and start asking questions about, “Well, what’s the effectiveness of your program?”

We’re just in the natural evolution of a typical new rule. There’s a lot going on where PHMSA’s looking at policy and procedure and they’re not really looking at implementation. That’s coming.

A lot of operators are going to have problems with team training from an implementation standpoint, because it’s a tough one to do the way that PHMSA expects the operators to do team training.

Ross:  That’s right. Looking at this data, that may be true in the team training space, but then you look at Alarm Management or some of these other sections. Point-to-point was another element that scored high in terms of where the findings were. No records or documentation was maybe more the area of focus in those places.

If I’m an operator, I have to be thinking about all of those pieces of my approach, the policy and procedure, the implementation, and the records that come out of it. You and I have talked about the concept of natural compliance and the tools that are available to try to support that because that is a lot to manage.

Russel:  It is.

Ross:  As new regulations come out, team training is a great example. Of course, we’ve had it for a few years, but we’ve had to try to organize and mobilize significant portions of our organization to participate in this training, and then we had COVID happen right in the middle of a lot of that.

It’s not been easy by any means. When we look back at what PHMSA’s focusing on, certainly, the policies, the foundations, the frameworks, the processes, the implementation itself, but even perhaps, more importantly, the records and the documentation to prove that the implementation occurred in accordance with what you said you were going to do.

Russel:  I want to talk a little bit about what the challenges are that people are coming up against in team training. I will say just right at the beginning of this conversation, I don’t have any good answers yet for a solution. I wish I did, but I don’t.

One of the things that the operators are really trying to get their hands around or their heads around is, “How do I do this for all of the people that are impacted or need to be impacted by team training?” I know of at least one operator where I asked the question, “How many people do you have to get through this training?” The number was 1,500.

The reality is, when you talk about all of the people who might have a need to interact with the control room, that’s a big number for most operators. It’s a big part of your employee set.

Ross:  When you go and look at the definitions and the qualifications for those that have to participate, there are a few keywords to pay attention to. “Reasonably be expected” is one of the qualifiers and then “operationally collaborate” is the other.

While there is a wide net that has to be cast to incorporate people that fall into both of those areas, attention needs to be paid to those as qualifying factors.

Russel:  The other thing that’s stated in the FAQs around the team training requirements is that computer-based training (CBT) is inadequate to meet the requirements of the rule. They don’t say you can’t use it, but they are saying you can’t use it exclusively, which warrants some discussion.

Ross:  The approach that we’ve seen folks take that I believe is in the spirit of what PHMSA’s put out on this is that CBTs may be adequate for the soft skills element of the requirement. However, at the end of the day, when people are operationally collaborating, they’re not doing it through a computer, necessarily. There needs to be a human element involved in the exercises that are performed.

We’ve talked about this in the team training podcast. The original Crew Resource Management whitepapers from the European airline agencies that talk about certain best practices and having shorter exercises – maybe 20 or 30 minutes with an hour debrief – those make sense. They make sense as you have to get these people in a room and move them through normal, abnormal emergency operating conditions, and engage them in the superseding of controllers by others if that’s part of your program.

As I sit here and talk about, for the listeners, that maybe are less familiar with this requirement, you’re starting to feel the weight of the complexity. I would just say that that’s okay. We sit and listen to a lot of folks talk about how they’ve implemented this so far. I think there’s a lot of variation to date. I would point folks to the FAQs as a valuable resource.

The other thing is the integrated inspection questions for Control Room Management do account for team training. There is particular emphasis on the ways and then the content of records that’s required for team training. Some of the findings that were pertained to team training certainly had to do with some of those subtleties in the record-keeping requirements. That’s another thing that folks might want to pay attention to.

Russel:  Certainly. You know me well enough to know that whenever there’s an industry need like this, I’m always noodling on how can we come up with a way to creatively solve the problem. I would define the problem for operators related this kind of training as two parts.

One is the large number of people that are impacted by this. The other is that those people are widely dispersed across the organization. They’re not geographically located in a single area. Those two things cause some real problems around doing the training. Then, of course, a number of my shift workers so that also as additional complexities.

One of the things that has been frustrating for me is that we’ve had some people, some operators contact us and say, “Hey, could you help us with this?” We’ve had to say, in some cases, “Sorry, no, we don’t really have a good solution for that.”

In my noodling about, if you will, I had a conversation, some people will recognize this name if they’ve listened to the podcast for a long time. A gentleman by name of Clint Bodungen has been on the podcast a number of times. Clint has a company called ThreatGEN, and they have built a game that facilitates training and capability development around cybersecurity.

Clint’s going to be on the podcast some time in the next few weeks. We’ve got the recording scheduled. I don’t know exactly when it will drop. It’ll be coming out soon. We’re going to talk about how a gaming platform like what he’s built for cybersecurity training and incident response be repurposed and used to do something like control routine training.

What are the features that you would need? To your point, one of the key things that PHMSA is looking for is actually want to see people interact, which means they have to talk to one another. You actually need to have people talking to one another to have the exercise create the benefit you’re looking for.

Likewise, just last night, I had a conversation with Doug Rothenberg, another name that people might recognize because he’s an alarm and control room guru who’s written some great 500-page textbooks around these subjects. I was having this same conversation with him about what his thoughts were, about how would you create a toolset to facilitate this kind of training.

To me, that’s a really interesting idea, but it’s very, very new. It’s not a developed idea yet. I would say that it’s certainly something we’re looking at and trying to see if we can come up with a solution for.

Ross:  Interesting. We’ll have to tune in to find out more is what it sounds like.

Russel:  [laughs] I’m going to have to record to find out more. What I’m planning on doing is going through this process and doing the discovery as I record podcasts to just let people listen as I’m noodling on idea and see if I could come up with something that works and somebody else might listen and have a better idea than me. If they do, God bless them. We’ll see. We’ll all discover.

Ross:  That level of innovation and collaboration is a true value that this industry has. It’s always exciting to explore those spaces.

Russel:  Ultimately, we’re all trying to figure out how to get better and how to do that in a way that’s affordable and reasonable, given how we operate. I want to ask another question along this vein of these findings and violations. I know that EnerSys wrote a pretty extensive blog on this topic. It’s on the EnerSys website. What other kinds of things are you planning to do with this content?

Ross:  Thanks for asking, Russel. Our team put this data together. We’re going to continue to dig into this topic because it’s an important one. Just as I said, this innovation and this collaboration is important within industry. You got some ideas for that in the team training space. We want to be at the forefront of that across the board from a control room management space.

One of the things that we want to do is, as lessons learned are such a big part of the Control Room Management Rule, we want to perpetuate that concept by continuing to share key information around PHMSA’s findings. We’re going to run a series of blog posts over the course of the year. Each month, we’ll focus on a particular element of the rule.

We’re going to break down what some of the audit findings were, some of the data around it, but also just the key lessons learned, and then dive into solutions because at the end of the day, what we’re reading and learning about is one story, but there’s always another path we can take.

Without turning your podcasts into a sales pitch, we want to be holistic in our ability to provide solutions via natural compliance to our customers. We’re learning from a lot of what industry is learning, and we’re trying to incorporate it into solutions that will make a difference for people who are looking for that transformational impact in their organizations.

Russel:  That’s a mouthful. Just for the listeners, as always, we always try to link up resources. We’ll put a link on the Pipeliners Podcast website episode page for this episode, and we’ll give you a way to get to the blog that was written.

Likewise, we’ll put a link in that’ll allow you to get connected in case you want to see the other information that EnerSys is going to be putting out as it puts it out.

Ross:  The bottom line, we’ve put out an introductory blog post. We’re going to continue to lean into the conversation. As operators are continuously improving their CRM programs, we’re going to be supporting that effort and continuously improving ourselves and our solutions as well. That’ll be an exciting journey that we’re going to be taking over the course of the year.

At the end of 2022, which I’m sure we’ll all blink and it’ll be here, we’ll do the same thing all over again in terms of looking back at PHMSA’s findings and where they may be focusing in the future.

Russel:  That will definitely be an ongoing opportunity right there. Ross, what do you think operators ought to be doing right now as it relates to team training? What do you think would be most beneficial for operators in that domain to try and get ahead of where PHMSA’s headed?

Ross:  That’s a good question, Russel. The approach is consistent. If you asked me, “Hey, what do operators need to be doing to comply with and get ahead of what PHMSA’s doing in all of CRM?” the answer is still somewhat the same. You need to do an assessment of where you’re at, what have we done in the past, and see if it is sufficient in terms of the playbook that PHMSA has given us.

The inspection protocol is, of course, a great resource, the rule itself, the FAQs. That’s PHMSA’s playbook. I would encourage your listeners to expand their philosophical understanding of team training. Go read the whitepapers on Crew Resource Management. We certainly have a podcast about this and some other articles that we’ve written.

As you’re looking to establish an understanding for what your gaps are, build your understanding of what their expectations are. Once you’ve defined your gaps, create a plan and implement your plan for closing the gap.

What that might look like is you’re in a situation where, “Hey, look, we have not done team training. We were going to do it, and then COVID hit and so we’re in a situation where we’ve never done it.” That’s fine. Start by contacting HR and getting a list of all of the job titles within operations, go through that list, and identify who fits the criteria for needing to participate in team training and then work on your program. I would recommend keep it simple and grow from simple.

We want to be able to discuss the soft skills and educate our personnel on the soft skills. We want to have some meaningful exercises as well and, of course, document all of that accordingly. It’s a much larger conversation. We have done consultations in this area in the past. It’s been really fruitful.

There are experts out there that can lend some further understanding. Certainly, I’d be happy to have that conversation with any of your listeners that still have questions.

Russel:  The biggest thing I hear about team training is people are pretty clear as to what PHMSA’s asking. They’re really struggling with, “Well, how am I actually going to do this and not have it be too costly or too disruptive to operations?”

Ross:  Similar to the challenges of Alarm Management, these are both intensive efforts and ones that may require collaboration with other groups or departments. If you’re a control room manager and you’re asking a question, “How do I get all my alarms rationalized?” or if you’re asking the question, “How do I implement team training?” there may be other resources within your company that can help.

As it relates to Alarm Management, maybe it’s the SCADA group, for example, that can provide some support in that area. If you’re talking about team training, maybe you have a training department that has put on other types of training. They’re not going to know the requirements of the CRM Rule. It’s incumbent upon you to help connect them with the right resources as you’re learning.

There are options even within your own organization for expanding your resources that you can tap into. The field does emergency response drills. While that is not all of what team training is, there is an element to those drills where there may be some overlap. Having conversations with the folks that put those drills on may be another place where you can cover some ground quickly as opposed to trying to carry the whole load yourself.

Russel:  Just to summarize, what I hear you saying is, one, don’t try to do it all yourself. Look for other organizations and for other things that are happening in the organization that you can piggyback on being one strategy. The other is do your analysis and do something, which is better than doing nothing.

Ross:  That’s right. Make sure you have records to prove that you did it. They’re all in one place and they’re easy to find, and so on and so forth.

Russel:  Exactly. Don’t forget the paperwork.

Ross:  Does this sound similar to what we’ve talked about before? It’s important that we continue to beat the same drum because this is the playbook for how to get these programs created and implemented. There are folks that have been working on a CRM program for 10 years now, and they’re still learning. They’re going to continue to learn. They’re going to have to continue to adapt.

If you’re relatively new to this conversation, just know that there are folks out there that are working to innovate. They’re willing to collaborate. There are plenty of resources to learn from. While it’s overwhelming, it is achievable to dig into this. If you hear me saying some of the same things over and over, it’s because we may be on to something.

Russel:  The reality of it is, at a high level, the steps are pretty simple. Figure out what you need to do. Write down what you’re going to do. Do it. Make sure you have records that show you did it. That sounds real easy. What tends to happen is the first two pieces are so hard that a lot of people, a lot of us, myself included, get a little lax on that on that last step.

We do this same kind of stuff internal to our companies. We have all the same challenges that all of our customers do. All that being said, as we get clearer about what we’re going to do and as we have records we can go back and look and say, “Did we do it,” we get better. That is the truth. We do get better.

Ross:  No doubt.

Russel:  Listen, Ross, this is fun as always. Hopefully, we didn’t babble too much for the listeners. I always appreciate having the opportunity to visit with you and always enjoy talking to you. Thanks for coming on.

For the listeners, once again, I put Ross on the spot at the last minute because I’ve been having some scheduling difficulties. I want to make sure that we have something meaningful to talk about every week. Everybody say thanks to Ross for stepping up and bringing it once again.

Ross:  Russel, thank you. It is a pleasure to be here. If you are listening and you think, “Well, hey,” I’ll tell you this, if Ross can do it, anybody can do it. If I can do it, anybody can do it. There’s plenty of you out there that have something interesting to share. This is a great opportunity. I really enjoyed being on the show always and I’m grateful that you’d have me on, Russel. Thank you.

Russel:  My pleasure, Ross. We’ll talk to you again soon.

Ross:  All right.

Russel:  I hope you enjoyed this week’s episode of the Pipeliners Podcast and our conversation with Ross. Just a reminder, before you go, you should register to win our customized Pipeliners Podcast YETI tumbler. Simply visit pipelinepodcastnetwork.com/win and enter yourself in the drawing.

If you’d like to support the podcast, the best way to do that is to leave us a review. You can do that on Apple Podcast, Google Play, Stitcher. You can find instructions at PipelinePodcastNetwork.com.

[background music]

Russel:  If you have ideas, questions, or topics you’d be interested in, please let me know either on the Contact Us page or reach out to me on LinkedIn. Thanks for listening. I’ll talk to you next week.

[music]

Transcription by CastingWords