In this week’s Pipeliners Podcast episode, Russel Treat welcomes back Ross Adams of EnerSys Corp. to discuss the important elements of preparing for a PHMSA control room audit.

The conversation is based on many discussions at industry conferences this year, where pipeline operators, control room managers, and other personnel have shared lessons learned on how to achieve compliance, improve efficiency, and utilize software to prepare for an audit.

Listen for three key takeaways to directly apply to your operation and why you should start implementing new processes right away.

Preparing for PHMSA CRM Audit: Show Notes, Links, and Insider Terms

• Ross Adams is an Accounts Service Specialist at EnerSys. Find and connect with Ross on LinkedIn.
  • Find out more about the EnerSys holistic approach to control room management software through POEMS (Pipeline Operations Excellence Management System).
• The GPA Midstream Conference is a midstream-focused gathering of international oil & gas professionals that provides opportunities for networking, learning, and shared lessons on issues impacting the midstream industry. This year’s event was held April 15-18 in Austin.
• The API Pipeline Conference & Cybernetics Symposium features important sessions on pipeline safety, CRM Rule compliance, leak detection programs, and the latest technology for pipeliners. This year’s event was held April 24-26 in St. Louis.
• The AGA Operations Conference is the natural gas industry’s biggest worldwide gathering of natural gas utility and transmission company operations to share technical knowledge, ideas, and practices. This year’s event was held June 26-29 in Washington, D.C.
• The SGA Operating Conference & Expo is a gathering of operators and pipeline professionals to share lessons learned and receive education on the latest practices in the industry. This year’s event was held July 30-August 1 in Tampa, Fla.
• The Texas RRC (Railroad Commision Conference) combines pipeline safety and oil & gas regulations into one event providing information and training on laws, rules, and procedures. This year’s event was held August 21-22 in Austin.
• The CRM Rule (Control Room Management Rule as defined by 49 CFR Parts 192 and 195) introduced by PHMSA provides regulations and guidelines for control room managers to safely operate a pipeline. PHMSA’s pipeline safety regulations prescribe safety requirements for controllers, control rooms, and SCADA systems used to remotely monitor and control pipeline operations.
• Alarm Management is the process of managing the alarming system in a pipeline operation by documenting the alarm rationalization process, assisting controller alarm response, and generating alarm reports that comply with the CRM Rule for control room management.

Preparing for PHMSA CRM Audit: Full Episode Transcript

Russel Treat:  Welcome to the Pipeliners Podcast, episode 44.

[background music]

Announcer:  The Pipeliners Podcast, where professionals, Bubba geeks, and industry insiders share their knowledge and experience about technology, projects, and pipeline operations. Now your host, Russel Treat.

Russel:  Thanks for listening to The Pipeliners Podcast. We appreciate you taking the time, and to show that appreciation, we’re giving away a customized YETI tumbler to one listener each episode. This week, our winner is Alex Popov with National Grid.

Congratulations, Alex, your YETI is on its way. To learn how you can win this signature prize pack, stick around ‘til the end of the episode.

This week on the Pipeliners Podcast, Ross Adams is returning to visit with us about preparing for the PHMSA Control Room Management audit. Ross, welcome back to the Pipeliners Podcast.

Ross Adams:  Thank you so much for having me back. It’s been a lot of fun. As I’ve been around working with different companies, they’re starting to recognize Pipeliners Podcast. I get asked a lot of questions about it. You guys are doing a great job.

Russel:  [laughs] That’s awesome feedback, man. Thanks. That’s good to know.

One of the interesting things about doing this podcast thing is you get together with somebody on the microphone. You talk. You record it. You put it out. You don’t always know how it’s being received, but I’ve certainly heard some of the same kind of feedback. It’s nice to know that it’s being heard by others, as well. That’s awesome.

I asked you on because I know that you have been to a bunch of conferences, as have I. We’ve had some conversation. It seems like there are some themes that are coming out of those conversations.

I thought I’d bring you on and we would talk about that. I guess maybe the place to start is where have you been? What conferences have you been to?

Ross:  It’s been a busy conference season. We’ve been all over the U.S., which has been a lot of fun. Got a chance to go to GPA Midstream in Austin and the API Pipeline Conference up in St. Louis.

I spent a little time at the SGA Conference in Tampa, which was wonderful, and then went back to Austin for the Texas Railroad Commission Conference here, just last month.

Russel:  That’s a bunch. Like you, I’ve been to several of those. I think the only one I went to that you didn’t go to was the AGA Operations Conference.

What kind of themes were you hearing in the conversations? I guess it’s important for the listeners to understand that you and I both focus on control room and pipeline operations, leak detection, those subject matters.

Certainly there’s a lot of other subject matters covered at some of these conferences, but that’s kind of the domain we’re in. What are the themes you’re hearing in that domain?

Ross:  I think first and foremost, I’ll just say that it’s been a real pleasure to attend these conferences, because you’re getting to know and network with some really tremendously talented people who are passionate about what they do.

They’re passionate about the sharing of ideas, and they’re willing to talk about the needs that they have in the context of their operations because for the most part, I think they understand that that cooperation between operators is really important for finding, and identifying, and putting into practice these best practices.

In terms of themes, I think we’re seeing that industry is working diligently to catch up to not only the original rule, but especially some of the more recent addendums to the CRM, control room management rule.

Specifically around the implementation of others who have authority to direct or supersede the technical actions of a controller, as well as the necessity of implementing a team training program, for those who would be reasonably expected to operationally collaborate with your controllers in your control room.

There’s a lot of catch up going on, there’s a lot of diligent work going on, and a lot of sharing of ideas.

Russel:  I certainly saw that as well, and certainly there was a lot of conversation about team training, and who are these others, and what does it mean to supersede, and what is a technical action? There was a lot of conversation about that.

In fact, I saw Byron Coy make a presentation at the AGA Ops conference to the Gas Control Committee on that very subject. I talked about that in the AGA Ops recap, but I think there’s a couple of other things around the audit, so before we dive into that, in your conversation, what would you say the overall state of operators being…

Where are they with regards to their understanding and implementation of the control room management rule, just in your experience and in your conversation?

Ross:  I think we’re at a place where people are getting their feet underneath them to a certain degree, and of course industry is in a healthier place than it was three or four years ago.

What we’re seeing is that not only are operators seeking to reach a place of compliance, but they’re asking a lot of questions around, what are the best practices in terms of how do I make this effort to seek continuous improvement in my CRM compliance an easier task, which points to the use of software.

It points to the integration of resources, streamlining the means of documentation, and so on and so forth.

Russel:  I would characterize it as, everybody has something.

Part of that is that they had to, but everybody does this, making their best efforts to do what’s reasonable, but like any other kind of new business practice or new technology, you do something, and in the doing of that, you learn a lot, and now you start asking a whole bunch of questions about, is there a better way?

Can I make this, like you said, more streamlined, easier to accomplish, and so forth. I think there is a lot of conversation going on around that.

I think there’s a lot of conversation in particular about alarm management and rationalization and analysis, and actually having that be meaningful without being burdensome. There certainly seems to be some conversation about that.

There also seems to be conversation about preparing for the audit, and I know that you had some of those conversations at SGA. I certainly participated in some conversations at AGA about that.

What I would say is that the effort to pull everything together to be ready for the audit for some operators is fairly burdensome. Would you say that’s a fair commentary?

Ross:  I’d say that’s about right. What’s great about these conferences is that people are open about their experience, which only adds to others’ understanding what may be the best way to go about things, or what they need to learn in advance of an audit to make their experience a little less painful.

Russel:  Did you have the opportunity to talk to any operators about their recent audit experience, and how would you characterize that?

Ross:  I did, and those were some great conversations. There were some differences, I’d say subtle ones, between what we saw at the Texas Railroad Commission Conference and what we’ve seen at the conferences that are more national, that are more PHMSA-oriented.

I think on the whole, what we’re seeing is, there’s not a clear expectation for the audit experience. PHMSA obviously has different regional offices, and the auditors are out of those regional offices.

Each operator has had auditors who, even though they use the same audit protocol questions, tend to focus on different areas and tend to have their own unique methods for going through the audit process.

Experiences may vary, I guess is the way to sum it up, [laughs] which I understand makes operators a little bit nervous, as they head into what may be their first audit, but I think that as we listen to the feedback that we’re getting from other operators, and kind of grow in each operator’s own program, there’s no reason to fear.

There is plenty of things to do to prepare yourself. Be prepared to where that experience is a good one.

Russel:  A couple of themes that I think I’m hearing, one is, and I think you pointed this out in what you just said, but there is some differences between areas as to what the auditors are asking for, and in some cases, and how the auditors are interpreting specific aspects of the audit or the audit protocol.

I don’t know this, but I think it makes sense given where we’re at, is part of that is, a lot of these state agencies are just now beginning to kind of get their capability to the place they can perform audits.

I think the other thing that I’m hearing thematically is, people are going deeper when they’re doing this. The auditors are going deeper.

Where two years ago most of the audits were looking at your plan and does your plan conform, now they’re actually looking at, does your plan conform, and what actions should you take, and do you have the records to demonstrate you’re taking the appropriate actions?

I think as we go forward, they’ll be diving even a little deeper and looking at the records in particular, and saying, “Are those records reflective of what you say you’re supposed to do and what’s required under the rules?”

From that standpoint, I think I would use the word that it seems like the audits are becoming more thorough, maybe more intense.

Ross:  I think that’s right. I know that there’s some data that PHMSA released through one of their conference presentations from 2014, where you saw the incidences of non-compliance.

They were really heavy on what we’d call Section A of the CRM Rule, which is the general section, which, to me, indicates that they were at that point looking more so to see that you had a program, that you were leaning into their compliance requirements.

As the rule itself has grown, the supporting documents have grown, and therefore the operators’ ability to adhere to those requirements, and at times surpass them, in terms of best practices.

The data has shown a shift even from 2016, and even more recently, to where those non-compliance instances are spreading out into the more meatier sections of the CRM Rule.

Russel:  I think that’s just because of where the auditors are in their expertise and experience, and what they’re looking at.

Ross:  That’s fair, and I think too that the operators are becoming more thorough. As you add more detail to your plans, there is certainly more to look at when the auditors come in.

Russel:  I think that’s right as well. A couple of key themes, we talked a little bit about differences in interpretation. One of the ones that came up in particular, and primarily I heard this in the AGA Conference, but it was around what is a deviation, and what’s the operator requirement for recording a deviation?

My understanding and this comes from attending some PHMSA presentations back in November 2010 and that timeframe when they were explaining the rule.

As I recall, the explanation of deviation was, if you did anything that was different than what was required by the rule or by your plan, you need to record that as a deviation and explain why it was required.

It raises the question of what is a deviation, because there seems to be some difference in the way that deviation is being interpreted depending on who the auditor is. Have you heard any of that kind of conversation?

Ross:  [laughs] That’s an interesting observation, and certainly I’d like to sit down with that presenter and dig into that a little deeper. What I’ve seen from both the presentations at these conferences, and in talking with other operators, and even in my own interpretation of the rule, you’ve got almost three different types of deviations.

You’ve got a deviation in terms of how you write your policy from what PHMSA’s expectations are, you’ve got a deviation in terms of your operations from what you put in your policies and procedures, and then more specifically, you’ve got an hours of service, a deviation that’s pertinent to your fatigue management program.

Each of those, there’s evidence of requirements around those in the audit protocol. There’s some requirements in terms of how you will document each of those deviations, so I guess in short what I’m hearing is that, deviations are often emergency deviations, but that may not always be the case.

I think that in terms of accounting for them, what’s important is that you’re documenting them in accordance with the requirements of the rule, but that you’re also being aware of how those deviations affect your operating condition, and whether you are able to operate in a safe and effective manner. If not, are you mitigating that?

Russel:  That’s actually one of the things that came up. This conversation that I had at AGA was largely around hours of service deviations, and when are they pipeline safety related? [laughs] That was a really interesting question, do I need to document the deviation?

Do I only need to document it when it is pipeline safety related? If it’s pipeline safety related, when is it pipeline safety related? On the surface, that sounds like an easy question, but like with all other things in this kind of domain, it starts getting real complex once you start unpacking it.

Ross:  That’s right. [laughs] I guess in my mind, which is what it is, but any time you’re operating a pipeline, if you’ve got an increased level of risk, that’s safety related as it relates to a deviation.

Russel:  Yeah, but where do you draw the line? If I identify a deviation, and it’s going to occur in two weeks, and I put in place a fatigue mitigation plan, is that deviation pipeline safety related?

If it is pipeline safety related, when is it? When I exceed the hours of service limitations, is that when it becomes pipeline safety related?

My point is, simply it’s not that clear cut, and I think what that illustrates is that as people are now… most operators have a number of years under their belts operating under these requirements. They’ve developed their own policy, and practice, and approach. What they’re finding is, what they thought they were doing versus what maybe an auditor thinks they ought to do, is not necessarily trued up exactly.

It just adds to complexity, and it comes from people maturing in their understanding and implementation of their own practice to do these things that are required.

I think the other thing on this audit theme that I want to talk about, and I think this is material, what is the state of tools, particularly software tools, that people have implemented to do all these control room management things?

Ross:  [laughs] I think that the state of those tools is about as diverse as the understanding and expectations that we’re seeing and talk about in terms of the rule. The industry has certainly moved in a way that I believe, for the most part, it’s identified that software is the solution.

There’s not that many control rooms anymore in my experience where you have controllers who are making their documentation by hand. With that, there’s opportunity in the market space for folks to create software, but I’ve seen logging being done at SharePoint.

I’ve seen task management being done in group messaging apps, seen logging done in Microsoft Word, or OneNote or something to that effect. There’s certainly some rudimentary practices that it’s still digital documentation, but then I’ve also seen some companies who have developed advanced software for the purpose of CRM compliance.

A lot of times, I think those companies will focus in on one particular area of the CRM Rule. Maybe it’s alarm management, that’s a good example, and then there’s a couple of companies out there of course as we know that have tried to produce a holistic approach to CRM tools from a software perspective.

Russel:  One of the things that I heard in the conversations about the audits is that the people are beginning to get a sense of the effort required to pull the documentation together to be ready for the audit.

One of the things you don’t want to do when the auditor gets there is when they ask for something, send somebody out of the room to go find it. Anybody who’s been through an audit knows the quicker and cleaner you can answer the question, the quicker and cleaner goes the audit.

Time is not your friend in an audit. The quicker it can go the better. That’s not just in a PHMSA audit, that’s any kind of audit.

If you’re quickly finding your records and it just demonstrates you’re on top of it, and I heard from several operators who, preparing for an audit, would put two or three people in a room for two to four weeks in advance of an audit, going through and making sure that all the documentation was where it needed to be, and that they knew how to find it.

That seems burdensome. That’s the word. That seems really burdensome. Are you hearing that kind of thing from some of the operators, as well?

Ross:  I am. It’s interesting. As I’m listening, it kind of reminds me of a story from my first job out of college. I worked for a company that worked in petrochemical plants doing turnarounds, but they also shipped gas. They did fueling services, so they had trucks.

Some of the feedback they got around maintenance for the trucks was that the Texas Highway Patrol was more likely to pull over a truck that was dirty, because they figured if you didn’t take care of the cleanliness of your vehicle, chances are you weren’t taking care of the required maintenance that was taking place.

I think it’s similar in so far as during your audit experience, if an auditor sees that your vehicle is dirty, if you’re really having to scramble to collect a lot of documentation, chances are they’re going to dig deeper.

Russel:  They’re going to make some inference about the state of your compliance and the state of your operations based on that fact. Could be completely false, but they’re still going to make the inference.

Ross:  Right. I think there is a level of documentation, there’s software tools that can be utilized that can, as you use them over time, when it comes time to do an audit, you won’t have to scramble. You won’t have to put two or three people in a room.

You’ll just be able to pull those records up in a way that’s quick and efficient, and demonstrates, using the metaphor, that you’ve kept your vehicle clean, and that you’re committed to compliance.

Russel:  Certainly I think the challenge here is integration because, as I said earlier, all the operators have something. They may have their logbook, and SharePoint, or Excel, or OneNote in their SCADA system.

They might have their forms and records of compliance in SharePoint or organized on a file server someplace. What you tend to see, though, is four or five different software tools that are used to address the different aspects of the rule.

I’ll have an alarm management tool, I’ll something for logbook, they’ll have something for documentation, they’ll have something for point-to-point recordkeeping. They’ll have something for documenting training that might be in their LMS, they’ll have something else for documenting hours of service, might be in their timekeeping system.

It’s all these different tools, and that brings to the forefront the issue of integration. The ideal solution is all of that in a single database, so when you need to pull the records, you can pull them quickly and easily.

Ross:  Absolutely.

Russel:  If you were going to make a recommendation to operators with regards to their control room management tools, what would you recommend? This is your opportunity to do a shameless plug.

Ross:  [laughs] Certainly, I think at a foundational level, you’re going to want to look for something that’s easy to use, something that’s a one-stop shop, that’s holistic in terms of its approach to CRM compliance.

Nobody wants additional passwords they have to memorize by any means, but even more importantly, you have to consider that your controller’s job is to ensure the safe operations of the pipeline.

If they’re having to flip back and forth to either gather data in either an abnormal or an emergency operating condition, or it’s difficult to go find and effectively log actions or even find older logs that can help to make decisions in the present, that’s something that you’re going to want to avoid.

Working for EnerSys Corporation, our POEMS, Pipeline Operations Excellence Management System, has taken that holistic approach to the control room in terms of compliance and control room management, so I’m a big fan of it.

I’ve been a big fan of it. It’s one of the reasons I joined the company. I’m a big fan of where it’s going. I’m not sure if I’m allowed to talk about it, but the task management MOC process is one that we’re hearing a lot of folks are struggling with.

In terms of integration, that’s difficult because you’re having to integrate not just software, but also people and teams.

To have a tool that can link those teams together, the control room and the field, the control room and management, so on and so forth, and then close the loop on required tasks in a way that brings the control room to the center of operations, that’s something that really excites me.

That’s one of several tools that…Obviously we have several tools now, but that’s one that we look forward to adding, and hopefully we’ll round out a package that makes everyone’s job a whole lot easier and makes those audits go a whole lot smoother.

Russel:  I think ultimately, that’s what you’re trying to do. You’re trying to simplify the workflow for the people that are actually operating the pipeline. The fewer tools they have to deal with and the more common the interface, the easier that is.

Secondly, you’re trying to make sure that the recordkeeping is just the necessary result of the work you would otherwise be doing, so the records just happen as the result of doing the work.

I think those are a couple of key things that have already been accomplished with what EnerSys is doing with its Control Room Management Suite.

Of course the other thing, I’ve been doing software, Ross, for a long time. I’ve been doing software longer than maybe even than you’ve been alive. [laughs] The thing about software is it never ends.

Everything you create creates the need for something else, so it just keeps on keeping on. The trick is to add to capability without adding to complexity or destroying ease of use. That’s one of the reasons I like working with you is, we have a lot of really good challenging conversations in that domain about, what does that look like?

We’ve had this conversation about how your focus is more about compliance, mine is more about operations, but those are just two sides of the same coin if you’re doing it right.

Ross:  That’s exactly right. At the end of the day, it’s always a pleasure for me in working with clients, either around the regulatory requirements but also with the software.

When they get it, when they break through that threshold and are using the software in a way where they’re starting to really see the value, that value compounds because they lean into the software even more, and to know that because they’re using the software, because we also at EnerSys are in this process of continuous improvement, that doesn’t end with the operator.

All of that speaks to the fact that pipelines will be operated in a safe manner not only for the benefit of industry, but also as a means for stewardship for the planet, and for people. I’ll tell you what. That’s great stuff.

Russel:  I’m with you. It’s one of the things I love about our business is, it’s not necessarily well understood in the public domain, but it is really critical to our nation’s success and prosperity and safety and environmental protection.

The things that we’re doing all have an impact in that domain, and that’s a big deal. It’s a reason to get up in the morning and it makes work interesting and compelling. I’m with you on that, for sure.

I think we’ll leave it there. I think that kind of wraps things up. I think I’d like to do the typical thing I do where we try to come up with three key takeaways. I’ll do that and then, Ross, maybe you could kind of make some final comments. Tell me if I got it right or if I missed the mark.

Here’s my three key takeaways for preparing for the audit.

1. To the extent you’re able, consolidate all the records into a single location.
2. To the extent you’re able, make the record production a seamless part of otherwise doing the work.
3. Then lastly, expect that as we go forward, there’s going to be more detail, more diligence, more challenge in getting through the audit cleaning and quickly.

You think I summed it up well?

Ross:  I think that’s excellent. I think, just kind of as a word of encouragement, the sooner you start, the easier it is. Certainly, I’m always happy to have a conversation with anyone who’s inquiring as to what those three key takeaways may look like in regards to their specific operations.

Russel:  Yeah, and I think that’s a key point. Every operator is unique, every operator is different. That’s for valid reasons, given the nature of their operation, the nature of their facilities, the nature of the areas they’re operating in. Everybody has to figure out how to make this stuff work for themselves, specifically. Yeah, that’s very true.

Ross, thanks for coming back to the Pipeliners Podcast. I’m sure you’ll be back again as we discover more or as we get new rules coming out and think we need to talk about them. Thanks again.

Ross:  Absolutely, I’ll keep my ear to the ground. It’s a pleasure as always.

Russel:  I hope you enjoyed this week’s episode of The Pipeliners Podcast, and our conversation with Ross Adams.

Just a reminder before you go, you should register to win our customized Pipeliners Podcast YETI Tumbler. Simply visit pipelinepodcastnetwork.com/win to enter yourself in the drawing.

Likewise, if you would like to support the show, the best way to do that is to go to your podcast app on your smart device and leave a review. We would really appreciate it.

[background music]

Russel:  If you have ideas, questions or topics you would be interested in, please let us know in the Contact Us page of pipelinepodcastnetwork.com, or reach out to me on LinkedIn.

Thanks for listening, I’ll talk to you next week.

Transcription by CastingWords