This week’s Pipeliners Podcast episode features Ross Adams of EnerSys Corporation discussing the challenges and opportunities presented in a PHMSA control room audit.

In this episode, you will learn about the PHMSA audit process and how to maintain a posture of audit readiness. Russel and Ross also provide specifics on what can and cannot be controlled in an audit, the trend of PHMSA going deeper with their audit questions, the opportunity to utilize tools to streamline the audit, and how interdepartmental collaboration can help.

Control Room Audit: Show Notes, Links, and Insider Terms

• Ross Adams is the General Manager of EnerSys Corporation. Connect with Ross on LinkedIn.
  • Listen to Ross’ previous appearances on the Pipeliners Podcast.
  • EnerSys Corporation is the current sponsor of the Pipeliners Podcast. Find out more about how EnerSys supports the pipeline control room through compliance, audit readiness, and control room management through the POEMS Control Room Management (CRM Suite) software suite.
• AGA (American Gas Association) represents companies delivering natural gas safely, reliably, and in an environmentally responsible way to help improve the quality of life for their customers every day. AGA’s mission is to provide clear value to its membership and serve as the indispensable, leading voice and facilitator on its behalf in promoting the safe, reliable, and efficient delivery of natural gas to homes and businesses across the nation.
• PHMSA (Pipeline and Hazardous Materials Safety Administration) is responsible for providing pipeline safety oversight through regulatory rulemaking, NTSB recommendations, and other important functions to protect people and the environment through the safe transportation of energy and other hazardous materials.
  • PHMSA’s inspection and enforcement department promulgates safety and training standards by working to ensure that PHMSA-regulated operators comply with regulations and meet the expectations for safe, reliable, and environmentally sound operation of their facilities.
• The CRM Rule (Control Room Management Rule as defined by 49 CFR Parts 192 and 195) introduced by PHMSA provides regulations and guidelines for control room managers to safely operate a pipeline. PHMSA’s pipeline safety regulations prescribe safety requirements for controllers, control rooms, and SCADA systems used to remotely monitor and control pipeline operations.
  • Access Rev. 3 of PHMSA Inspection Guidance for Control Room Management Inspections.
  • Access the referenced Powerpoint presentations from Byron Coy of PHMSA.
• Control Room Management is regulated by PHMSA under 49 CFR Parts 192 and 195 for the transport of gas and hazardous liquid pipelines, respectively. PHMSA’s pipeline safety regulations prescribe safety requirements for controllers, control rooms, and SCADA systems used to remotely monitor and control pipeline operations.
  • Alarm management is the process of managing the alarming system in a pipeline operation by documenting the alarm rationalization process, assisting controller alarm response, and generating alarm reports that comply with the CRM Rule for control room management.
• CRMP (Control Room Management Plan) captures the policies and procedures that are to be followed in the control room to ensure the safe operations of pipeline assets. Operators that are subject to the PHMSA CRM Rule are required to have a CRMP.
  • Learn more about the requirements to review and update your operation’s CRMP.
• HMI (Human Machine Interface) is the user interface that connects an operator to the controller in pipeline operations. High-performance HMI is the next level of taking available data and presenting it as information that is helpful to the controller to understand the present and future activity in the pipeline.
  • HMI Philosophy dictates the policies regarding the high-performance HMI style and design.
• SCADA (Supervisory Control and Data Acquisition) is a system of software and technology that allows pipeliners to control processes locally or at a remote location. SCADA breaks down into two key functions: supervisory control and data acquisition. Included is managing the field, communication, and control room technology components that send and receive valuable data, allowing users to respond to the data.
• Operations and Maintenance Manuals (O&M) is a manual of written procedures for conducting operations and maintenance activities and for emergency response. These manuals are required to be prepared and followed by 49 CFR 192.605.
• “Terri’s Team” refers to Marketing Refresh, founded by Terri Hoffman. Marketing Refresh is a Houston-based Digital Marketing agency that uses digital marketing to improve results and drive more business for its clients.

Control Room Audit: Full Episode Transcript

Russel Treat: Welcome to the Pipeliners Podcast, episode 209, sponsored by EnerSys Corporation, providers of POEMS, the Pipeline Operations Excellence Management System, compliance and operations software for the pipeline control center to address Control Room Management, SCADA, and audit readiness. Find out more about POEMS at EnerSysCorp.com.

[music]

Announcer:  The Pipeliners Podcast, where professionals, Bubba geeks, and industry insiders share their knowledge and experience about technology, projects, and pipeline operations. Now your host, Russel Treat.

Russel:  Thanks for listening to the Pipeliners Podcast. I appreciate you taking the time. To show the appreciation, we give away a customized YETI tumbler to one listener every episode. This week, our winner is Shea St.Cyr with Marathon Pipeline. Congratulations, Shea. Your YETI is on its way. To learn how you can win this signature prize, stick around till the end of the episode.

This week, Ross Adams with EnerSys Corporation returns to talk about the challenges and opportunities related to the control room audit.

Hey, Ross. Welcome back again to the Pipeliners Podcast.

Ross Adams:  Hey. Good to be here. The last time I saw you, we were at AGA.

Russel:  Having fun in Orlando being out in the marketplace seeing people. It was cool.

Ross:  No doubt. You might hear a little bit more pep in my step. I’ve been thankfully racking up some of those frequent flier miles lately, and I’m very grateful that we’re able to get out and about, and see customers, and see a little bit more of the world other than the four walls of the home office.

Russel:  Exactly. Listen, I asked you on to talk about what you’re finding as you’re out and about, talking to people that are going through the audit. Maybe a good place to start is I’ll just ask you, what are you seeing as the Control Room Management audit challenges these days?

Ross:  I guess just a little bit of context. Obviously, in my role, working for EnerSys, I do a fair amount with control rooms, helping them establish Control Room Management programs, and then implement those programs. Of course, when those programs come through an audit, we get the call to come support those programs and the teams. And to speak to your question, I got called up recently.

Russel:  Like “Ghostbusters,” right? Audit busters, who’re you going to call?

Ross:  Exactly, yeah.

Russel:  [laughs]

Ross:  In fact, I’m going to make a note, we need to work on a theme song.

Russel:  [laughs]

Ross:  Terri’s group can get something together for us. That’d be great.

Russel:  I like it.

Ross:  I like the jingle, for sure. To get us back on point, I did get called up to support a customer through an audit process. It’s interesting because we’ve been in this game for a while, and certainly, there were ways that we were suggesting people prepare. The time that we had with the AGA gas control committee is always exceptional.

What I picked up on from that group was that the way that we have been encouraging people to prepare, and we’ll get into that a little bit, having an additional level of rigor and needing systems, and all these things, was really the need that they were experiencing and communicating. Then to go and sit through those audits for a week, I think it was a real affirmation for us that this approach we’ve been talking about and trying to communicate to others is really what makes a difference in the audit experience.

I look forward to talking with you about that today.

Russel:  One of the things I always wondered about and one of the things I think it’s always important to talk about when you’re talking about these audits is there’s two domains of things. There’s those things you can control and those things you can’t control.

It’s probably a good idea to start off and asking, what are the things that you can’t control? I think this is probably true of any kind of audit throughout the spectrum of those things you can audit. What are those things you can’t control?

Ross:  It’s a great question. Establishing what you cannot control helps a little bit with the audit preparedness in that it helps you focus your skill of work and keeps you on an even keel and not stressing out about things that are maybe outside of your domain.

I think, and we heard this, and we have been hearing this, not just through the folks that we connect with in the committee space, but elsewhere as well, our customer base, different courses that we’re engaged in, and what have you, but we keep hearing feedback around the differences and experience as it relates to the audit teams. The PHMSA audits may be a little bit like a box of chocolates, and that’s not a knock on them to any degree. But there’s a variation between the different regions and the different kind of backgrounds that these auditors have. 

I think that the move that PHMSA made to really create some specialized subject matter experts as it relates to controller management and get them geared up and educated and equipped has been really helpful. In fact, in the experience that we had recently, the audit team that we were working with was exceptional, very constructive, really had a mind for supporting industry’s understanding of PHMSA’s expectations, but that’s certainly not always the case.

You have to prepare for the worst in that domain and hope for the best. If the auditor’s one thing, I think another thing that really is outside of our control but is manageable is what happened before we were put in charge.

If you’re the control room manager, you’ve been a year on the job, and PHMSA’s asking you for five years worth of records — which they’re prone to do these days, take note of that — it’s hard for you to be able to control what happened in those four years before you. Understanding that but still having a plan of drawing a line in the sand implementing a new plan and sticking to that plan is helpful.

Then, perhaps, a third point here — and one we’re seeing often — is I think it’s almost a trend in industry for there to be silos within certain organizations. There are things that the control room is going to be asked in a CRM audit, and we’ll touch on this probably in more depth as we have our conversation today.

Things that are outside the control rooms control your documentation, program systems, etc., that are going to get brought up in an audit, where the control room really needs the support of some other departments. That can prove to offer some complexities as well.

Russel:  What are the things you can control?

Ross:  I think you can control your own preparation and approach. If you’ve been there a year, that year is yours. Certainly, you get the notice from PHMSA that PHMSA is going to come for a visit. The time between that notice and when they show up is all opportunity, really, for you and for your team to adequately prepare.

Really what that requires is a fair amount of things. One, first, really, don’t panic. There is a playbook for success. I think systems are a big part of that. You and I spoke to that, Russel, when we talked about Natural Compliance last time I came on board.

The real focus is on making sure that your policy is well established as a first step. Doing a gap analysis is a great way to ensure that your policy accounts for all the expectations of the PHMSA rule.

One of the things I picked up on this that I think has been helpful for us is if you’re going to do an analysis in your policy, you really want to make sure that your foundation is set. Go back and use the old CRM-only inspection questions. I think is Rev. 3. They’re on the PHMSA website still. Those have all of those additional bullet points we call considerations. Those will really help you get a very thorough understanding of PHMSA’s expectations and your response.

You can control your policy. You can control your understanding of your gaps. You can control your approach to implementation. You can control the development of systems and processes and the addition of software, whatever it is that helps you to make sure that the policy is being implemented properly, and there are no gaps or limited gaps or gaps at least that are manageable.

Maybe one last point, since we were talking about things you cannot control, and we talked about silos. For things that you can control, especially as it relates to the audit week or maybe even the weeks before as you’re prepping people, but having members of other teams on standby that are well-coached, understanding the context for CRM, readily available, that’s something you can control that will add a lot of value for your audit experience as well.

Russel:  I can’t add much to that. You summed it up pretty well. You can’t control who the auditor is. You can’t control their attitude or approach or what they are going to talk about. You can’t control what happened before you showed up with responsibility.

Another thing you didn’t talk about, and this is always something I think is big in this domain is you can’t control what you do not know that you do not know. You can mitigate that by doing a gap analysis and working through the plans, and doing all that thorough stuff, but sometimes things just come at you from left field that you’ve never considered.

That happens. You can’t control that. To some degree, your comment about those things that go outside the control room, that seems to be one of the biggest challenges because you as a control room manager or the department getting audited may not be able to control that, but the auditor expects you to have control over that. That’s one of those areas that can get challenging.

Ross:  That’s certainly the case. I think, too, that if you have people within your organization that have been through audits before, and maybe not even a CRM audit, it would be worth your time to sit down with them and get an understanding for what that person’s audit experience was.

These audits largely function in a similar fashion. Auditors of one variety aren’t that different from auditors of another. We had some folks in the room that managed some other safety programs that were there just to get the audit experience so that when it came time for their turn, they would have gotten a little bit of an understanding for maybe the PHMSA approach and an auditor’s approach.

Bringing a consultant [laughs] may help as well. Someone that’s seen multiple audits and can help you ask questions that you might not have thought of…

Russel:  I might advise that if you’re going to do that, you should do that well ahead of the audit. [laughs]

Ross:  At least a month, probably two depending on where you sit. Somebody that can just see your blind spots and help you reach out beyond those is always good regardless of the area of focus.

Russel:  What is PHMSA seeming to be focused on in terms of your experience and conversations with others? What are they doing that’s new or different that people ought to be aware of?

Ross:  We’ve been saying it for a while. You’ve seen the trends. You go look at the annual Powerpoints that Byron Coy puts out on the PHMSA web page. It’s true that PHMSA’s area of focus is going deeper. They are more thorough. I talked about how they are training people to be specifically CRM inspectors.

There’s a momentum towards depth and thoroughness that we’ve been speaking to that’s certainly coming to fruition. I mentioned that one of the ways to counteract that is using those CRM-only inspection questions, making sure that your policy is accounting for all the questions, all the considerations.

Now, it’s important to note that because that was published in 2012, there are some items that are newer that have to be addressed outside that context, like team training, roles & responsibility of others, etc. That’s a helpful approach. I mentioned, too, they are looking back further.

Traditionally, with CRM, you got to set your own document retention timeframe. Recently, that hasn’t held up as well as what we’re hearing. Auditors are looking back five years for proof of implementation and compliant implementation.

Something else of note is that they are treating your controller management plans differently. We saw this early on, people would just copy and paste the rule or the FAQ, and that would be their document. We’ve moved past that. Largely people have acquired information and knowledge of how they operate and how what PHMSA’s expectations are. Even beyond that, is where we’re getting to, which is it there an expectation that your CRMP is a document for the controllers. First off, educational.

Then, two, and this was the language that the auditor used is, “Hey, whether it’s your policy or your procedure if you got hit by a bus, are your documents such that when someone else takes over, they could implement your program up to the same level of compliance?” We looked at each other and say, “Well, we don’t really intend to get hit by a bus anytime soon.” Although maybe that’d be a little more palatable than a PHMSA audit, but regardless the point stands. The procedures, especially, what we do is we create a CRMP, and we put the procedures as appendices, and we make reference to them.

We do that so that as you cycle them you can just pull the one page and replace it as opposed to the whole document. Those procedures can’t be three or four lines. They’ve got to be pretty specific. Who’s doing what? How do we know that it got properly? Is there documentation that has to be created? Where is it going?

I remember we got caught because we created a new procedure and forgot to put a reference in one of the sections of the CRMP to the procedure. Again, they are looking for a tight ship as it relates to your approach to policy, your approach to procedure, and making sure that everything that you put in there is replicable.

Russel:  That’s absolutely right. I always tend to take an aviation model to all this. I guess, probably because I spent some time in the Air Force, and I know probably more about aviation than I care to. Those policies and procedures in aviation are extremely tight. They are extremely thorough.

You get trained in detail on all that stuff. Part of what’s going on because the FAA and PHMSA are both part of DOT is there is some information sharing and in particular, the PHMSA looks at the control room as an aviation-like or cockpit-like thing. That’s where they’re driving us. I’ve been saying that that’s where we’re headed for years, and I think that’s starting to manifest.

Ross:  A couple of other things I thought were interesting. They were not just focusing on procedures, but also on the effectiveness reviews. A lot of people ask often, how do we address those? We have an approach and one of the things that certainly came up is PHMSA is not just looking for a qualitative approach. Certainly, they call that out.

You have to have some, excuse me, a quantitative approach, rather. They do call out the need to quantify certain things, but they want a qualitative approach as well. For example, if all you do to prove alarm program effectiveness is demonstrate a change in your alarm activation KPIs, that would probably not be sufficient.

There’s a qualitative requirement now, and that could be solved for via control surveys or gap analyses on your alarm philosophy, any number of things. That’s something to pay attention to as well.

Then we’ve talked a little bit about silos. I would tell you that in this audit experience, we looked at alarm philosophies, HMI philosophies in style guides. We looked at emergency response plans and procedures. We looked at owning manuals, and probably two or three other documents, all of which were created by, owned, and managed by individuals and departments outside of the control room.

I think a real message here… every control room manager who’s listening to this is going to take this and wave this through the halls in their office. But, Control Room Management is clearly going beyond the confines of the control room now in the way that auditors are approaching it, but also just in terms of how to implement an effective Control Room Management program that adds the operational effectiveness as well as the focus on safe operation.

For example, making sure that when an abnormal or emergency operating condition is detected in the field that there is in the O&M manual a need to contact the control room, that kind of a thing is often overlooked. It’s in the CRM regulation, but the folks that are writing the O&M manuals aren’t reading the CRM regulations.

That takes a little bit of collaboration and an awareness by other departments in the company to facilitate not just the expectations of PHMSA for CRM, but really an effective, safe, proactive approach to pipeline operations.

Russel:  I think this conversation about qualitative versus quantitative analysis and going beyond the control room, that’s one of those things that I think both of those are probably in their infancy. I would expect that as we move forward and as the Control Room Management rule matures, we’re going to see the auditors pushing more into those kinds of questions and those kinds of things.

Ross:  It’d be interesting, too, an arena that I’m largely ignorant in, but I know there’s a lot of folks that are doing a lot around predictive analytics in the alarm space and measuring to see how we think about effectiveness, not just from a reactive standpoint, but from a proactive standpoint as well.

Russel:  This also goes to the whole pipeline safety management conversation because that’s also looking for qualitative analysis. The idea of collecting information from the controllers and asking them a series of questions about the go-to, the effectiveness of the alarm management program, that could be really informative and really valuable, right? And it’s not necessarily quantitative.

Ross:  Yeah. That’s a great point. I don’t think I’d connected those dots just yet. It’s a good point.

Russel:  I give that to you for free. [laughs] What should the pipeline control room be doing about this reality, the increasing level of detail, the five years of records, and so forth? What should the controller be doing to react or deal with these realities?

Ross:  The easiest answer is what we spoke to you last time we got together, which is developing these systems that promote natural compliance. A byproduct of that is ongoing audit readiness.

If you’re doing the work that you’d normally be doing and the outcome of that is documentation records that you’re storing a central location is readily available, then you don’t have near as much of a gap to close a once that phone call comes through.

If it does come through then there’s a process for preparation that will benefit you. Obviously, there are quality consultants out there that can support this effort and support the identification of blind spots.

One of the mottos that we had, mantras, for this last audit and the work that we do with this customer was “No surprises.” It’s like we may not be perfect, but we’re not going to have any surprises. We really worked hard to be thorough in our preparation in our analysis, so that we were at least aware of and mitigate as much of what we needed to as we could.

That process — I think I touched on it early on — gap analysis, understanding where your gaps are, policy revisions, making sure your framework and your foundation is set, and then what you’ll actually see if you did another compliance review or a gap analysis after you had done an update to your policy is that your implementation scores and gaps will probably get worse before they get better. I wasn’t expecting that. [laughs] I don’t know that we thought that all the way through.

If you can think about it, if your policy used to be a little bit more ambiguous and now it’s a little more refined, and your expectations are a little more refined, when you go to score your implementation, you’ve got to score your implementation a little bit tougher in relation to your policy, so something to be prepared for, but then create the systems to make sure that your policies are up to par, and then do another compliance review somewhere down the line.

That may be a six month or a year process or even longer depending on how much time you have to be able to put in all this, but I think doing that and then potentially even doing as part of your annual program review, maybe an annual mock out will help you be perpetually audit-ready and then making sure your team is on board, too.

Just writing it all down well doesn’t get the team on board. There’s a lot of training, a lot of cultural shifts, focusing on the little things, and inspecting what you expect is all helpful for controller managers who are in this boat.

Russel:  I would say that to the extent, you have the ability to take your Control Room Management program and review 1/12th of every month, so that every year, you go through it, and that becomes just a rigor of your annual cycle can have a really big impact on being perpetually ready.

The other thing I would say is train, train, train, train, train, train, train. I’m not talking about all the training that’s required under the rule. I’m talking more about train the people in the plan, in the procedures, in the systems and how they work, so they understand not only what they’re doing but why they’re doing it and how that supports a process of getting better.

That’s really critical because you never know who’s going to be the person, who’s ends up running point, or ends up in the conversations with the auditors.

Ross:  That’s right.

Russel:  The auditors have the ability to say, “Where’s your list of controllers? Give me these three guys I got questions for.” They have the ability to do that, right?

Ross:  Yeah. That’s right. One of the things we did, just keep in mind, in this particular business case, with the CRMP, we’d done a rewrite and so is relatively new.

As part of the audit prep, I always spend time with all of the controllers to make sure that they’ve gone through whatever questions we would expect them to be asked so if they do get that call-up, it’s not the first time they’ve heard that question. They have the answer.

What we found as we were trying to do this training is that they were guessing at more than a few things. That’s no discredit to them. Actually, they want wanted to be right. They wanted to have the right answer. I thought they did a great job, but what it told me is that with this newer policy and procedure, we hadn’t effectively gotten them up to speed and developed a sense of ownership with them.

That was on us for us to do. I think that speaks to what you’re saying. Certainly, you want them to be all over it. Breaking it up, as you said, over the course of a year is a great way to help them focus in and develop that sense of ownership.

Russel:  Yeah, absolutely. What do you think that all pipeliners should know about this? When I say all pipeliners, what I mean is all those people who aren’t in the control room, what should they know about the control room audit?

Ross:  I think there’s a few things. I think…kind of look at this as it relates to the other departments. We’ve spoken to how PHMSA in the control room context is branching out and asking questions of other departments, well within the scope of the rule, but there are expectations for certain policy and implementation performance on other groups.

Whether it’s the field, SCADA especially. Getting those folks on board, having them build an awareness, or even just, if you work in one of those rooms, going up to your control room saying, “Hey, look, we heard the podcast and how can we help? What do we need to know?” That’s helpful.

I think, too, what we’re seeing is Control Room Management is here, and it’s here to stay, and the regulations have grown over time, PHMSA has gotten more thorough. We’re seeing more inspections, more funds, more focus.

From a midstream management perspective, there are tools, resources, systems that your control room manager needs to be effective and really to help them focus on the day-to-day of moving the product while still being operationally effective, and compliant, and safe.

A conversation perhaps with your control room manager, “Hey, what do you need to be successful?” is one that would benefit your organization. Again, just based on the trends we’re seeing coming from PHMSA, that’s certainly top of mind.

Russel:  I think my answer to that question would be, just to try to hit the high points. One is the audits are getting more thorough and they’ll probably continue to move in that direction. Think about policy, procedure, records, and analysis driving reports and change. Get that plan, do, check, act cycle working in the control room.

All the people who impact the control room and the control room’s ability to be successful, the SCADA support, the field automation, the field communications, the HMI developers, the O&M guys, and anybody who has to interact with the control room and would be covered under team training. All those folks really need to be aware that they’re part of that process as well. They don’t get to punt it, if you know what I mean.

Ross:  Yeah, absolutely.

Russel:  Listen, man. This has been fun. You got anything you want to add before we sign off and pour the first cocktail on a Friday evening?

Ross:  No. It’s back to the trenches for me. There are more audits coming and more work to be done. I look forward to it. We really enjoy being able to support operators in the space. It’s always a learning experience.

I appreciate you having me on. I hope this was helpful for your listeners. If they have any follow-up questions, I’m always open to more conversations.

Russel:  Absolutely. They can, of course, find you through the Pipeliners Podcast website. As always, we’ll put this all up on a web page on the site with show notes so people, if they want to go and read this and take notes, can help themselves.

Ross:  Sounds great. Thanks, Russel. Thanks for having me. Best of luck to you.

Russel:  All right. Take care, man.

Ross:  Goodbye.

Russel:  I hope you enjoyed this week’s episode of the Pipeliners Podcast and our conversation with Ross. Just a reminder before you go, you should register to win our customized Pipeliners Podcast YETI tumbler. Simply visit pipelinerspodcast.com/win to enter yourself in the drawing.

If you’d like to support the podcast, please leave us a review on Apple Podcast, Google Play, or on your smart device podcast app. You could find instructions at pipelinerspodcast.com.

Russel:  If you have ideas, questions, or topics you’d be interested in, please let me know on the Contact Us page at pipelinerspodcast.com or reach out to me on LinkedIn. 

Thanks for listening. I’ll talk to you next week.

Transcription by CastingWords