Russel Treat talks with Vickie Cain, who chaired the American Petroleum Institute committee that led the update to API RP 1168, the recommended practice for pipeline control room management. Vickie, who spent about thirty years as a pipeline operator before moving into consulting, walks through what changed in the new edition, including point-to-point verification, control room applicability, director/supersede procedures, backup control room testing, and shift turnover, and explains why each change matters to operators.

Updated API RP 1168 for Pipeline Control Room Management Show Notes, Links, and Insider Terms

• API RP 1168: The American Petroleum Institute recommended practice for pipeline control room management. It covers personnel roles and responsibilities, shift turnover, adequate information, fatigue management, change management, training, and controller workload. This episode covers the update to a new edition.
• API RP 1167: The API recommended practice for pipeline SCADA alarm management, referenced in the episode as the next standard up for a work group Russel Treat plans to join.
• API RP 1165: The API recommended practice for pipeline SCADA displays, referenced in the discussion of point-to-point verification and prior standards work.
• PHMSA: The Pipeline and Hazardous Materials Safety Administration, the federal regulator that enforces the Control Room Management rule and participates as an observer in API standards development to keep new standards aligned with existing regulation.
• Control Room Management (CRM) rule: The federal pipeline safety rule addressing human factors and control room operations, discussed throughout the episode as the regulatory backdrop for API RP 1168.
• Point-to-point verification: The process of confirming that field instrumentation and the corresponding SCADA display accurately match, discussed as one of the most expensive and highest-risk compliance areas for operators.
• Control room determination/applicability: The process of evaluating whether a given facility, console, or field office actually meets the regulatory definition of a control room, a new area of guidance added in the updated standard.
• Director/supersede: A control room practice where someone other than the assigned controller directs or overrides a controller’s actions. The episode discusses new recommended procedures covering roles, qualifications, and documentation for this practice.
• Shift turnover (shift handover): The exchange of operational context between outgoing and incoming controllers, expanded in the new standard to include any events, not just third-party incidents, that could affect operations.
• SCADA: Supervisory control and data acquisition, the system controllers use to monitor and operate a pipeline remotely.
• Flowstate: This episode’s sponsor, a provider of pipeline leak detection software designed to help operators detect anomalies, reduce false alarms, and improve response.

Updated API RP 1168 for Pipeline Control Room Management Full Episode Transcript

Announcer: Welcome to the Pipeliners Podcast, episode 449, sponsored by Flowstate, providers of pipeline leak detection software built to remove barriers and empower operators. Flowstate helps teams detect anomalies, reduce false alarms, and improve response with practical tools designed for diverse pipeline operations. Learn more about Flowstate at flowstatesolutions.ai. The Pipeliners Podcast, where professionals, Bubba geeks, and industry insiders share their knowledge and experience about technology, projects, and pipeline operations. And now your host, Russel Treat.

Russel Treat: Thanks for listening to the Pipeliners Podcast. I appreciate you taking the time, and to show that appreciation, we give away a customized Yeti tumbler to one listener every episode. This week, our winner is Michael Abar with TC Energy. I hope I got that last name right, Michael. If I didn’t, just feel free to correct me. Anyways, congratulations, Michael. Your Yeti is on its way. To learn how you can win this prize, stick around till the end of the episode.

This week we speak with Vickie Cain, who recently led the API RP 1168 update for pipeline control room management. Hey Vickie, welcome to the Pipeliners Podcast.

Vickie Cain: Thanks, Russel.

Russel Treat: It is great to have you on. For the listeners, Vickie and I have known each other for, well, longer than either of us care to admit. I’ll just say it that way.

Vickie Cain: Perfect. That’s a common way I say that.

Russel Treat: Hey, would you do me a favor? Would you tell me a little bit about your background and what you’re doing now, and how you got involved with 1168?

Vickie Cain: Sure. I have been in the industry about thirty years now. I came into the control room in 2013, shortly after a lot of the regs came into place, which is, I think, about the time that we met, Russel. I’ve been in several different aspects of the industry, from transmission pipelines to midstream and gathering, to an LNG startup facility. With 1168, I came in helping as an operator. I was chairing the API control room committee and stepped up to actually volunteer, because 1168 was up for reaffirmation, and the committee thought that maybe we should go ahead and take another look at it. Maybe some things had changed since the initial version, edition one.

Russel Treat: Yeah. Awesome. So what drove the need to do a new version of 1168?

Vickie Cain: I think operators really just felt that there had been a lot of, not changes to the regulation, but changes to the understanding of the regulation, based on inspections and maybe the inspectors learning more about what the control rooms were doing and how we were doing it. So we really took a look at what we’d learned over the last ten, fifteen years, and how we could improve, not just for ourselves, but for new operators too.

Russel Treat: Yeah. So you and I have both been involved with control room management pretty much since its inception. The rule was announced in 2007, published in 2010, and became fully effective in 2012. So we’ve got, call it fifteen years living in the CRM reality. What would you say is different about our understanding of CRM, and where the inspectors are at, versus where we were when we started?

Vickie Cain: I think overall, if you think about what precipitated the rule coming into place, there have been a few incidents. There had been some concern by the government wanting some regulation placed. And if you look at what our regulation is for control room management, it’s only a page and a half in the register. There’s a lot of FAQs that go along with it, a lot of things that are recommendations or incorporated by reference. But overall, it’s been a lot of just asking questions of other operators, asking questions of the regulators, and some of the findings we’ve found have gotten much more strict recently, through inspections or through other incidents that have happened since the rule came into place.

Russel Treat: Yeah, I agree with all that. I would say that, like a lot of rules, they tend to mature over time, because the operators and their practices mature, and their regulators and inspectors mature. They have a better understanding of what they’re looking for.

Vickie Cain: Yeah, I don’t know, I think I’m a little muddled sitting here talking with you, it’s been a busy day, sorry. But I think that we as operators have a much better understanding of CRM than we had even five years ago. When the rule first came out, I probably did forty presentations to various operators and at various events, just talking about, well, look, this is beyond what’s in the regulation, which you’re right, it’s a page and a half, there’s not a lot there. It’s like, what’s driving this? What are we really trying to say? And that was a whole conversation about human factors engineering and situational awareness, and what is good alarm practice, and why does that matter, and what is good organizational practice, and why does that matter. And I think where we are now is operators have a much better understanding, particularly the larger operators, of what they really need to be doing. A lot of them are working on this really hard. And Russel, I didn’t say earlier, but I’ve transitioned now from operator to consulting, much like you, where your business is. So we get to see this from different aspects now, from new operators to those that have been doing it a long time.

But even the understanding between those different spaces, I feel like operators have put a lot of focus on one specific section of the rule. Like, I think we all do fatigue management pretty well, everyone figured that out fairly easily, the rule was much more clear on that, and there’s a lot of recommendations. But when you look at things like point-to-point verification, there was no real guidance given on that, or if there was, it was vague enough that some operators would have questions.

Russel Treat: Well, what I would say about the guidance that was given on point-to-point is it was pretty clear what the expectation was. But for a lot of operators, it just wasn’t practical.

Vickie Cain: That is true. That is true.

Russel Treat: Right. And that’s the kind of thing you hope an API standard would close the gap on.

Vickie Cain: So, it was an inspection about a year ago, and the inspector said, “Do you have a whiteboard?” And we said, “Yeah, behind you.” And she said, “Can I have a marker?” And we’re digging around, because, you know, in a different space, markers never seem to work. So we gave her a marker, and she immediately started writing up exactly what she wanted to see on a point-to-point.

Russel Treat: Oh my.

Vickie Cain: Yes. So again, it’s not codified, it’s not anything, it’s just what they’re expecting to see. It’s kind of talked about in the FAQs, but it’s not really elaborated.

Right, so what we did when we were discussing 1168 was I brought that up and said, hey, here’s what the inspectors are wanting to see. I had taken a picture with my phone and was able to share that across the room, but someone else said, “Oh, I have that, I have that pretty much same picture.” And it was a different inspector, which tells you the inspectors are talking about what they want to see.

Russel Treat: So building that into the training.

Vickie Cain: Absolutely. So we were able to put some of that into what was practical for operators. We put it into the new 1168.

Russel Treat: Interesting. So yeah, that’s a good tee-up for my next question. So there’s clearly some stuff in there about point-to-point. What else is in there? Actually, before I ask that, let’s unpack the point-to-point thing a little bit. What did y’all come up with as recommendations for point-to-point in the new version of 1168?

Vickie Cain: So there is an FAQ, don’t ask me to recite it off the top of my head right now, but one of the things we wanted to do was clarify that a little, especially for the newer operators, or folks that wanted to revamp their point-to-point process. So things like location, the dates and names of everyone involved, not just your SCADA tech or your controller who was checking it, but also that field employee who was physically out in the field, and what those field parameters are versus your SCADA displays. They wanted to see that one-to-one ratio of what you were seeing in the field truly matching what you were seeing in your SCADA system, because some folks were only recording one. And then the results of the test, was it a pass or fail? I know as an operator, in the past, that wasn’t something we did. We only recorded the ones that passed, and it’s like, how do you have proof of that?

Russel Treat: Right. Yeah, okay, that’s a really typical type of inspector question.

Vickie Cain: Absolutely.

Russel Treat: Right, what about all the records about when it didn’t pass?

Vickie Cain: Yeah.

Russel Treat: So maybe you put it into your procedures that you only record those that pass, right?

Vickie Cain: So yeah.

Russel Treat: Interesting, interesting. Well, I think point-to-point is one of those things that’s extraordinarily expensive to do.

Vickie Cain: Yes.

Russel Treat: It’s an issue that, if you don’t do it right, causes all kinds of problems operationally, because of all kinds of risks. In my experience, things not being mapped correctly, or not presenting correctly, while rare, it’s not nearly rare enough.

Vickie Cain: Yes. That’s probably the way to say that. It’s like, well, I’ve got a thousand points here and two of them are bad.

Russel Treat: Yeah, but do those two matter? Right? They all matter to a controller when you’re giving them that information. It’s the reason it falls under adequate information, you want them to have the best information to make good decisions.

Vickie Cain: Well, that’s certainly true.

Russel Treat: I think the other thing that’s also true is a controller will trust until they have a reason not to trust. And when they have a reason not to trust, they don’t trust anything.

Vickie Cain: Absolutely. And that makes sense, right, because they’re not necessarily trained in, and shouldn’t necessarily be trained in, all the details about how that data gets from the field to their screen. They just have to rely on it being right, that it’s good and it’s what they need.

Russel Treat: Yeah. So what other things did you address in the 1168 update?

Vickie Cain: One of the items we actually came back to later in discussion was control room determination and applicability. I think back in 2010 to 2012, when everyone was designing their control room management plans, they listed where they had control rooms, primary and backup, and said, “Here are our control rooms.” And we’ve found that’s not what the inspectors are wanting to see, or probably what operators need to be doing. So we designed some questions, based on feedback from different operators, to walk an operator through determining whether a control room really was a control room or not.

Russel Treat: Okay, I want to dig into this one a little bit, because now that you’re a consultant, you get into this conversation way more than you want to, right?

Vickie Cain: Yes, yes.

Russel Treat: So when we say control room in the context of 1168, are we talking about whether or not it’s a control room, as in, is it a control room, capital C, capital R?

Vickie Cain: So I like to call this the control room with a capital C, capital R. It’s the determination of what PHMSA defines as a control room. It’s really gone a lot further than just a qualified controller sitting at a screen using a keyboard and mouse. There’s that whole dispatching question, there’s a lot of other things, are you inside a fence or outside a fence, are you making decisions based on the information you’re seeing in SCADA. So those are some of the questions that walk someone through answering, to give them the information to put on their form that an inspector might be wanting to see.

Russel Treat: Do you have any idea why the inspectors want to see that? It’s not a records question, which is interesting, because it’s a procedural question, but they ask for a record nearly every time, a record of the determination.

Vickie Cain: I don’t understand that either, I’m sure there’s a valid reason, I just don’t understand what it is. I think inspectors want operators to look at some things that aren’t traditionally considered a control room, but could potentially be. Russel, I’m sure you’ve seen it too, that field office that has access to SCADA, or has its own HMI, controlling a valve that’s only five miles away, or whatever, that’s local.

Russel Treat: Yeah, local wants to consider it local.

Vickie Cain: Yeah, so this is to help operators through that, and to decide where they want that control to be. Can I give you some heartburn, Russel?

Russel Treat: No, no. Well, maybe. Vickie can see me because we have our cameras on, we can see each other as we’re talking here, so she’s seeing the look on my face, and my wheels are turning as I think about this.

When you start getting into this conversation about gathering versus not gathering, and all the new things that kind of got swallowed up in the PHMSA regulation that would have historically been considered gathering, but PHMSA said, “Yeah, but that’s a 2,500 psi, thirty-inch line, that shouldn’t be treated like a low-pressure gathering line, that looks more like a transmission line,” there’s a lot of places, particularly in the shale plays, where there are questions about relatively short runs of pipe that basically just go between two points, a compressor station and an extraction plant, with really nothing on it other than those two points. And those things are typically controlled at the extraction plant. So it feels like PHMSA is teeing something up to dig into.

Vickie Cain: So the other thing, again, this isn’t part of the API discussion, but keep it in mind, there were some inspectors who wanted to take things back clear to the PLC, to determine whether people were in a control room at the local PLC level. And we really wanted to get their focus off of that with these questions.

Russel Treat: Oh, that makes sense.

Vickie Cain: And that would go to maybe an inspector who’s more junior and doesn’t really understand what they’re looking at, or you’d be surprised.

Russel Treat: Oh really?

Vickie Cain: Yeah, I’m sure you would.

Russel Treat: I don’t want to be surprised. Okay, so control room applicability, getting some guidance in place, that makes sense why that became an issue. This is kind of the boundaries of what we as an industry consider a control room. And really, if you’re following the API 1168 recommendations, again, this isn’t incorporated by reference, at least this section, but it should start giving some guidance to operators. Did PHMSA participate in the writing of 1168?

Vickie Cain: They actually did. They had someone in the room the whole time.

Russel Treat: Yeah. And just to be clear, because I’ve had this experience when I participated with 1165, the person from PHMSA is not there to tell industry what to do, and isn’t there to tell them what the agency’s going to do given that standard. They’re there to try to make sure that you’re not doing something that causes problems down the road, if we want to later incorporate that standard by reference.

Vickie Cain: Right. And they wouldn’t give interpretation or anything, though there were a few times they’d say, “You might want to look at this,” or “This doesn’t specifically line up with our definition here,” or “This isn’t in line with an FAQ.” So they really kept us, as operators and consultants, in line with what’s currently adopted.

Russel Treat: Yeah, no, and that’s a very important role they play.

Vickie Cain: Absolutely. It’s very resourceful to the committee, because it helps you stay on the rails and move things forward without creating new questions. We had an example about testing alarms during point-to-points. Honestly, I don’t know where the recommendation came from for people not to test them, that they could test them later, once it was in production, or something, and I kept getting pulled back, the PHMSA representative kept pulling us back to, “Here’s what it says, here’s what we expect from you.”

Russel Treat: Yeah, so that’s one of those areas where the regulator expectation and the operator history and practice create friction, for sure.

Vickie Cain: Yes, right.

Russel Treat: What else did you guys look at in 1168 and revise? This is juicy stuff, man.

Vickie Cain: Director/supersede. It kind of came out a little bit, a lot of focus came out after the regulation came into play. It actually recommended that operators proceduralize what they’re doing for director/supersede. Many operators just say it’s not going to happen in our control room, or maybe only a supervisor, or only a qualified supervisor, can do it. So what this workgroup did was we recommended what should be in your procedures if you allow director/supersede, things like what the roles and responsibilities are of that person who can direct or supersede, what specific actions they can take and when they can take them, what qualifications they need, do they need to be qualified on that specific system or just general abnormal operations, that kind of thing. And then, most importantly, the documentation each operator is going to require if director/supersede happens.

Russel Treat: Okay, yeah, I’ve spent a lot of time in rooms having that conversation. I will tell you, I’m not a fan of director/supersede. I feel that we put controllers in the seats for a reason, but that’s my personal opinion.

Vickie Cain: You know, I tend to agree with you. I think it adds all kinds of complexity that’s not helpful from an operating perspective.

Russel Treat: Yes. And what you really need to do is be cognizant of what you’re asking people in the chair to do, and give them, make sure they’re properly trained, properly qualified, and properly supported to be able to make the decisions they need to make.

Vickie Cain: Absolutely. I was just getting ready to say that. And the other thing is the whole conversation about technical actions, because they say director/supersede technical action. So telling a controller “we need to reduce pressure here” is not a technical action. Telling a controller to move a valve to a specific position, that’s a technical action.

Russel Treat: Yeah, I used to have this conversation all the time. It’s one thing to make recommendations, it’s another to step in and say, “You shall do it this way.”

Vickie Cain: Right, well, it’s one thing to say, “This is the operating outcome I need.” It’s another thing to tell somebody how to get there.

Russel Treat: Correct. And once again, it all goes back to training and the tools we’ve given the controller.

Vickie Cain: Yeah, well, and not only that training, but those who might be in a position to direct or supersede.

Russel Treat: Right, right. But I’m with you, if I’m working with a customer, I try to avoid that.

Vickie Cain: Yeah, you can’t always, it depends on the nature of the operation and their staffing and all that kind of stuff, but yeah, try to avoid it for sure.

Russel Treat: What else?

Vickie Cain: A small thing, we in 1168 originally had talk about how to test a backup SCADA system, but not much detail on testing your backup control room. So we added some new recommendations there, what you should test when you go, and how you should document it. Because, once again, I think when the regulation took place, maybe not everyone was worried about having a backup control room, but I think operators have really relied on their control room to take over that remote operation, and not have as much staffing elsewhere. So I think it’s important for control rooms to have that backup, and we should be testing them.

Russel Treat: Well, yeah, I think one of the big things we’ve seen in the last fifteen years is operations has moved from the field to the control room, and that’s a good thing. Having a centralized point of operations is a good thing. That doesn’t mean we don’t still have people in the field doing all the things they do, including moving valves, it’s just that we’re centralizing operations, and that’s a good thing.

Vickie Cain: Well, and they begin to rely on the controllers, so the controllers should have those backup control rooms. And just a reminder to operators, if you have more than one, make sure you test more than one of them.

Russel Treat: Test them all, right.

Vickie Cain: Yes, exactly.

Russel Treat: What else?

Vickie Cain: I can only really think of one other thing, and it’s a small change, but it is incorporated by reference for at least 1195. During your shift turnover, previously it stated you should review third-party incidents, and we hope our third parties aren’t having incidents every day, so it needed to be a little more broad, is what the group thought. So instead, they changed it to any events, including third-party incidents, that will directly or indirectly affect your operations. So things like, if they have an outage downstream or upstream of you, you should be sharing that at shift turnover. Again, controllers are doing that, so we just thought we’d put it in there as a reminder.

Russel Treat: Yeah, no, I think controllers do a pretty good job of that as a matter of practice. Those guys and ladies that run those consoles, they know what they need to know about that operation, and they do a pretty good job of handing that off. I think that’s another area where we, as an industry, have improved orders of magnitude from where we were when the rule first came in.

Vickie Cain: Russel, I always get passionate about shift turnover. You can ask anyone.

Russel Treat: Oh, I remember, when you were one of our customers, we were in the early days of our CRM tools, having some really detailed conversations, hours of conversation, about shift handover.

Vickie Cain: But to me, it’s one of the biggest places for error, because someone would have forgotten something, or communicated it poorly, or just been tired at the end of a shift. So I think especially newer operators having that reminder of what they should be sharing during shift turnover makes it a lot better.

Russel Treat: I’ll tell you, we spent way too long talking about where the word “turnover” came from. Is it shift handover, is it shift turnover, you know how these rabbit holes go. It’s like, I don’t care what you call it, it’s when this guy leaves and this guy comes in, how do they communicate operating context. Give it a short name, but understand that’s what we’re talking about.

So Vickie, what would you tell operators about the new standard? I mean, this new standard is not incorporated by reference, but PHMSA has had their opportunity to put their eyes on it and provide input in the process. So what would you tell operators about this, and what recommendations would you give them?

Vickie Cain: So what I would say is, it’s not huge changes for anyone. Operators weighed in on this very heavily, so it’s probably what a lot of existing operators are already doing, although I can tell you I came away with notes from items other folks were talking about. It has gone through final editorial and should be published shortly. It was voted on, and there were no nays on it, which was a good thing, I guess. This was my first experience leading the workgroup, so I’d just say, take a quick read of it, see if there’s anything you could actually learn from it. What do you have going on that you’re struggling with, there may be some new recommendations. Just remember, if you’re a liquids operator, sections five and seven in the previous version are incorporated by reference. I don’t expect them to change that very quickly, although operators have made it clear they would really like to see those references updated, because this is the third edition, and we’re still referring to the first.

Russel Treat: Yeah, well, I know there’s a lot starting to come out from PHMSA, they’re filling up the regulatory agenda. A lot of it is procedural and administrative stuff, and this kind of fits in that category. So, brand new standard, is it published yet, or is it going to be?

Vickie Cain: I think it should be published, if I would guess, it would be probably by the first of June, based on the timeline we were given. By the time this episode airs, it should be published.

Russel Treat: Perfect. So that means it might not get caught in this legislative session, but it might get caught in the next one.

Vickie Cain: Yeah, that’s really what the group expected.

Russel Treat: All right, well listen, this has been awesome, we should have this conversation more often, I mean, I love talking control room, these are my peeps.

Vickie Cain: Mine too, Russel. Just a quick reminder that API RP 1167 is forming a workgroup right now, make sure your name was on it this time, Russel.

Russel Treat: So I’m good, and I had a few others from a recent conference that I asked to be added to it, so I would recommend operators get involved in this, because again, it’s not a huge time commitment, a few hours here and there, but it’s the discussions that happen in that room that sometimes really expand your knowledge.

Vickie Cain: Yeah, that’s one of the things I’m really so grateful to API for, being a sponsor and organizing these conversations, because anybody who’s worked with API standards, you can sit down and read this stuff and understand what the words say, but if you weren’t involved in the conversation that got the words on the paper, you don’t really understand what the standard is trying to accomplish. And one of the great things about this is, through these kinds of conversations, we’re actually able to capture that for people, because not everybody can participate in that stuff.

Russel Treat: Right, so it’s great to have you on, and, well, we need to do it again. Are you going to participate in 1167?

Vickie Cain: I will participate in 1167, I’m not sure who’s going to lead that group, to be honest with you.

Russel Treat: Yeah. So I have to ask you one last question. How do you like being a consultant versus on the operator side?

Vickie Cain: You know, it’s still an adjustment most days. I still catch myself in that space in the control room, which maybe makes me better at what I do now. But I love seeing how different operators tackle the same regulation, the same code, and how they’re all doing it a little differently. And I also like being able to recommend something I’ve seen with one operator, having that broader view now.

Russel Treat: Yeah, no, I think that’s one of the great things about being on this side, you get to see a lot of stuff. I have no idea how many different control rooms I’ve seen, but it’s a big number, and I’m always ready to go see another one. I love going into control rooms, I have, you could spend eight hours telling me about your control room, and I’ll learn more in ten minutes of walking around than I will in eight hours of conversation.

Vickie Cain: We spent a lot of time in 1168 talking about different aspects of it, about what happens with acquisitions or divestitures, because that’s one of the things PHMSA was focusing on. And I always would tell people, when they were on a team looking at an acquisition, this isn’t my operator days talking, but get into the control room, you will have the heartbeat of the company in the control room. You’ll know where there are issues, you’ll know where the system is working great, you’ll know all the best rumors.

Russel Treat: Oh yeah, yeah, yeah, yeah, that’s right. All you gotta do is find the console that’s running the asset you’re buying and sit there for a couple of hours, and you’re going to know more than you’ll know in weeks of due diligence.

Vickie Cain: Absolutely.

Russel Treat: So true, so true. Well, listen, this has been great. Vickie, I’m so glad to finally get you on the podcast. I really appreciate your time, and I look forward to working with you on 1167.

Vickie Cain: Okay. Thanks for the opportunity, Russel.

Russel Treat: I hope you enjoyed this week’s episode of the Pipeliners Podcast and our conversation with Vickie. Just a reminder, you should register to win our customized Pipeliners Podcast Yeti tumbler. Just visit pipelinepodcastnetwork.com/win and enter yourself in the drawing. If you have ideas, questions, or topics you’d be interested in, please let me know, either on the Contact Us page at pipelinepodcastnetwork.com, or reach out to me on LinkedIn. Thanks for listening. I’ll talk to you next week.