Would you like to know how a DOT-compliant control room is built from scratch? Do you want to understand the latest details of what should be included in a control room to comply with the CRM Rule?

In this episode of the Pipeliners Podcast, host Russel Treat welcomes Matt McClure of Oasis Midstream to discuss his experience standing up control rooms. This discussion covers the full spectrum of what goes into a new control room, including furniture, rest areas, SCADA systems, High Performance HMI, and more topics of interest.

Russel and Matt also discuss the CRM Rule for control room management, including the considerations that go into training controllers to operate in a pipeline control room, ensuring safety and compliance following the Marshall Incident.

DOT-Compliant Pipeline Control Room Show Notes, Links, and Insider Terms

• Matt McClure is a Control Room Manager at Oasis Midstream. Find and connect with Matt on LinkedIn.
• A DOT-compliant pipeline control room adheres to the CRM Rule introduced by PHMSA. The CRM Rule provides regulations and guidelines for control room managers to safely operate a pipeline.
• High-Performance HMI extends the capabilities of SCADA in pipeline operations and complies with the ISA 101 requirement by providing an HMI philosophy, style guide, and design guide.
• API 1165 refers to the Recommended Practice for Pipeline SCADA Displays. This standard outlines the best practices for designing and implementing displays that are used by controllers to evaluate information available in all operating conditions.
• The Bhopal Disaster was the 1984 gas leak incident at a pesticide plant that is considered the worst global industrial disaster. Contributors to the incident included poor maintenance of the facility and a leak from a storage tank.
• The Texas City Disaster was the 2005 refinery explosion that occurred when a raffinate splitter tower overfilled, releasing flammable hydrocarbons into the environment. Two key issues were operator fatigue and an improper handoff from the night operator to the day supervisor.
• The Chernobyl Disaster was the 1986 nuclear incident in the former Soviet Union that was caused by a nuclear reactor exploding, causing the deaths of thousands of people. The explosion occurred during a late-night safety test.
• Operator Qualification Training (OQ Training) refers to a process of training control room decision-makers who have the authority to act in normal, abnormal, and emergency situations. Read the Operator Qualification Overview published by PHMSA.
• PHMSA released new FAQs in January 2018 clarifying the Roles and Responsibilities and Team Training aspects of the CRM Rule. (Listen to Ross Adams discuss team training in the previous Pipeliners Podcast episode.)
• Soured gas refers natural gas that contains high concentrations of hydrogen sulfide (H2S). An H2S alarm is utilized at facilities that monitor the concentration levels of hydrogen sulfide.
• The Marshall Incident refers to the Enbridge Incorporated Hazardous Liquid Rupture and Release, which occurred on July 25, 2010, in Marshall, Michigan. Read the full NTSB Accident Report.

DOT-Compliant Pipeline Control Room Full Episode Transcript

[music]

[announcer intro]

Russel Treat:  Welcome to the “Pipeliners Podcast,” episode nine. Thanks for listening to the Pipeliners Podcast. We appreciate you taking the time to listen to this episode.

To show our appreciation, we want to let you know about our signature prize pack. We are offering a free customized YETI tumbler to one listener each episode. How do you register to win? Simply visit pipelinepodcastnetwork.com/win, that’s pipelinepodcastnetwork.com slash W-I-N, to enter yourself in the drawing. That is our way of saying thank you for being part of the show.

We’re very fortunate to have with us today Matt McClure. Matt is a control room manager for Oasis Midstream. He has the unique experience of standing up a couple of control rooms from scratch, everything from building out the facilities, building out the room, hiring staff, the whole thing soup to nuts. He’s going to join us today and talk to us about what does it take to get a control in place.

With that, let me introduce Matt McClure. Matt, welcome to the Pipeliners Podcast.

Matt McClure:  Thanks for having me.

Russel:  Welcome aboard. I asked you to join us on the Pipeliners Podcast because I believe the listeners would be interested in just what all’s involved in standing up a pipeline control room. I know you have that expertise and experience.

I guess the first question I’d like to ask is just what are all the aspects? What are all the various pieces and parts of getting a DOT-compliant pipeline control room in place?

Matt:  There’s lots of things to think about whenever you’re starting one up. First, mainly you have to find your room. When you’re building out the control center, you have to think about obviously current needs for consoles, whether it’s one, two, or however many.

Then, also if you’re planning on doing any growth, taking that into consideration as well, you don’t want to outgrow your room too fast and have to rebuild another room.

Russel:  Matt, is there anything unique about setting up a room where people are going to be there 24/7? We’re not really architects, but we have some experience working with customers, and coaching them through the process of making all the decisions that are unique to 24/7 operations.

Matt:  You don’t want it to be a cramped room. The guys can’t be on top of each other. If you have more than one, that’ll help with getting rid of background noise. If you have two people on the phone at the same time, it’ll gives them a little space just to get away from each other, and have their own space without interfering or distracting one another.

Russel:  One of the things that I think gets forgotten often is the pipeline controllers have a limited amount of time that they can be away from the console. Sometimes, they’re limited to as few as 15 minutes.

That causes some unique considerations around access to the restroom, and access to be able to prepare a meal. There’s other considerations, like dedicated temperature control, and in particular lighting seems to be a big deal, and adjustable lighting in the room. It’s a big factor, as that relates to fatigue.

I’ve seen control rooms where they have windows to the outside. Generally, those are a distraction. Would you agree with that?

Matt:  Yes, I would. I know if you ask the controllers, they would like to have that. When you have windows, you get glare on the screens. It’s harder to control the lighting. There is a recommended range for the amount of lighting that’s in the room.

It’s a lot harder to control the level of lighting. Then like I said, you have to deal with glare on the screens, glare through the window, things of that nature. Also, with the room, you don’t want it to be in a place where there’s a lot of foot traffic, a place where people have to go through the room to get to another place.

You want it to be, almost only the reason you’re going in the control room is to go to the control room. That’s why a lot of places, they’re self contained, like you said. There’s bathrooms, a kitchen or kitchenettes, all within the control room, where the guys don’t even have to leave their little space to do what they need to do for the day.

Russel:  We’ve talked about windows to the outside, but I think another consider is this, that at least in our experience, if a company spends the time and effort to put in a nice control room, a lot of times, they want to show it off, or they may need an area where they’re doing emergency response, where they have a place they can marshal people to make decisions, and they need a view into the control room.

What are your thoughts about a window with a view into the control room?

Matt:  I find the window actually reduces foot traffic. That way, if managers, investors, or whoever it may be want to see the control center, they can stay at arm’s length, and not provide a distraction to the controllers that are on the console.

Most of the time, the window is behind the controller, so they don’t necessarily even know people are watching them. I find that the windows into the control room are a benefit, as opposed to if there wasn’t a window, you would increase the foot traffic in the control room. That would increase the distractions as well.

Russel:  Another aspect of this, I think, is if you’re going to have a view into the control room, you might need to think about what are the people who are looking in the control room looking for.

That can have an impact on what you’re putting on the walls, in terms of displays, information, and so forth that may not necessarily be simply for the people in the control room, but might provide information to others who have an interest.

Matt:  The window does allow managers, engineers, or whoever may want to just walk by and check on things to look through the window, see the screens, get the information they need to get. That keeps them out of the control room as well, with distractions, asking them questions, and taking their mind off of watching the assets, and making sure they’re operated efficiently and safely.

Some places have larger screens that are dedicated to presenting that kind of information. People can walk by at a glance, and see how everything is operating.

Russel:  This brings up the whole conversation about, when you’re putting in a control room, it’s not really just about the people who are at the console, and operating the pipeline. It’s also about those people who need to interact.

When you’re putting that control room in place, you might need to be thinking about who’s going to need to interact with the controllers, how do we put the facility together in a way that facilitates that, at the same time, minimizes distraction, and streamlines or makes that process efficient?

Maybe this is a good time to segue. We can talk a little bit about furniture. I know that you’ve worked with people to design custom furniture for your control rooms. What do you think are the key considers when doing that?

Matt:  The ease of the controller to be able to have access to manuals and procedures. Both sets of consoles that I’ve had built had cabinets or shelving, so binders and stuff of that nature can be stored at the console. They don’t have to get up and walk across the room to find that stuff.

That also allows procedures to be more readily available during times of need. It helps to have a quicker reaction time. The sit/stand desks are a great way to help the guys stay alert during their shift. They can stand up if they’re feeling fatigued, and it helps with the ergonomics.

If you have a taller controller, they can raise the desk a little bit to keep the strain off their bodies, sitting at the console up to 12 hours at a time. The sit/stand desks are, in my opinion, important. Just as important as the console themselves is the chairs that they sit in.

If you go with a cheap office chair, two things are going to happen. One, they’re going to wear out very quickly because of the 24 hour use. Secondly, they’re going to be very uncomfortable for the controllers to sit in.

Getting a 24-hour rated, heavy duty chair is very important. In the long run, it’s almost a wash, with the amount of chairs you would go through of the cheap ones. The big ones won’t break as much.

Russel:  I think that’s a really interesting point. One of the things that is difficult for those that haven’t worked in a 24/7 environment is to wrap your brain around, if you will, the reality that you’ve got somebody who is confined to a very small space, and typically these shifts run 12 hours at a stretch.

They’re not allowed to be away from the console for a very long time, and then there’s more than one person who’s using the same console, because the console’s staffed 24/7. That’s just a lot of time to be in the same spot.

Matt:  The first thing that I think is of utmost importance is having a good SCADA system. There are two different ideals about the SCADA system. There’s the high performance HMI, which is grayscale, or there is leaning towards 1168, where there’s a little more colors, up to 11.

Having a SCADA system that is easy for the controllers to use and navigate, and doesn’t overload them with too much information at once, only presenting the information that they need for normal operations is critical for an efficient control center.

Russel:  I know that we could talk about furniture and screens for a long time, but maybe this is a good point to transition and talk about what other things do you think are key considerations.

Matt:  High performance HMI, and the API being updated.

Russel:  Right, so we did a whole episode recently with Jeremy Coleman, who’s a control center manager for a utility in the Northwest, and talked specifically about his experience with putting in the high performance HMI.

One of the points that Jeremy made, which I thought was very insightful was that you know when the HMI is good, when the controllers are using it. If you give them something new or different, and they don’t use it, then you’ve missed the mark.

Matt:  Absolutely. If the SCADA system is not standardized across the board, if it’s just thrown together, it causes confusion. The wrong information is possibly being used to figure out problems. It’s just a snowball effect whenever it’s not utilized properly by the controllers.

Russel:  The other point that Jeremy made is that to really optimize an HMI requires experimentation, and that’s kind of an ongoing effort. You’re quite likely to do something and have it not work, maybe a couple or even a few times before you find the thing that actually does work. I thought that was a very interesting point that Jeremy made, and like I said, we did a whole episode on that recently.

With regards to the standard, it’s actually kind of interesting. API 1165 is a fairly old standard, particularly when you consider the amount of change and the rate at which technology and knowledge is advancing in this whole aspect of a high performance HMI. API 1165 is soon to be getting updated and made more current.

The key thing about the SCADA system is having a toolset that allows you to create components and implement those consistently across an enterprise. Depending on the tools you pick as your platform that can be relatively straightforward or extremely difficult.

The other thing I think that’s important to understand about SCADA systems is often it’s very easy to make a change in animation, change a color usage, change a font. Those changes, to make them technically, is relatively a small level of effort, but the fact that those changes are often easy to make kind of gets in the way of implementing a standard HMI across the entire system.

Let’s keep going down this path, and exploring what all the different things are. What are some of the other things that need to be considered when putting up a control room?

Matt:  Implementing your control room management plan, and all the parts associated with that. The roles and responsibilities, alarm management, your HMI philosophy, training or just some parts of that that need to be created and implemented, and trained on to have a PHMSA or DOT-compliant pipeline control room.

Russel:  What’s the hardest part of all of this in your experience?

Matt:  In my experience, the hardest part has been the alarm management, and I think that’s been echoed throughout the industry, because if you don’t get a good grasp on it from the get go, it’s really hard to catch up with it as you’re adding systems.

If you don’t have your template and a good policy, and buy-in from all of your other colleagues to the SCADA group — engineers, operations — it’s hard to get the alarms where they need to be, and you’re trying to paddle up a creek once you start up without a good alarm management plan.

Russel:  I think you’re making a great point, Matt. Let’s table the alarm management conversation a little bit, because I want to dive a little deeper into the discussion about the control room management plan.

Is there anything other than alarm management that’s in the control room management plan that’s challenging to implement, in your experience?

Matt:  Just having the tools available to you for the documentation, with all the monthly reviews that you have to do, the yearly reviews, the training associated with it.

Just having the tools available to keep everything organized and making the process efficient, documenting point to points in Excel spreadsheet, and then having to put those in a binder for three to five years, it’s very hard to keep organized and understand what you’re doing if you don’t have correct tools to help you while you’re doing this.

Russel:  Can I make a shameless plug here and ask you what tools you’re using?

Matt:  We are using the POEMS software, the second installation that I’ve used, with the point manager system, and now the workload analysis tool, FatigueMgr. It makes it a lot easier to keep up with all your records. It keeps them in one place, organized electronically, easy to search for. That is a very helpful tool with keeping compliant with the CRM rule.

Russel:  I appreciate that brief commercial message. Thank you. We talked about the room, the SCADA system, the control room management plan. What comes next?

Matt:  After you have all this in place, then you have to hire your people. You’re looking at a minimum of five if your supervisor or manager is going to be considered qualified to be a controller, six if you do not want your manager to be a qualified controller, per console, actually.

The fifth person helps with overtime, keeping your hours of service in check. It’s not recommended to exceed certain hours of service in certain windows, so having the extra person available to lessen the workload on the controllers is necessary.

Russel:  Some of the hours of service limitations that are specified in the rule are things like an opportunity for eight hours of sleep once you complete a shift, no more than 14 hours in a single shift, no more than 65 hours in any sliding seven days.

When you think about that, and then you start considering time off, sick time, and things of those natures, it’s pretty much impossible to properly staff a control room with just four people.

Matt:  Absolutely, and even if your supervisor or manager is the qualified controller, it reduces their ability to do their job when they’re having to cover day shifts, work on a weekend.

Even further, to talk about the ability to get eight hours of sleep, you don’t just get off work, drive home, and go to bed. You go home, you get a bite to eat. Maybe then you go to sleep. You have to wake up, take a shower, eat breakfast, so really, it’s 14 hours, depending on travel time, is almost pushing your ability to get 8 hours of continuous sleep that they recommend.

Russel:  This hours of limitation thing, and how that impacts fatigue, and how fatigue impacts decision making is really a big deal. I know it’s certainly one of the things that the regulators take a very strong interest in, is how are you managing hours of service?

If you look at some of the major events that have occurred, like Bhopal in India, or the Texas City disaster, and so forth, Chernobyl’s the same way. Many of these events occurred at the end of a rotation in the early hours of the morning when people are at their most fatigued.

Matt:  Absolutely, and with the fatigue, even outside of the incidents, you don’t get as good shift relief. There’s other fatigue issues as well as even the guys falling asleep on the way home if they’ve been worked too hard, and their quality of life in general is affected.

Russel:  Exactly. In fact, I think oftentimes one of the most dangerous things that people do around the control room is drive home after an overnight shift at the end of a rotation.

What’s the biggest challenge in hiring? If you’re starting from scratch, and you’re building up a new team, what are you looking for in the people that you’re hiring?

Matt:  This is where they buy-in for the control room comes to be important, because if you don’t have capable controllers, depending on your assets, you’re opening yourself up to a higher liability. If you’re trying to penny pinch salaries for controllers, you’re not going to be able to get the experience that you need to monitor and operate your assets effectively.

If management has bought in that experienced controllers are necessary, that makes the hiring process a lot easier to go through, because hiring four green guys off the street that have never sat in a console, it’s a steep learning curve for those kinds of people to get acclimated to running a system, like I said, especially depending on the complexity of your assets that you have.

Russel:  I think you make a really good point. I’ve actually never thought of this before, but if you’re starting up a control room from scratch, the level of expertise and experience that you need in that initial team is actually higher, because you’re going to have to develop policy and procedure, and develop understanding about how to operate those assets. To try to do that with people who are learning, man, that establishes a pretty high hurdle, if you will.

Then after you have been at it for a while, and maybe you’re adding a second console, then you have the opportunity to maybe bring people in that are maybe not quite as experienced, and where you can train them and put them under the supervision of some of the people who are more experienced and have more knowledge of the particular system.

Matt:  Yes. Without having people already there to train lesser experienced people, they almost have to be hired on at a lot earlier stage in the development of the control room so they can get more training.

Then maybe you’re looking at purchasing training software on such as a simulator so they can get some board experience before you put them on your active system, the more experienced people that are going to pick it up faster than, say, someone that just graduated from school would do.

Russel:  The training part of the control room management rule is actually fairly comprehensive, and in fact, there’s some new training requirements that have been put in place just this last year, I think applies with those new training rules went fully into effect in January of this year.

Specifically, the new training requirement is what’s being called team training, and it’s the idea that, “I need to train others that work with the control room to do that effectively, and I need to train that as a team. I also need to train people who, in specific situations, can direct the pipeline controllers.”

Matt:  Yeah. Putting together our team training, we were looking at going with mainly an OQ based team training, and then assigning it to the people that we deemed necessary for the training.

PHMSA just came out with their FAQs, and they specifically stated that OQs only are not sufficient for your team training program, so you’re going to have to spend a lot more time with your training program now, so if your guys haven’t been in this type of atmosphere before, they’re not going to understand the concepts that are being taught in team training as well.

Russel:  My experience is training is like a lot of other things. It’s easy until you know enough about it. This raises, to me, an interesting question, and that is how difficult is it to train others that work with the control room, and in particular, how do you get their attention?

Matt:  It’s very difficult to train people how to interact with the control center. Just for example, the marketing group. They are always changing their NOMs, or get behind schedule, so it’s hard to keep them from contacting your controllers directly, because they may see a small part of a big picture, and they may not understand that there’s other issues going on that’s causing that.

Trying to get them on board with going through the proper channels such as the manager or supervisor to get the information to the controllers is vital to keeping your controllers’ distractions down as well as limiting the amount of people that are trying to tell them how to do their job, especially people that are not qualified or knowledgeable with operations of the system.

Russel:  Right. One of the things we’ve certainly seen in our experience is that there’s kind of this propensity, if you will, that because the control room is there, and because it’s a 24/7 operation, that things that need to happen after hours or on the weekends just tend to get loaded up on the control room without really understanding what their core job is.

One of the things that I think is true is that oftentimes the pipeline controller’s job actually gets more challenging after hours, because people that they need to get information from or talk to, they’ve got to call people and wake them up.

Oftentimes I think that the job of the controller after hours is more demanding than the job of the controller during regular hours. Of course, that can be different depending on any particular pipeline and its operation, how it’s staffed, and so forth.

Given your experience standing up a couple of control rooms, what would you do different if you were starting again from scratch?

Matt:  We can dive back into the alarm management on this one. That’s been my biggest headache with both control rooms. The first one was taking assets that are already in service and trying to do alarm management on those on the fly, as well as adding a lot of assets at the same time.

It’s a snowball that you can’t get out ahead of and stop if you don’t have — starting off — your templates set and the buy-in from the necessary parties to get it going.

It’s also a totally different mindset when you start talking about alarms and alerts to people that are not familiar with the CRM plans.

For the longest time, an alarm was the same thing for everybody, for everything, so the people that are not familiar with the alarm management in the CRM standards, it’s harder for them — in their mind — to distinguish between an alert and an alarm, and what actions or lack of actions those two require.

Russel:  I think you hit the nail right on the head there, Matt. This whole alarm management thing, in my experience, is really challenging, because it’s a radical re-think about what alarms are, how they need to be thought about, handled, and so forth.

I also think that when you look at what operations looks for out of alarms, and what pipeline control looks for out of alarms, it’s not necessarily the same thing, and arbitrating that difference can be very challenging.

Matt:  Absolutely. When you’re prioritizing your alarms, the chances, depending on the categories that you want to input — and I’ll throw in another little plug in for POEMS here    for the alarm manager — as you’re setting the different levels inside of your priorities for each setting, say, if you have a safety category, it’s going to be ranked different than if it’s a manned facility and if it’s not a manned facility.

Because the chances of something happening while someone is actually at the site is a lot lower if it’s unmanned than if it’s manned. As you’re going through this alarm management, the tendency is to do worst-case scenario for everything, so your ratio of your alarms gets skewed the wrong direction, whereas you want a low number of critical alarms and a higher number of low priority alarms.

People try to tend to flip it, and do a high number of critical alarms and a low level of low alarms.

Russel:  If you think about it, for any given console in the pipeline control room, there might be a large number of field operators that are working with the equipment in the field and interfacing or impacting what’s going on on that console.

The issue in the control center is to be able to maintain situational awareness and ideally maintain normal operations, where the issue in the field is knowing what’s going on and knowing where I need to be.

I’ll give you one specific example. If you’re dealing with soured gas — and I get a high H2S alarm; that an alarm in the control room might be treated differently than if I’m a field operator — and I’m driving up to the site, and I note that there’s a high H2S alarm. The whole context is different about: what am I going to do with that information?

We certainly have had the same experience in doing projects that involve all of this, and one of the challenges is this kind of constant training that you need to be doing to keep everybody on the same page about what are alarms, what are alerts, how are they different, and why, and how what the control room needs is different than what the people in the field need.

It’s an ongoing effort, and any time you have changeover in the people, you’re kind of starting over from scratch, because this is a relatively new way of thinking in our industry.

Beyond alarm management, what else would you want to do differently?

Matt:  I think from the get-go, explaining to everyone that could possibly be involved with the control center, management, engineering, SCADA, IT, the marketing group. Just getting better buy-in, and a better understanding on their ends of the CRM standards, and why they’ve been put in place.

Maybe give them an example of an incident that caused some of these standards to be implemented, such as the Marshall Incident, so they understand why these are necessary parts of having an efficient and safe control center.

If you have buy in from the get go, everything that has to happen is a lot easier when everybody’s understanding on why you have to do what you have to do. The CRM rule it’s a completely different animal from the rest of the PHMSA requirements.

Russel:  That’s very well said. Certainly, the control room management part of the rule is very different than the other parts of the rule that are related to integrity management, running pigs, and cathodic protection, and that type of thing, which has tended to be the focus up until the control room management room came out.

I say this quite often, a control room rarely, if ever, causes a problem, or causes an issue with the pipeline, but they are always a major factor in how quickly that issue is mitigated. Matt, one of the things I like to do as we’re wrapping up an episode is, I want to try to boil it down to maybe three key takeaways.

(1) In this case, I think the first key takeaway is that standing up a control center is really a much bigger issue than just the facility. Certainly, there’s a lot of facility considerations, but there’s a lot more than that, there’s the systems, there’s the staffing.

Some of the things that often get overlooked is the training and orientation of all the people that are going to be making decisions about the control room, or interacting with the control room. That, I would say, is my takeaway number one from this conversation.

(2) My second takeaway is the alarm management conversation, getting in front of that, and trying to have standards and understanding in place before you put the project in, and implement things in a consistent way.

Then even after you have it in, continue to educate and inform those that are interacting with alarms, and trying to help the whole organization understand what alarm management is, and why it’s important.

(3) Then lastly, I think this whole idea of training is a big deal, that there’s a lot there. There’s a lot that needs to be done. Certainly, the regulators are looking for things that are more like exercises, and less like sitting in front of your computer, and going through some videos.

Those would be my key takeaways. Hopefully, this has been of benefit for our listeners. I’ve certainly learned some things, and gained some insights. It’s always interesting to me that there’s somebody I’ve known and worked with.

We sit down and have one of these conversations, and I’ll always, always learn something new. Hopefully, our listeners did as well. Matt, I want to say thank you for being our guest. We’d certainly like to have you back at some point in the future.

Matt:  Absolutely. Thanks for having me again.

Russel:  I’m excited to announce our latest winner of the YETI tumbler. Congratulations to Adrian Maggard, of Caiman Energy. If you’d like to join the winner’s circle, be sure to go to the website and fill out the form to give yourself a chance to win one of these sweet YETI tumblers.

[background music]

Russel:  Thanks for listening and I’ll talk to you next week.

[announcer outro]

Transcription by CastingWords