This week’s Pipeliners Podcast episode features first-time guest Doug Rothenberg, a pipeline alarm management expert, discussing the importance of situation management.

In this episode, you will learn about situation management, how to improve situation management in your control room through Rothenburg’s book, “Situation Management for Process Control,” and the importance of understanding weak signals as a tool for situation awareness.

Situation Management: Show Notes, Links, and Insider Terms

• Doug Rothenberg is the President and Principal Consultant of D-RoTH, Inc., a technology consulting company providing innovative technology and services for industry. Doug’s specialty is control alarm management training and consulting for the industrial process industries. Find and connect with Doug on LinkedIn.
  • Get Doug Rothenberg’s book, “Situation Management for Process Control,” discussed in the podcast here.
• Situation Management is the competency, ability, and willingness of the human operator to properly and successfully manage the enterprise or activity under his or her charge.
• Alarm Management is the application of human factors along with instrumentation engineering and systems thinking to manage the design of an alarm system to increase its usability.
• Shift Handover is defined as transferring responsibilities and tasks from one individual to another or a work team and it is one of the best-known types of safety-critical communication.
• HMI (Human Machine Interface) is the user interface that connects a controller to the instrumentation and controls in pipeline operations. High-performance HMI is the next level of taking available data and presenting it as relevant information that is helpful to the controller to understand the present and manage future activity in the pipeline.
• Weak Signal provides a tool for operators to detect early or subtle problems in the making.
• Data Leak uses small indications to determine the presence of abnormalities compared to data expectations or other points of reference.
• Artificial Intelligence (AI) helps understand and predict equipment glitches, manage workers, and increase output.

Situation Management: Full Episode Transcript

Russel Treat:  Welcome to the Pipeliners Podcast, episode 101, sponsored by EnerSys Corporation, providers of POEMS, the Pipeline Operations Excellence Management System, SCADA compliance, and operation software for the pipeline control center. Find out more about POEMS at enersyscorp.com.

[background music]

Announcer:  The Pipeliners Podcast, where professionals, Bubba geeks, and industry insiders share their knowledge and experience about technology, projects, and pipeline operations. Now, your host, Russel Treat.

Russel:  Thanks for listening to the Pipeliners Podcast. I appreciate you taking the time, and to show that appreciation, we are giving away a customized YETI tumbler to one listener each episode. This week, our winner is Craig Brown with Magellan Midstream Partners. Congratulations, Craig, your YETI is on its way. To learn how you can win this signature prize pack, stick around until the end of the episode.

This week, we have with us Doug Rothenberg. Doug Rothenberg is the author of the seminal text on alarm management, and has recently released a new work on situation management. Doug and I have had the opportunity to work together on some projects over the years. He’s one of those guys who I consider a guru. Without further introduction, let’s just jump into it.

Doug, welcome to the Pipeliners Podcast.

Doug Rothenberg:  Russel, I am just delighted to be here.

Russel:  I am so excited we’re sitting down to have this conversation because…Well, I get to drive the nerd needle way over in the red zone [laughs] I think in this conversation, particularly in the domain that I have a lot of interest in. Thanks for taking the time, for sitting down with me. I’m looking forward to this.

For the listeners, why don’t you tell the listeners a little bit about your background, where you come from and how you got into this discipline of situation management?

Doug:  I started out as a baby electrical engineer. I figured out I didn’t know to do that well enough. I did advanced studies.  I went on to develop an expertise in systems and automation control. Had some fun teaching young engineers.

I finally realized I needed to go out and do this for myself. I’ve worked for major oil company for a long time. When they took the rug out from under themselves, I started my own business and I never looked back.

Russel:  What year was that when you started your business?

Doug:  1999.

Russel:  ’99?

Doug:  Yeah, goes back. This is the 20th-year anniversary, just got it.

Russel:  Awesome. Congratulations.

Doug:  Thank you. My intent was basically to find new tools that industry wasn’t using or old tools they were using that we didn’t know how to use well. It’s led me on this pathway to alarm management, situation management, and everything in between.

Russel:  I asked you to come in and talk about situation management. I met you because of your work in alarm management, and you’re the guy who taught me what I know about alarm management. I’ve made up a few things since we first got working together.

When I heard you were going to take on a book on situation management, we actually had this conversation. This has been quite the endeavor, I guess it’s what I am getting at. What caused you to want to write a book on situation management? I know it was a challenge. I am curious why it was such a challenge.

Doug:  I need to go back to how we write books. I identified alarm management, basically said, “Hey, this is only part of the solution. What happens to everything else? The operator has to find and we don’t give him the tools of alarm to identify them.”

I began to say, “Okay, let me see if I can figure this out.” I was able to go back and look at all the tools in the control room and find out how they fit together, and they weren’t fitting together so nicely. They all had names. They all were engineered, but they weren’t engineered to fit together, so I started writing the book.

During the process of writing, I discovered a lot of new things that I didn’t realize I was going to stumble across.

Russel:  [laughs] Want some examples of that. I know it’s like any endeavor you undertake to do something new, you’re learning as you’re doing it, right? I am curious what were some of the things you uncovered.

Doug:  That’s the amazing part. Shift handover. I mean the guy comes in, and the other guy leaves by, and we now learn, “Well, maybe we should overlap a little bit,” but we now know that that overlap was without a financial purpose.

We overlap to let the incoming operator understand what he’s going to face in the next shift. We didn’t understand that the incoming operator, after he sat down, was the most troublesome part of the operation for the next 10 minutes.

We needed the leaving operator to stay and they both collaborate on the transfer. It wasn’t enough to give him the information. He had to know what it meant and what he was facing.

Getting back to the book, I started writing the book. After about a year and a half, the book started writing me. I can’t stop. It was an amazing adventure. I uncovered a lot of things. The shift handover was one.

The whole book was developed without a concept of weak signals. I stumbled across it in the middle. I said, “Oh. That’s the missing clue.” I now can give the operator alarms for things we can figure out in advance and weak signals for things he has to notice and process.

Russel:  We’re going to talk about that more here in a bit. The chapter on weak signals and the conversations we have about that are really interesting, really compelling. They offer an opportunity to have a stepwise improvement in safety and operations effectiveness. It’s kind of exciting. The book’s on situation management. The first question is “What is situation management?”

Doug:  Situation management, in the vernacular, is “Hey Mr. Operator or Miss Operator, we’re going to put you in the control room. We’re going to give you this huge job of making sure nothing goes wrong. How do you do that?”

There are really four basic steps. See what’s going on. Identify things that are not quite right. Work on the solution, but that work has to do two things at once. You have to stay in familiar territory. You can’t invent anything. You have to look ahead to see where you’re going. The fourth step is to make sure you got where you’re supposed to go.

Russel:  It’s a little like driving a car. It’s a poor analogy, but we drive a car. So much of that is just automatic. We don’t really bring the process up to like, “What am I actually doing? What’s actually going on?”

I’m looking out the window. I’m looking at the road and the traffic. I’m making decisions about what I need to do based on that. Then I’m taking corrective action. Occasionally, I look down at my console and make sure my machine is operating well.

Maybe I’m doing other things that aren’t actually part of driving the car, like tuning the radio. We’ll go from there. There’s lots of things you can do in a car that are not driving a car.

I like trying to come up with simple analogies. To me, that’s a simplified analogy. There’s a lot of operating a pipeline system that is taken for granted. I know it so well. I know how to do it. I can drive a car. I can run a pipeline like I drive a car, unconsciously, because I know it so well. Is that what you’re doing, is taking that process and breaking it down?

Doug:  That’s an amazing analogy, Russel. The part you mentioned was certainly right. I agree with that. What it evokes is another whole concept in safe driving. It’s called defensive driving.

There are five or six simple rules for defensive driving. One is aim high. Look down the road. Give yourself, always, an out. If you’re doing something, you make sure you can do something else if it doesn’t work.

Russel:  Proper following distance. Time to react.

Doug:  Following distance. If you need to pull off the road, where can you pull off? Make sure other people see you. If you’re making an operational change, it’s not in a vacuum. If somebody else is around who takes your product or you take their product, they need to know what you’re doing.

I can’t remember all the other steps, but these defensive driving concepts work very well when we recast them for the control room. I call it defensive operating.

Russel:  I presume you’ve got all that elaborated in the book.

Doug:  In detail.

[laughter]

Russel:  You can’t remember in your book. I’m not surprised. I’ve got the book. I’m just, when Doug first sent this to me…It’s 700-plus pages. It’s dadgum treatise on situation management. It’s readable, too. That’s the other thing. It’s got lots of cool pictures, which I like. [laughs] Pictures are good for engineers. It helps with understanding.

Why do you think situation management is an important subject to be coming up for process operators?

Doug:  The process industry, again, has all the tools and pieces, parts, but we really didn’t know how they fit together. Once we said, “Okay, Mr. Operator, Miss Operator. You’ve got to do your job,” let’s examine the tools we’re doing the job with. Out pops HMI. We all know we shouldn’t overuse color in the HMI. We should do this. We should do that. We need to group things in the HMI.

Now we have additional concepts. We have to say, “Mr. Operator, when you’re looking at a screen, what do I as the designer of the process need you to look at to be able to find whether or not things are okay or not?” Now I don’t put stuff on the screen that says, “Oh, I need this. I need that.” Why do I need it? Let me put it there.

Russel:  Doug, we could do a whole podcast just on that comment right there. Doing a high-performance HMI is very different than building an HMI. Building an HMI is putting numbers and things on a screen.

Building a high-performance HMI is understanding the process and how it’s operated and looking towards how do we optimize that operation. That’s a very different discipline. At least that’s my experience.

Doug:  It is a different discipline. The more important thing is when you do it right, the results are also very different.

Russel:  That’s right. The results. How are the results different when you’re doing a good job of situation management?

Doug:  When you’re doing a good job of situation management, the first person who knows that is the operator.

Now he feels comfortable walking in on his shift and not saying, “Oh my God. Is this the one that’s going to wreck my reputation?” or “Is this one that I’m going to just be glad when it’s over?” or “Is this one that I can feel proud of being on shift, my shift, and contributing to the operation?”

Russel:  Again, that’s another deep concept. Historically, we haven’t thought about teeing an operator up or what, in the pipeline world, we call a controller, teeing a controller up to not only be successful but to feel good about their success and be able to see and understand the contribution they’re making, why their role matters.

Doug:  There’s an interesting follow on. Most operators finally realize that they don’t know everything. We can’t expect them to know everything.

Russel:  Course not.

Doug:  We ask him, “Okay, let’s know the limit of what you know. Let’s put in place a process of collaboration, delegation, and escalation so that when you get into trouble, before you get into trouble, when you sense there’s something going amiss, you now have a formal process of inviting others to assist.”

Russel:  What are some of the competencies that make up situation management? We talked about shift handover. We’ve talked about the HMI. We’ve talked about alarm a little bit. What are some of the other competencies that are important elements of situation management?

Doug:  The list is long but not infinite. [laughs] If I were to try to decide, the primary expertise of an operator is to know where you are and what you know and where you want to go. Once you have that knowledge, then you can use the tools properly. If the tools are dependable, you can get there. If you can’t get there, you’ll know that you can’t and you need to get help.

An underlying concept of all of this is what I call the safety net. Despite everything, good design of the plant, great maintenance, operator enthusiastic, really doing everything, you can get in a situation where you or nature or everything in combination says this isn’t going to work.

One of the safety nets, I call permission to operate. It says, “If you, operator, ever get lost or think you’re lost, you’ve got to stop trying to find your way and turn the process into a safe state.” Safe state might be shutting down, or it might be into a parking operation where it can go along and you’re buying time.

Russel:  You might reduce pressure. You might reduce flow, something that allows you to get to a…You don’t want to be running wide open when you don’t understand what’s going on.

Doug:  Yes. There’s another step beyond that. What you’re doing when you’re doing that is you’re buying the most precious commodity you have, time. Situation management, in a nutshell, is time management. How you use that time and what you do during the time that you have will depend upon whether you can operate safely or not.

Russel:  [laughs] If it’s that easy, why do I have to read the 700-page book? [laughs]

Doug:  Two reasons. We have too many trees. No, it’s easy in concept. We want the operator to not have to give up too soon. We want the plant manager to know his operator is going to not have to give up too soon.

Russel:  Doug, I remember when I was going through your alarm classes. I tell people this all the time, that I had to go through your alarm…I consider myself a pretty bright guy. I had to go through your alarm class three times before I got it. The whole idea of permission to operate, again, it’s a really simple concept, but it’s very complex in terms of implementing it.

Really, what you’re trying to do, if you do a good job with the HMI and the alarm system and the other things that you’re talking about in situation management, I actually expand the domain under which I can have permission to operate.

As I have more knowledge and more insight, more visibility, I can look and control more things. Particularly, if I’ve put some thought into when the situation changes — how do I know and what do I do — and that’s planned in advance, again, that increases my domain.

The idea that when I’m lost I make it safe, to me that’s transformational. It’s a really powerful, powerful idea. I think one of the challenges I’ve always had with your work is it’s so simple, and yet so hard. Does that make sense?

Doug:  Yes. What I think hard is, is that sometimes it may be a little difficult of going where you are now to go to a different concept and understand what it is. I think it’s easy, just you like mentioned, once you understand it, it makes sense. Most people have trouble.

They say, “How can I do it any other way now that I understand it?” I think that’s of value for what I do.

Russel:  I would agree. I think that some of the content in your book, the things that…I’ll say it this way. I think that it’s pretty easy to deal with normal. What’s hard is dealing with abnormal, to anticipate the pitfalls, the traps, the things that we can get cloistered off into without realizing that’s happening to us, what I would call getting elevation on the problem.

I’m looking down at the problem versus in it, looking around in the problem. That kind of thing.

Doug:  That’s a really good introduction to two important parts of this whole situation management game. The one part is you don’t know it’s normal and you don’t know it’s abnormal. We call it off normal. It’s this whole area of, where is it? I can’t put my finger on it. I want to put my finger on it. I know where I might put it but I don’t know where I should put it.

Russel:  Let’s unpack that a little bit. I’m going to try and make sure I understand what you’re saying. Basically, you’re using the term off normal, meaning I know I’m not normal, but I don’t know where I’m at.

Doug:  That’s a little too strong. I know I’m not abnormal. I think I’m not normal. How can I understand where this gray area is?

Russel:  Again, my wheels are turning. You can’t see it on the podcast, but Doug can see it on my face because my wheels are turning.

Doug:  Your wheels are turning mine, as well.

This is where the whole thing of where the HMI comes in, to the depth of the design, to a weak signal comes in. Once it transitions to abnormal, we’ve got the alarms to handle that.

The other part of that equation that you mentioned was when it seriously goes wrong. The handbasket is getting broken. We sometimes fall back into entrenched positions that have nothing and no reality to operational procedures. We sometimes say, “I’ve got an hour invested in this problem. I’m going to work another hour to solve it. By golly, I’m going to do it.”

Or, “It happened earlier, two years ago. Jake just kept going. Jake did this and it worked out okay. Why don’t I do what Jake did?”

No problem repeats itself that well.

Russel:  That’s right. That’s what Scott Williams, my lead software developer…He and I have worked together forever. That’s what he calls an edge. It’s an edge of the problem. It’s the edges it gets you into trouble.

Doug:  Yes. They get you into trouble two ways. Number one, they have no boundary. You can cross it real easily. Sometimes you don’t know where it is.

Russel:  Exactly. It’s not like it’s a sharp, clean edge. It’s more of a fog you get into and you find the edge when you fall off of it.

Doug:  Exactly. You only find it when it’s too late.

Russel:  That’s right.

Doug:  Situation management is all about trying to avoid getting close enough to the edge, which you can’t see, before you have to fall off and find it.

Russel:  I’ve told you this, but this bears repeating. Doug sent me a signed copy of his book when it first got published. I don’t know if it was…I know I was one of the first guys to get one. I really appreciate it.

When it came in, I pulled it out. I had somebody take a photo of me holding the book. Then, I put a post out on LinkedIn basically saying, “Congrats to Doug for getting this work completed and out to the market.”

One of the commenters made the comment that the chapter on weak signals is worth the price of the book. Let’s talk a little bit about weak signals because, again, I think that’s a transformational concept. What is a weak signal?

Doug:  I’m going to go there, but I don’t want the listener to think that that’s the only power in the book.

Russel:  Gosh, no.

Doug:  Weak signals basically fills in a gap that I only discovered when I was working on it. What is a weak signal?

Let’s start with a concept called a small indication. If a really big problem, a really big situation, is going south there’s going to be cracks. There are going to be leaks. It’s going to be telling you in subtle ways, “Hey, I’m really in bad shape. Come find me.”

A weak signal is let’s take those small indications of things going bad — the leaks and the cracks. Let’s pull them out. Let’s see them. One by one, let’s ask, “Could these be meaningful?”

Weak signals is a process by taking a small indication, a leak and a crack, and asking yourself, “What could this mean if it really were something very substantial?”

A lab value that’s changing much more than you thought but not so much that it’s unreasonable. What could it mean?

It could mean that the laboratory screwed up. Or it could mean that you have some analytical information that the process itself is making a step that’s really going to go wrong. It’s whispering in your ear, “Hey, look at me.”

Weak signal says, “Take that questionable lab value. Imagine in your mind what it could mean if it really were true.” You need to understand the process to be able to do that. This lab value could basically mean that my reaction has changed to a different product. I’m producing something different. It’s in the early stages, so it’s going to be small.

You’ve imagined that. Now that you’ve imagined it, go back and look for other leaks of that problem.

Russel:  Other small indications that that problem’s going on.

Doug:  Other small indications. If you find them, you’ve confirmed the big problem. If you can’t find anything, two things are happening. Either it’s too soon. You saw the leak early, but it wasn’t big enough. Or it was just noise.

Russel:  I’m going to play with the semantics here a little bit. You’re using the term leak. In pipelining, that has a very specific meaning and it’s bad. That’s not the kind of leak we’re talking about here.

The kind of leak we’re talking about is a data or an information leak. Some information’s leaking out of the process. It’s an abstraction. You don’t really know what it means, but there’s something going on there.

Doug:  Yes, but the leaks are actually a little bit more physical. It could be a temperature profile that doesn’t look right. It could be a flow rate that’s getting more noise and you never thought it should be noisy. It could be that a step change or a change in set point didn’t have the same result as it did last time.

These are all real things, for the most part, but there’s another side to this. There’s a back of this page. They could be things that bother you. You could have a hunch. “Hmm, I don’t know what’s wrong but I don’t feel good about this.”

Russel:  That right there, that’s where…To me, that’s the real value of this idea because a controller that knows his pipeline or her pipeline. That knows it well. Their Spidey senses will tell them if something is not right.

Doug:  Exactly.

Russel:  That’s a valuable, meaningful thing. There’s all this talk about AI and data analytics. It can’t do that. It can’t do intuition. Only human beings can do that.

Doug:  Only human beings can do it. Only human beings can be worried about it.

Let’s talk about the other side of weak signals. Sure, you can find small indications. They come to you. Sometimes you look for them. I suggest looking for them right after you get on shift, middle of the shift, and just when you’re ready to go off shift. You’re writing the shift report. Go back and do a weak signal sweep.

Look for anything that doesn’t look quite right.

All the rest of the time, you can’t be looking for them. It can drive you crazy, so let’s manage this thing. Let’s manage hunches. They’re okay. They’re real. Let’s process them.

If you have a hunch about something, it doesn’t look right, try to think what could be going wrong. Process it. Get back to your job.

Russel:  That is such a radically different way of thinking because what we’re taught to do is understand the process, look at what’s going on, and relate it to the process. That’s pretty linear and it’s kind of constrained as a problem.

This is more nonlinear, abstract, and unconstrained. Look for the things that don’t fit your mental model. Don’t do it all the time, but look for them because they could be an indication of something more substantial.

Doug:  Let’s take this and let’s turn it inside out. We’re asking the operator, find a small indication. Imagine what it could be if it were really an indication of something really serious. I’m going to stop using the word leak.

Imagine that. Get it in your mind, and then take that serious thing and try to think of what would you look for to prove or disprove that’s going wrong.

The reason I say that I’m turning it inside out was if you have an operator or a controller that can’t do that imagination right, makes mistakes, or does it right but doesn’t know where to go and look, you’ve got a training or process understanding issue.

Right in the midst of your job, you’ve got a tool, weak signals, that can also talk to you about your training and process understanding.

Russel:  One of the big things that pipeline operators are talking about is, how do I get lessons learned? How do I find lessons learned? This is actually a way to mine for and discover opportunities for capturing lessons learned because now…

It does have other legs and opportunities, but to me when we were…You introduced this concept to me, I don’t know, it’s been a number of years ago now.

I remember over dinner, we got to talking about this. It was blowing my mind up about what you could do in a control room if you could get to the place that you had enough vigilance time, and take the vigilance time, and do something deliberate with it around this idea.

The opportunity, I wonder how many of the incidents that have occurred in the last 20 years would have been discovered if that had been done, particularly if it’s in a broader context than just the control center.

You take pipeline integrity and cathodic protection and some of the other things that contribute to incidents and look for weak signals within that broader system. It’s a revolutionary concept, Doug. I’m very honest about that. It’s very cool.

Doug:  It is, really is. This conversation has gotten me to the point where I want to reemphasize the value of the pieces, parts of situation management. We’re talking about a tool, HMI, or weak signals, a shift change.

The tools are useful in their implementation, but they’re also useful in terms of “Hey, Mr. Systems Designer, did you know the operator is going to do it this way? He needs to do it this way. Do you understand what he’s going to use?”

It basically changes the entire design concept. One of the things I say to engineers, chemical engineers, mechanical, electrical, controls. Design what you’re doing to be understood, to be managed. Don’t build it and cobble together, even though it’s a masterful job, if it’s too hard to understand or you didn’t understand it and it’s too hard to manage.

Russel:  Yes, yes, and amen. We have to remember that we’re here to fly the airplane, not build the airplane, that the business is flying the airplane and that somebody has to understand how to fly the airplane, drive the process. Right on point. Right on point.

Doug:  Again, I want to turn that around.

Russel:  I’m sorry. We could spend all night. We’ll have a four hour long podcast, if we’re not careful. You and I will go off into that soup really easy. Here’s what I would like to ask you.

If you were counseling somebody who said, “You know what? I want to look at situation management. I want to think about how I might implement a program like this in my control center,” how would you tell him to start?

Doug:  That’s a really easy answer. It’s an easy answer because the situation management whole concept basically introduces an operator or controller to his whole sphere of influence, his whole sphere of activity. It introduces a pipeline owner or an industrial plant owner to what do you have to provide to make this thing work.

The bottom line in everything I talked about in situation management is not…Except for weak signals, I don’t want you to go out and find new tools. I want you to go out and say, “What am I missing?”

I want you to take the whole totality of the tools and concepts of situation, bend them, forge them, change them to fit into your way of doing business, and then fold them back in. Don’t add new stuff until you understand it and make it your own.

Russel:  I’m going to say the same thing to say it in my words, just to make sure I’m understanding. That is that really the way to start, this is not about changing how you’re doing things. It’s not about changing your systems. It’s much more about creating a system for understanding and a system for managing around situations.

Doug:  That’s a great summary. It’s my way as an engineer to pay it forward.

Russel:  I’ve got to tell you it’s a great book. I would like to tell you I’ve read the whole thing. I have not, but I have read parts of it. I’ve got to tell you I think it’s a seminal work. It’s worthy of your career, Doug. I congratulate you on the effort. I encourage those of you that are pipeliners and have interest in such things to get a copy and set about reading it.

Once you’ve got it read, maybe you can call me up. We’ll have a conversation about how you think you might add value in your pipeline operation taking the work that Doug’s done and codified in his book here. Doug, thanks for being on the podcast. It’s a pleasure.

Doug:  Russel, pleasure is mine.

Russel:  I hope you enjoyed this week’s episode of the Pipeliners Podcast and our conversation with Doug Rothenberg. Just a reminder before you go. You should register to win our customized Pipeliners Podcast YETI tumbler. Simply visit pipelinepodcastnetwork.com/win to enter yourself in the drawing.

If you would like to support this podcast, you could leave us a review on Apple Podcast, Google Play, or whatever smart device podcast app you use.

[background music]

Russel:  You can find instructions at pipelinepodcastnetwork.com. If you have ideas, questions, or topics you’d be interested in, please let us know on the Contact Us page at pipelinepodcastnetwork.com or reach out to me on LinkedIn. Thanks for listening. I’ll talk to you next week.

Transcription by CastingWords