Before you can take a deep dive into SCADA technology, you need to understand how to simplify the SCADA build.

In this episode of the Pipeliners Podcast, Russel Treat and guest Scott Williams review the ins and outs of setting up SCADA systems, getting buy-in from key stakeholders, and cutting off issues in the beginning to prevent headaches later in the process.

You will learn that simplifying the SCADA build is about analyzing your available systems, planning the build, communicating in all phases, and making sure teams are on the same page to complete the project.

Simplifying SCADA Build: Show Notes, Links, and Insider Terms

• Scott Williams is the Manager of Development and SCADA for EnerSys Corporation. Find and connect with Scott on LinkedIn.
• SCADA (Supervisory Control and Data Acquisition) is a system of software and technology that allows pipeliners to control processes locally or at remote location.
• Control Valves regulate the flow of products in a pipeline. For instance, a control valve could be set to target a specific pressure or could control mixing to target a specific temperature.
• Shutdown Valves stop the flow of products in a pipeline. They are commonly used to stop flow when a leak is detected or when an adverse operating condition is detected.
• PLCs (Programmable Logic Controllers) are programmable devices that take action when certain conditions are met in a pipeline program.
• The alarm rationalization process is a component of the Alarm Management process of analyzing configured alarms to determine causes and consequences so that alarm priorities can be determined. In addition, this information is documented and made available to the controller to improve responses to uncommon alarm conditions. (Read about the ALMgr software capabilities offered by EnerSys.)
• Fixed function RTUs (Remote Telemetry Units) are electronic devices placed in the field. RTUs enable remote automation by communicating data back to the facility and taking a specific action after receiving input from the facility.
• A type of RTU is an EFM Flow Computer that measures the flow of gas or fluid and reports the data back to the facility. It differs from an RTU in that it is designed to compute flow using standard flow equations with specific timing and reporting requirements.
• ISA 101 (Human Machine Interfaces for Process Automation Systems) is the standard, recommended makeup of HMI (Human Machine Interface) in pipeline operations.
• High Performance HMI is an advanced HMI system that improves a pipeline operator’s situational awareness during normal and abnormal situations. (Listen to an earlier Pipeliners Podcast on the topic of High Performance HMI.)
• P&ID refers to Piping and Instrumentation Diagrams that explain the interconnection of process equipment and the instrumentation used to control the process.
• A trend chart is a graph of a value or values over time.
• Factory acceptance testing (FAT) refers to the important pre-implementation step of testing new processes, software, and hardware “in the factory” prior to installing it in the field. During the FAT, you can test new functionality without the risk of upsetting the running system.
• Site acceptance testing refers to testing of new processes, software, and hardware in the field. This ensures that equipment is set up, communicating, and functioning correctly.
• Point to Point verification confirms that input or output from each field instrument is accurately and reliably reflected in the SCADA information presented to the controller. (EnerSys offers PointMgr to manage this process.)
• A site execution plan or project execution plan is the document that establishes the means to execute, monitor, and control pipeline projects.

Simplifying SCADA Build: Full Episode Transcript

Russel Treat:  Welcome to the “Pipeliners Podcast, Episode 11.”

[background music]

Russel:  Thanks for listening to the Pipeliners Podcast. We appreciate you taking the time to listen to this episode.

To show our appreciation, we want to let you know about our signature prize pack. We are offering a free customized YETI tumbler to one listener each episode.

How do you register to win? Simply visit pipelinepodcastnetwork.com/win — that’s pipelinepodcastnetwork.com, slash, the word “win” — to enter yourself in the drawing. This is our way of saying “thank you” for being part of the show. We are excited to announce our latest winner. Congratulations to Travis Friesenhahn.

This week, we have as our guest, Scott Williams with EnerSys Corporation. Scott is the Manager of Development and SCADA at EnerSys. He has 20 years of experience building SCADA systems from scratch in multiple different applications and in multiple technologies, all in oil and gas. Scott, welcome to the Pipeliners Podcast.

Scott Williams:  Thank you.

Russel:  It’s really good to have you. For our listeners’ benefit, we put this upfront. How long have you and I been working together?

Scott:  Let’s see. It has been a while. I’d say certainly over 20 years. My kids are in high school now. The kids weren’t around when we first got started.

Russel:  You and I have spent a couple of decades plus doing SCADA and measurement projects. What we’d like to do today, what I’d like to do today is have a conversation that maybe others that are learning this business could benefit from. Maybe they can learn from all the things that we’ve done wrong, although I don’t know if we can get all that in a single podcast episode. [laughs]

Scott:  It’s just a fair bit of that.

Russel:  The first question I’d like to ask is, you’ve been doing this a long time. What do you think is the most important when you’re looking at doing a major upgrade or a change out of a SCADA system? What do you think is the key thing you need to worry about upfront?

Scott:  One of the first things you want to do when you’re evaluating your old system, your existing system, as part of an upgrade is to get a handle on what you have, because just looking at it…It may look like this huge task. When you start digging into it, there’s a lot of similar pieces in that entire system.

One of the first things we’ll do is look at how you want to organize your new system. What we tend to use is a hierarchy based organization. We call it “our area of operations.”

You may have a west operating region. You may have some facilities out there. You may have, let’s say…It’s called a “gas gathering system.” These all are sub areas within what I’d say east system.

At a particular facility, you’re going to break that up based on how you operate. You may have the inlet metering and the inlet process. I’ll use a gas storage because that’s pretty common. A lot of people are familiar with those.

You have your metering coming into the facility. You’ll have your compression processing the product, putting it into the ground. You’ll have your pressure regulation heaters for when you’re taking it back out of the ground, and then output metering. You may break those up into areas.

Within those areas, then you start looking for common equipment. Heaters are a pretty common thing. You’re going to have valves all around that facility. You have the meters, your meters at the in and the out, and so on.

Once you’ve identified that key equipment, then you start thinking about, what are your different variants? Because you’re probably not going to have very many.

Consider a meter. For the most part, if it’s a gas meter, you’re going to have pressure, temperature, and some other flow related field. It might be differential if you have an orifice meter.

If it’s liquid, you may have a couple of different fields. Meters are pretty common things, regardless of the type of equipment you have.

Valves are another really big one. In your facility, you’ll have valves for safeties that will be sitting at a particular position, until you have to execute an ESD or some sort of emergency shutdown. Then you’re going to open those valves. Those are simple one-function valves. You’ll have other valves that control the flow throughout the facilities. You open or close those.

Last, you’ll have your common control valves. How much gas are you running through the heaters or your pressure regulators?

There’s three different variants of valves. After you’ve got those variants, then it’s really a matter of which kind of actual physical valve corresponds to which of those variants, which of those models is for a valve.

Russel:  As we’re having this conversation — and you and I have had this conversation a lot and in depth — I’m trying to think about how somebody who maybe is new to this might be thinking about that. The question that comes up for me is: why do you need to organize it that way?

Really, it’s just a bunch of points. It’s just a point to open, a point to close, and a point or more that tells me what the position of the valve is. Why do I got to classify valves by type, like shutdown versus lineup versus flow control?

Scott:  Compare a control valve to just an open close valve that you use to control flow through your facility. Your open and close valve, let’s say, you have an open command, a close command, an open limit switch, and a close limit switch, just a very basic valve.

For a control valve, you’ll have a manual position set point. With that, you’ll tend to have an auto manual control. You’ll have whatever process variables that tie into that control.

Just looking at those two, there’s these fairly different bits of data associated with each. To the root of the question, why is that important? Well, I can look at my drawings, and I know where I have valves. It should be pretty straightforward to identify, “This is a control valve. That one is an open close valve.”

When I go look at my data as I’m sorting through my SCADA data points — just a bucket of points — I can identify that, “Okay, here’s valve 542. Here’s the open limit switch, but I don’t have a close limit switch.” Because I’ve normalized my standard stuff, I know what points I’m expecting to find.

If I don’t find those points, then I have to dig deeper and figure out why. Maybe I didn’t even have that point in the first place, which suggests an error in your SCADA system, or maybe it wasn’t an open close valve in the first place. It was a shutdown valve.

Russel:  This is the stuff that always drives you crazy when you’re trying to get organized to do a project, right?

Scott:  Yes.

Russel:  It’s really getting an understanding of the data. Just because you got a valve on the drawing, and you’ve got points in the PLC, you’ve got to make sense of all that, right? To the extent you can get through that quicker, the better.

The thing that tends to drive me most crazy when we’re doing projects, it’s not the points that are missing. It’s the ones that are there, and we don’t know where they go.

Scott:  Yeah.

Russel:  Because you’re always worried, “Is that something that’s really important to the process? Is that something that’s important operating information?”

When you start organizing these things by what they’re used to do from an operating perspective…because in a lot of cases, an open close valve for a lineup in the PLC, it’s the same thing as an emergency shutdown valve. The points are the same, but the use is completely different.

The other thing about creating these standard valves by their purpose is now, I can start getting the alarming and the alarm rationalization standardized. We talked about this in an earlier episode with Matt McClure about his biggest challenge in standing up a control room is always in the area of the alarming and the alarm management.

Part of the reason is, there’s the disconnect — or the difference may be a better way to say it — between people who are building the automation logic and people who are understanding how to operate the process. That’s a different view to the data, if you will. It raises the question.

In your experience, when working with a legacy SCADA system, an older SCADA system, and you’re doing an upgrade project, what do you think the typical error rate in the data…I’m defining error as a missing point or a misspelling on a point or a point that’s assigned to the wrong place. What do you think the error rate typically is?

Scott:  I’d say if it’s a SCADA system that’s been around for a while, then it typically is, when you don’t upgrade or change your SCADA system after a couple of years — you do it after 5 or 10 years — because it’s painful. It takes time.

With that kind of age, the likelihood that you have an incorrect point is pretty low. However, that doesn’t necessarily mean that you don’t have extraneous points, because that’s extremely common. That’s just age.

I would imagine if you were to study it, you can say for every number of years a SCADA system has been around, you’ll gain this percentage of points that don’t do anything anymore. Maybe controls that have never been stroked for whatever reason. That’s not supposed to happen, but it doesn’t mean it doesn’t.

Russel:  Just if I might add, there can be a distinction between the points the controllers are seeing, and the entirety of the points that are actually in the SCADA database. I’ve seen situations where it looked like people started to build a meter, didn’t finish.

Then they started to build the same meter, they didn’t finish, and then they built the meter up. You’re back behind them. You’re looking at all the points and like, “It looks like this meter is in here three times.”

Scott:  Yes. I’ve seen that before. The other thing that tends to make a pretty big difference into your ability to apply consistency for the new system is the nature of your automation.

If you have a lot of, let’s say, fixed function RTUs, like a ROC Flow Computer, things like that, those tend to have a pretty standard set of points you’re going to be looking at. They’re the same across the different ROC versions.

If you have a bunch of ROC meters, they’re probably all right. They probably are the same. You probably have all the data.

Now, you go look at PLCs. There are some companies that do a really good job of using consistency in their PLC loads across different their sites. Those guys are sitting pretty for SCADA upgrades because, at a very high level, the PLCs are all similarly put together.

You have systems that evolved over time. You have different vendors doing the PLCs, and that kind of consistency wasn’t enforced. Your risk of finding errors and having errors is much higher in that case.

Russel:  Maybe not even an error. I would want to make a distinction between errors and inconsistencies, right?

Scott:  Right.

Russel:  One challenge is an error, but the other challenge is inconsistency. It could be inconsistency in naming convention, or inconsistency in the way the data is structured. It could be an inconsistency in the way the automation is approached. There’s lots of ways that things can get different. [laughs]

I’ve never started a project like this, where people didn’t say, “It’s all the same. We did it all the same,” until you start pulling it back. You’re looking at it. I was like, “Yeah, they are all a blue suit, but none of them the same size.” That kind of thing.

The other thing that’s present for a lot of people doing this stuff is that, if I’ve got a mature pipeline operation, a large transmission, the nature of how it’s operating is not changing, then those guys operate — their whole thing is risk averse and optimization.

If, on the other hand, I’m looking at a midstream operator, they’re building facilities, and their ability to get the facility up and running is what’s critical, then…I’m doing this with multiple facilities. It’s easy for things to diverge.

Scott:  Certainly. If you have a single facility done first, then you can promote a certain consistency. “Here’s the model we want to use.” That can make a huge difference.

If you got multiple sites going up, you’re going to have different vendors working on the PLC loads of those two sites. Keeping everybody on the same page is going to be tricky.

Russel:  Scott, you said it very well that you need to understand what you have. To the extent you really wring that out in detail upfront, and you understand what you have    regardless if it’s very well organized, and it’s very consistent or not, you’re going to do a better job of executing the project.

Let’s follow up with this question. We talked about the need to get the data well organized. What do you think is the second most important thing?

Scott:  That’s a tricky question. This may follow up the first one. Having the customer resources to help you work through those, that’s a definite plus. That’s not really what you’re asking, though. [laughs]

Russel:  You’re making a good point. This is one of those areas where you and I come at things from a different perspective.

Customer involvement, whether you’re doing this internally as a SCADA group for a customer that has pipeline control or whether you’re an integrator or what, having the user involved upfront and through the process — not full time involved but so they know what decisions are being made and they understand what’s coming — is really critical to success.

Particularly when you’re doing an upgrade to something that already exists, the worst thing you can do is not plan for the need to train and facilitate change when you start bringing this up and operating this new system.

It’s not going to be the same. Even if you sit down and you’re going to copy, it’s not going to be the same. The only time it’s the same is when I’m just running a software upgrade.

Scott:  That triggered a very good thought for me. As you mentioned, having the end user people available makes a big difference.

The other thing that will help is when you’re defining what screens you actually need. At the point you’re building this new SCADA system, adding screens and doing that is relatively low cost compared to doing it down the road.

Think about what the controllers do every day. Are there screens they have that they never use? Are there screens that combine what was on a couple of different screens in the old system, that would make particular tasks they do a lot easier?

Your screens aren’t just, “This is my system. This is how I’ve organized it.” Your screens also should support what the controllers need to do.

If there’s a task that involves what would normally be, let’s say, on a cavern screen and a pressure regulator screen and a heater screen, put the key bits on one single screen so they can manage what it is they’re trying to accomplish without switching back and forth between the massive screens.

Russel:  Exactly. To use your gas storage example, gas storage facilities operate basically four ways. They’re either injecting into the caverns, they’re withdrawing from the caverns, they’re wheeling, meaning, they’re moving gas through the header, but they’re not really putting gas in or out of the caverns, or they’re just dormant. Nothing is going on. Those are the four states they operate in.

What I need to see when I’m injecting is different than what I need to see when I’m withdrawing, because I’m running different equipment. Likewise, if I’m wheeling, what I need to see…

If you use the ISA 101 model or the high performance HMI model…and you talk about, about level one, two, and three screens. What we’re talking about here is building task-specific screens, so that rather than an operator having to jump between two or three screens to do a task, you put all of it on a single screen so that they can do the task.

Scott:  Right.

Russel:  Whatever you do, you want something that supports organizing the equipment into hierarchy. It makes these things easier.

You want things that are templated or objects so that…I take valves and meters and things. I put that together. That’s a meter skid. I can just take that meter skid and drop it on a screen.

When you start doing that, then it gets easier. The challenge is when it starts getting easy to build screens, the gravity will take us to building a whole bunch of screens that nobody is actually going to use.

Scott:  Having conversations with the guys that are actually running the system and learning what they do day to day or week to week that is painful in the current system because of the screen layout, just having that conversation with them can help you find the screens that they may not even know they needed. They’ll love you for it.

[laughter]

Russel:  Exactly. That’s another really excellent point you’re making, Scott. Because in my experience working with the controllers, the guys that are actually in front of the screens, doing the work, they are a wealth of knowledge, but you have to mine for it.

What I mean by that is, what they tell you they want — because they don’t really have the context of how you build these things — may not be that helpful. What they know they need that you have to — through conversation and working together — pull out is extraordinarily valuable. Those guys are going to have ideas about, “Well, if I could do this and this and this on the same screen.”

You also have to talk about maybe some different ways of looking at the data, other than just valves and P&ID diagrams and trench charts. You may want to figure out a way to combine things in a way that maybe they haven’t thought about because they’ve never seen that kind of screen.

Scott:  Having done a number of systems, you can bring ideas like that to the table. In my career, I’ve seen dozens of SCADA systems. I doubt that it would be easy for you to find a controller that could say the same.

Russel:  That’s a good point. Here’s the other question I want to ask. I’m going to tee this up because I think the third most important thing is how you do the factory acceptance and site acceptance testing.

If you do that well, you streamline and simplify the entire project. If you don’t, that’s where you’re going to get into the project that never ends. What’s required to do an efficient and effective factory acceptance, and an efficient and effective site acceptance?

Scott:  The factory acceptance, there’s a couple of different things that help a lot. Ideally, at your factory acceptance test, you’re going to have the availability of at least one of each type of field equipment.

You can actually look at specific screens against specific devices, and see how the interactions work. At another level, you’ll want to be able to walk through the screens and see how tasks are performed.

The FAT is about, does the basic functionality do what it’s supposed to do? It’s also an opportunity, let’s say, a last look of, “Is there anything here that will make our life difficult where it doesn’t need to be?” Once you get in the field and you start installing this stuff, it’s too late for that.

You don’t want to have those kinds of changes come up when you’re in the field. All the services are planted. People are waiting to start using the new system. The more that you can get in the FAT before you’ve gone to the field, the better off you’re going to be.

Russel:  What about the site acceptance?

Scott:  Under the best possible circumstances, you can have the old SCADA system and the SCADA new system both running, both live at the same time. That promotes a couple of different things.

Unless the controllers get a little hands-on of saying, “Okay. This is what I saw at the old one. This is what I saw at the new one,” it’s an additional way to prove out your data. It’s like, “Here is what I’m seeing on the old screen. Here is what I’m seeing on the new screen.”

In addition to all of that, you want people in the field that can manipulate the IO, can manipulate the field equipment, so you can prove it gets to your system correctly. It’s your point to point 101. That’s critical especially for controls because it’s a new system.

All those are different things that are involved in the SAT, the site acceptance test. You want at least one or two of the key day to day controllers involved in that, not just the SCADA supervisor, because that’s their first chance to see it with their own data and to start the real adoption process.

Russel:  One of the things you didn’t talk about, and I think is the best practice, is you should also have a site execution plan. The site execution plan lays out the process to get the equipment installed, to establish communications, and to call, “Commissioning complete, and I’m ready to do site acceptance testing.”

Where these projects tend to go is the commissioning and site acceptance testing get blended. They’re not really distinct parts of the process. That can cause thrashing. I go out to the site. I try to get everything commissioned and tested. I find a problem. You need to do that during the installation side.

Scott:  Absolutely. The site execution plan, which details what you’re getting set up, you want to be through that. You want your systems in place. You want them working.

You want communications to be up. You want to be able to get to that point and say, “Okay. This should be working now,” before you start the actual formal testing.

Communication things are going to come up any way, but you don’t want to start there. Let that be an exception rather than, “The comms aren’t quite working yet.” Just get through all that. Get all that done, then start your testing.

Russel:  Probably one of the most valuable things that we do in our process is we review the site execution plan during the factory acceptance test. If you do a thorough review of the site execution plan right after you are finishing up or as you’ve wrapped up the FAT, it’s fresh on everybody’s mind.

All these questions about, “How are we actually going to get two sets of consoles where we can see both of them? How’s that going to work?” Those kind of nitty gritty details tend to get wrung out more effectively, particularly once you’re actually sitting in front of the new one and have a sense of what it is and how it’s going to be implemented.

Scott:  Certainly. Not to mention what field resources you’re going to need on site, because doing point to points, you need people to be out there. Depending on the size of your system, you may have multiple teams.

Having that conversation well in advance, unless the customer planned for that, make sure they have the appropriate people available, so you can actually get through any installation issues as well as site acceptance issues when you’re on site.

Russel:  The other thing that’s important to understand is the kind of people you need to do installing and commissioning. I’m calling commissioning. I’ve got the comms in. I’ve got the data lit up. I’m ready to start actually doing site testing.

Those skills are actually different than the people that are going to go out and hook up the instruments, and range them and support the point to point. Those guys want to move fast, because generally, the people you have doing that, you’re taking them off of other duties.

Scott:  That’s part of why you want to get everything commissioned and working before you start getting them involved because that’s the worst thing. They’re out there working on something. You say, “Oh, wait, wait. Comms aren’t working. Let me get the comms up to that site.” They won’t be happy with you.

Russel:  Exactly. We’ve been talking almost 30 minutes, I believe. We really hadn’t talked at all about technology. We’ve been talking about how to simplify the SCADA build.

Really, simplifying the SCADA build, it appears, based on our conversation, that it’s about the analysis of what you have, the planning, the talking, the communication, and the teamwork to get it all done.

Scott:  That’s pretty accurate. There’s multiple SCADA products. I know which ones I prefer. I prefer them for technical reasons because they support the way I like to build systems, but there’s more ways than one to skin a cat.

Russel:  If you could tell young Scott something important that he needs to know, what would you tell young Scott?

Scott:  I’d say some of the most important things that make the most difference at the end of the day are how consistent you can be building your system, how close to reality you can get your FAT, your factory acceptance test.

There’s always a limitation. You’re not going to have all the RTUs, all the field equipment, but how close you can get to reality for your FAT, and then how rigorous and well defined your site execution plans and your site acceptance test plans are. If everybody is on board with those before you get to them, your life goes so much easier.

Russel:  I think I would tell young Russel the same thing. Exactly the same thing. The point you made — and we probably ought to explore this a little bit — is make the FAT as real-world as possible.

People tend to simplify the FAT rather than work to make it as real-world as possible. In my experience, the more real-world the FAT, the better everything else after the FAT. The less real-world, the more you’re going to be sorting out in the field, where everything costs more and takes more time and people get irritated.

Scott:  Exactly. One other thing that I want to tell old Scott. Talk to the controllers from day one.

Russel:  Oh, gosh, yes.

Scott:  Get their input from the very, very beginning.

Russel:  I would tell you, Scott, that at this point in my career, I don’t think I’d want to do a project where the people in front of the console are not involved from inception. I don’t mean everybody and all the time, but enough so that they feel that their input is heard and put into the project.

Ultimately, that’s who you’re building this for. They’re the ones that are going to use it. They’re going to determine whether it’s a success or a failure.

Scott:  On any controller team, you’re going to have a few guys that are the old hands or that are the recognized voice of the group. You get them involved, everybody else will be fine.

Russel:  I agree with you. I agree with you. Again, we only talked about this in an earlier episode with Jeremy Coleman, as he was implementing the high performance HMI in his control room. He talked about the need to have those guys involved, and how you know whether what you’re building is good or not. Do they use it or not? If they use it, it’s good. If they don’t, it’s not. It’s pretty simple. [laughs]

Look, this has been great. I know that we could probably unpack this more, but this is probably a good time to bring it to a close. I’d just like to say thank you very much. I appreciate it. We’d like to have you back at some point in the future.

Scott:  It’s been fun. I look forward to it.

Russel:  I hope you enjoyed this week’s episode of the Pipeliners Podcast and the conversation with Scott Williams. I certainly did.

Just a reminder before you go, that you should register to win our customized Pipeliners Podcast YETI tumbler. Simply visit pipelinepodcastnetwork.com/win — that’s pipelinepodcastnetwork.com, slash, the word “win”– to enter yourself in the drawing.

[background music]

Russel:  If you have ideas, questions, or topics that you’d be interested in, please let us know on the Contact Us page at pipelinepodcastnetwork.com, or reach out to me directly on LinkedIn. My profile name is Russel Treat. That’s R-U-S-S-E-L, with one L, and Treat, just like it sounds, T-R-E-A-T.

Thanks again. I’ll talk to you next week.

Transcription by CastingWords