This week’s Pipeliners Podcast episode features Scott Williams of EnerSys Corporation returning to the podcast to discuss what you should know about SCADA for pipeline operations when setting up a new system or making a change. 

In this episode, you will learn about the important components of SCADA that can shape key choices, including reliability, communications, redundancy, data collection, and more. Scott and Russel also expand on the importance of having future-centric systems, how it affects communications and security, and how to understand the differences between SCADA systems and platforms.

Scott also highlights SCADA tools to be cautious about and provides recommendations on how to have a properly functioning system between your SCADA engineers and controllers. As an added bonus, this episode is filled with laughter as Scott and Russel recount their past projects and what they’ve learned through the years.

SCADA Integration: Show Notes, Links, and Insider Terms

• Scott Williams is the Development Manager for EnerSys Corporation. Connect with Scott on LinkedIn.
  • EnerSys Corporation is the sponsor of this month’s Pipeliners Podcast episodes. Find out more about how EnerSys supports pipeline operations compliance, audit readiness, and control room management through the POEMS software suite.
• SCADA (Supervisory Control and Data Acquisition) is a system of software and technology that allows pipeliners to control processes locally or at remote locations.
  • Redundancy consists of setting up one or more servers to ensure the SCADA system is always available, helping to prevent a server failure by allowing the redundant server to take over as the primary server in the interim. [Learn more about SCADA Redundancy from the EnerSys website.]
• Modbus is an older protocol that enables communication among many devices connected to the same network. The drawback is delays in the communication, oftentimes creating timestamp discrepancies.
• BSAP (Bristol Standard Asynchronous/Synchronous Protocol) is a poll-oriented communication protocol for horizontal, vertical, and multi-layer networks. BSAP is the foundation for a proprietary network that has a tree-structured topology.
• TCP/IP (Transmission Control Protocol/Internet Protocol) allows digital computers to communicate over long distances by collecting and reassembling the packets of internet data and sending them to the right destination. 
• MQTT (Message Queuing Telemetry Transport) is a publish-subscribe protocol that allows data to move quickly and securely through the system and does not bog down the system with unnecessary requests.
• HMI (Human Machine Interface) is the user interface that connects an operator to the controller in pipeline operations. High-performance HMI is the next level of taking available data and presenting it as information that is helpful to the controller to understand the present and future activity in the pipeline.
• TCP Radio (Transmission Control Protocol) is a communication protocol that allows users to run multi-connection TCP applications for ham radio systems.
• EFM (Electronic Flow Meter) measures the amount of substance flowing in a pipeline and performs other calculations that are communicated back to the system.
• OPC (Open Platform Communications) is a data transfer standard for communicating device-level data between two locations, often between the field and the SCADA/HMI system. OPC allows many different programs to communicate with industrial hardware devices such as PLCs. The original system was dependent on MS Windows before shifting to an open platform.
  • OPC DA (or OPC Classic) is a group of client-server standards that provides specifications for communicating real-time data from devices such as PLCs to display or interface devices such as HMIS and SCADA.
  • OPC UA (Unified Architecture) is a platform-independent service-oriented architecture that integrates all the functionality of the individual OPC Classic specifications into one extensible framework.
• Distributed Services Architecture (DSA) is an open-source IoT platform that facilitates device inter-communication, logic, and applications at every layer of the Internet of Things infrastructure.
• Monolithic Architecture (Monolithic Design) is a traditional unified model for a software program design meant to be self-contained, interconnected, and interdependent rather than loosely coupled. If any component of the program required updating, the entire application would require a rewrite. 
• ClearSCADA is a SCADA system compromising one or more ClearSCADA servers. The server contains the database (configuration and data for each item on the system) and communicates with the other devices that comprise your system. The Server typically monitors such devices and retrieves their data.
• CygNet (CygNet SCADA) platform was designed for pipeline and midstream operators to process diverse data from multiple sources. The platform allows decision-makers, users, and stakeholders to prioritize and analyze real-time data to support daily operations and decision-making.
• AVEVA (AVEVA Enterprise SCADA) is a Pipeline Management System that serves as a digital transformational platform for midstream operators to leverage advanced analytics and cloud capabilities to deliver safe pipeline operations, leak detection, and enterprise decision support applications.
• Wonderware is a SCADA solution that is being used by many different industries, including oil and gas. The Wonderware platform can easily interface PLCs with HMIs across different platforms and has intuitive and creative graphics used in the software.
• Archestra (Archestra Control Engine) is a suite of software components used for the planning, development, and deployment of real-time control applications for industrial machines and robots.
• Ignition (Ignition SCADA) combines an unlimited licensing model, with instant web-based deployment, and the industry-leading toolset for supervisory control and data acquisition (SCADA) — all on one open and scalable universal platform.
• Colonial Incident (Colonial Pipeline Incident) is the ransomware attack that targeted Colonial Pipeline on May 8, 2021. In this incident, the attackers stole more than 100GB worth of data prior to encrypting Colonia’s network, leading to fuel shortages across the U.S. East Coast.
  • Listen to a Special Edition of the Pipeliners Podcast covering this incident. 
• ICS (Industrial Control Systems) captures the control systems and instrumentation used for industrial process control. These systems are used in oil & gas and other key industries.
• GIS (Geographic Information System) is a vital tool for data creation, analysis, maintenance, and storage in the pipeline industry.

SCADA Integration: Full Episode Transcript

Russel Treat: Welcome to the Pipeliners Podcast, episode 204, sponsored by EnerSys Corporation, providers of POEMS, the Pipeline Operations Excellence Management System, compliance and operations software for the pipeline control center to address control room management SCADA and audit readiness. Find out more about POEMS at EnerSysCorp.com.

[background music]

Announcer:  The Pipeliners Podcast, where professionals, Bubba geeks, and industry insiders share their knowledge and experience about technology, projects, and pipeline operations. Now, your host, Russel Treat.

Russel:  Thanks for listening to the Pipeliners Podcast. I appreciate you taking the time, and to show that appreciation, we give away a customized YETI tumbler to one listener every episode. This week, our winner is James Perigo with Phillips 66 Pipeline. Congratulations, James. Your YETI is on its way. To learn how you can win this signature prize, stick around till the end of the episode.

This week Scott Williams, product manager at EnerSys Corporation, is joining us to talk about SCADA integration — what every pipeliner should know. Hey, Scott, welcome back to the Pipeliners Podcast. It’s been a while.

Scott Williams:  Hey, great to be here.

Russel:  What you’ve been doing for the last year?

Scott:  Staying home, staying safe. Got a kid off to college.

Russel:  That can be good fun. That can be stressful. That can be good fun all at the same time.

Scott:  All the above.

Russel:  I thought what we would do, seeing how it’s been a year since we got really geeky and talked about things that I knew. I’ve been talking about integrity management on the podcast a lot for the last six or eight months.

I thought it was time I talked about something I knew a little something about. I asked you to come on so we can get really, really geeky. Maybe not as geeky as we are capable of getting, but geeky enough.

Scott:  Geeky enough. There you go.

Russel:  Our topic for this evening is SCADA integration — what everybody should know. I might rephrase this as what I wish everybody knew about SCADA integration. The first question that will be fun to talk about is, if you’re putting in a brand-new SCADA system, or you’re thinking about doing a change from what you have now to something else, what are the things that really matter? What are the things that are really important?

Scott:  There’s a number of SCADA products on the market and they all have different strengths and weaknesses. What you got to look at is what are going to be my most important factors.

There’s certain SCADA systems that scale better, higher point counts, and they better support that than others. There’s redundancy. How important is redundancy, what kind of redundancy do you want?

There is, just call it external connectivity. Your back-office folks, the folks that aren’t in SCADA, who need to see the data. Do they need to see the data? Do they need to see the screens? Those really big picture things should shape your choice of SCADA system.

Russel:  I agree with all that. I want to unpack a little bit higher reliability. Before we do that, the other thing I would say is you want a software tool that you think is future-centric.

The thing that seems to be changing the most and all the time is the communication protocols that are available and how they work. When you and I first started, everything was serial and everything was Modbus, it may be an oversimplification, but not by much.

Scott:  I think my first system was BSAP. [laughs]

Russel:  I see. For those of you that are super geeks and really old, you know what that means. That was an old legacy serial protocol. You had to really look at the bits and the bytes and the hex and all that kind of stuff to understand what was going on under the covers.

Nowadays, most everything we’re doing is TCP/IP and encapsulated, and we’re dealing with whatever the manufacturer’s native protocol is, right?

Scott:  Right.

Russel:  To be future-proof or future-centric, I probably need to be looking at a couple of things. One is I need to be looking at MQTT. For folks that don’t know what that is, that’s a communications protocol.

It’s where the industry is moving at the moment because it allows you to get a lot more data and a lot more data value, probably a better way to say that. That’s what’s going on with the factory floor. It’s quickly moving into oil and gas.

Then the other thing is applications that are data promiscuous, meaning they’re easy to get the data from one place to another, and I can get it to a web presentation so I have the ability to access things remotely if I choose to open that up, this whole other conversation.

Anyways, let’s talk about high reliability because I don’t think a lot of people really understand what that is. Every SCADA tool out there can do some high reliability. Sometimes, it’s built into the tool and it’s just a matter of configuring a few things. Sometimes, it’s a matter of building a whole system to maintain and manage the high reliability.

What is your experience with folks that are building high-reliability SCADA? What I think of that, I’m thinking about a large pipe liquid control center, and it can never ever, ever go down, at least in terms as far as the guys running the consoles are concerned.

Scott:  When you’re thinking of high reliability, there’s two major components, and I haven’t seen anything to shake me of this opinion yet, but the hardest part is communications. Getting a proper, true, redundant communications is a very hard problem. Let’s say not very hard, I mean. Obviously, AT&T, your phone company, knows how to do it. That’s the enterprise, a large operation doing those kinds of things.

Russel:  Having the ability to the phone network goes down. I got another way to get out to the remote side. Right?

Scott:  Right, and how far do you want to set out and how much money do you spend on that.

Russel:  What rules do you build in the SCADA system for how long can I operate, and how do I operate when I lose communications, and all that kind of stuff. Certainly, that’s key.

The other thing, too, that’s missed a lot of times, people tend to think about hardware redundancy. A redundant server that’s sitting there ready to go, but they don’t often think about software redundancy.

I have to know if the software quits working the way it should, it should failover. Hardware is still fine, but software’s messed up. It needs to failover because it got messed up. How do you do that?

Scott:  When it happens, you want to have minimum downtime, of course, but also you want minimum data loss. You’re pulling real-time communications, and then it just stops, that takes 15 minutes to recognize it and switch over. You’ve got a data gap of 15 minutes. Depending on your operation, it might be fine, but it might not be.

Russel:  That’s exactly right. The appropriate people need to be notified, but the folks working the console ideally will not even know that it happened, unless they get some kind of notification that, “Hey, you’re now running on the backup server.”

Scott:  You’ll see a little status change up in the corner, you go, “Oh, look at that.”

Russel:  It shouldn’t affect my ability to operate. That’s the point.

Scott:  Yeah, absolutely.

Russel:  That’s the biggest difference. To my mind, that’s always the biggest difference between a true pipeline operations center SCADA product and a station HMI. Station HMI looks to the operator identical more or less to a high-availability SCADA system, but the underbellies are very, very different.

We talked a little bit about future-centric. You said the biggest thing is communication. What are some of the electric communications subtleties that people ought to understand when they’re talking about SCADA?

Scott:  We’ve got a couple of different factors. We talked about how the shift is toward TCP/IP communications. Most people think of TCP/IP happening over an Ethernet cable, which in today’s day and world, that’s going to be 100 megabits and faster, plenty fast. However, the communications out to the field are not Ethernet and high bandwidth all the way.

Very likely, you have data radios in there. Depending on how old it is, they might be slow. You got this big firehose of TCP trying to speak through this little tiny straw over the last mile radio network.

Russel:  The other thing too that a lot of people don’t understand is that radios fundamentally are serial devices. When I’m doing TCP over a data radio, the radio can really only handle one communication session at a time.

That might handle really small sessions for very short periods of time and handle a lot of them. Fundamentally, you can only handle one communication session at a time.

Scott:  If you’re using the, I’d say, newer just compared to the old serial radios, I mean, there are TCP mesh network radios that stay hide or obscure a lot of that. However, you still have the reduced bandwidth. You don’t have 100 megabits over radio unless you spend an awful lot of money on your radio.

Russel:  That’s right. If you do those, typically, they’re point-to-point, they’re not canopies. They’re microwave point-to-point that is kind of like building an Ethernet backbone in the air, but you’re not going to use that for every single radio in a field. It’s just not feasible.

Scott:  They’re not going to have the power for it.

Russel:  One of the key things to remember about any kind of field communications, they’ve got to be engineered. They’ve got to be monitored and managed if they’re going to be optimized.

Scott:  With the bandwidth problem, as soon as you remember that there’s effectively a low bandwidth serial-like communication in the loop, all the things you learned of tuning old serial radio networks, they still apply. They’re a little different, but they still apply.

Russel:  That’s exactly right. Just because you’ve got a TCP radio and you hook it up, doesn’t mean you’re going to move all the data you think you are going to move. I’ve seen that happen over and over again. Radios, by their nature, are complex. The other thing is that a lot of people miss the difference between doing polling to operate versus doing polling to collect all the measurement data. That’s pretty fundamentally a big dang deal.

Scott:  Yeah, depending on the protocol involved. Different EFM collection protocols handle this differently, but some require fairly reliable communications to get the EFM data to come in. Even if your communications are even a little bit iffy, you’re just not going to collect the EFM. There’s a sufficient number of transactions that have to occur to get the complete EFM record. Data is not the thing.

Russel:  Data is typically little bitty packets of data, and EFM is the great, large dataset.

Scott:  Multiple records to get right in. Yeah, all that.

Russel:  The other thing we got to talk about, too, because a lot of times people are working on this stuff. This is a good example. They’ll ask, “Does your product support OPC?” That sounds like a simple question.

Scott:  It used to be.

Russel:  Yeah, it used to be, but it’s not really because now there are multiple different versions of OPC. There’s OPC DA, which is the old stuff. There’s OPC UA, which is universal access, which eliminates a whole lot of the connecting over a wide area network issues. There’s also OPC for alarms and OPC for events and all these other kinds of versions of OPC. It’s not as simple as, “Is it OPC or not?” It’s, “What kind of OPC is it?”

Scott:  Right. The nice thing about UA, it supports your modern Internet stuff. You can have certificate authentication authorization on both ends, so you can securely talk OPC UA, whereas DA doesn’t do that.

OPC UA is defined by a TCP specification. You can have a Linux box or Windows box, or you can go build a field device that spoke OPC UA because of the nature of the standard, whereas with OPC DA that doesn’t apply.

Russel:  Exactly. What are the kinds of differences, as you find between the different SCADA tools, the different SCADA platforms?

Scott:  As we talked about, there’s the redundancy, how easy is the redundancy to set up, and how seamless is it to the user, the guy sitting on the console.

What’s another big one? The scalability of the SCADA system. Some SCADA systems split their functionality into separate pieces. You can run those pieces on multiple machines and you can have multiples of each. You can scale pretty well.

Russel: Yes, to very, very large — to millions of points or in billions of transactions a day. That is doable in some of these environments, the ones that have what I would call a distributed services architecture.

Scott:  Sure. There are other ones that are monolithic. I’m happy to be corrected, but in my opinion, with the monolithic ones, redundancy is easier.

Russel:  Yeah, I would agree with that. Absolutely.

Scott:  Because everything is in one place. All your eggs are in one basket, and then you have swapped to another complete basket, and it all goes with it.

Russel:  The trade-off is that the monolithic tools are not infinitely scalable but the trade-off is they’re much simpler to maintain from a pure infrastructure standpoint. The things that are infinitely scalable are much more difficult to maintain and operate because of the complexity they have to introduce to have that infinite scalability.

Scott:  Let’s say you have a very large system, and you have 10 servers that manifest the entirety of your system. How do you do redundancy on that?

Does each of those 10 servers have its backup redundant one? I can think of a number of different architectures, but none of them are simple. How do you deal with scalability when you have a 10 server system and all 10 servers actually working?

Russel:  It’s a really interesting question, and it gets to one of the big platform differences is, what do you need in the way of people to support and operate? Because some of these systems, once you get them up and running, they pretty much run. They don’t need a lot of attention.

We have customers that one person supports the entirety of the SCADA infrastructure and of those small systems. They don’t spend a lot of time on it because they just turn them on. They just run.

Scott:  You can set it up and even launch.

Russel:  There’s other customers that have whole teams and teams of people to support all the infrastructure and keep it all running the way it needs to run.

Scott:  Obviously some of that factor is the amount of churn or change you have in your system. If your system is relatively static, you can get by with a smaller staff.

Russel:  No doubt, no doubt. The other thing I would say is every platform has a lot of subtle differences. At the end of the day, when I build it and I’m interacting with it, that they might be very similar, but for the engineers and technicians that are building out the screens and maintaining the system, and configuring the alarms, those interfaces and how they do that can be very substantially different.

Scott:  How that can manifest is, there are some systems that, when used in the best practice way, make it very easy to be very consistent with how you build your system. There are other ones that are in the middle. Then there are the ones that just make up a bucket of points, and do what you want to do. The system doesn’t care.

Russel:  Oh, my gosh.

Scott:  You’ve got the gamut. How maintainable your SCADA system is. No matter the tool, if your team’s not committed to doing it, it’s not going to happen.

Russel:  A lot of times, even if your team is committed to doing it, it’s still bad, come difficult.

Scott:  Sure. That’s where you have it. If your tool supports it and if your tool promotes being consistent, it will help.

Russel:  That goes very much to how the configurations are managed and how you build and store and keep the various widgets that you’re creating to deploy the system. That is a mouthful. I can’t tell you how many times I’ve gone in and started looking underneath the covers of a SCADA system. When you look at it from an operator consult, it looks really clean and polished. Then you pull open the covers and you look under the covers, you go, “Oh my God. I’m sad. I’m sorry I looked at that.”

What tends to happen over time, as one person builds it, somebody comes along and maintains it, a few years later somebody else comes along and maintains it, over time, they accumulate deadwood. The consistency gets “I’ll just make this one little change here and we’ll get this working again.” Time rolls by, man.

Scott:  That’s where my previous statement about what the best practice of the SCADA system comes in. You take it with a little grain of salt. There are SCADA systems that promote consistency, and I’ve seen messes built with them. There are SCADA systems that that’s not really part of their architecture, but I’ve seen systems that are well constructed with those products. It’s way more about the team.

Russel:  That’s right. You can take a system. Exactly. It’s about the team and the leadership and its commitment to its practice. Very true. Very true.

The point I was trying to make in that is that even if SCADA systems look the same from the outside, they’re not the same from the inside. One of the considerations is always one of the people that are on my team familiar with because making a change to a different platform, there’s a large cost associated with that beyond just change. There’s all the retraining and skill development for the team.

Scott:  What I refer to as the best practice approach. That’s not always obvious at first glance. It might take a couple times to really own what that means and leverage it. You read all the materials and go, “Oh, yeah, well, this is how we’re supposed to do it. Version 2.0 is going to be a whole lot better than 1.0 was.”

Russel:  That is so very true. Maybe, what we’ll talk about here is what is the difference between a SCADA platform versus a SCADA application. This is something that a lot of people that don’t work around SCADA don’t really understand.

Scott:  The SCADA platform, that’s basically your product. I’m going to use ClearSCADA and Cygnet, AVEVA, Wonderware, Archestra, Ignition. I’m going to choose my vendor platform. Most vendors call them platforms, so use your vendor platform. That’s the toolset.

Now, your application is “What widgets am I going to build? How am I going to build my screens? Where do I put my menus? How do I do my navigation?” That’s using the tools to build your SCADA system, your application. It’s the next step beyond…You pick your tools, and then you spend the money, but now I get to add these thousands of points and I get to build these 200 screens.

Russel:  I got to build by valve object and my pump object and my pressure point, my custom trends, and on, and on, and on.

Scott:  Many of these scattered platforms have examples that you can use but they’re almost never exactly what you need. Think of it more as a tool for learning how the product works, how the platform works, not necessarily as widgets that you’re going to end up using. You might, but SCADA systems of any size, you’re going to build your own widgets to work the way you want them to work.

Russel:  You’re absolutely right. I’ve never seen two organizations that could agree on the right way of doing it.

Scott:  That’s for sure.

Russel:  You take two gas pipelines with very similar operations, and they could have fairly distinct and different ways of building out their SCADA screens.

Scott:  What’s interesting, though, is if the people building those two applications are strong in the tool, they both could learn a lot from each other.

Russel:  That’s right.

Scott:  “Oh, feature A. Oh, what’s that? I haven’t used that feature before.”

Russel:  That’s right.

Scott:  “We’re using this feature C. I’ve never seen that before. That’s pretty slick. I want to incorporate that.”

Russel:  That’s a true geek conversation right there. It’s like, “Oh, there’s this cool new little thing I can do. Look what I can do with the animation.”

Scott:  Don’t get bit on the fancy graphics. [laughs]

Russel:  Versus what the operator actually needs to do their job, right?

Scott:  Right. Don’t bite on the fancy graphics. [laughs] I got stories there too back in the early days.

Russel:  Oh my gosh. When I first started putting the animation into computer HMIs and Wonderware and such, probably back in the mid-late 80s, people would build stuff into these SCADA screens. The first time you see it, you go, “Oh, that’s really cool.” About the 15th time, you saw it, it’s like, “How do you turn that off?” [laughs]

Scott:  One of the guys that I worked with, imagine you didn’t know what this was, but somebody said, “Oh, I built a pig launcher screen.” Let your imagination run wild and yes, that’s what we built. [laughs] It was fantastic. It was like an Easter egg in there. It wasn’t on all the time, because we knew it wouldn’t fly. “Look at what these graphics can do!”

Russel:  The controller says, “That’ll happen when pigs could fly.” You go, “Here it is. Hit this button right here, where you watch a pig fly.” Anyways, now we’re getting off in the weeds.

Scott:  Just because the graphics let you do something, just because you can do it, doesn’t mean you should.

Russel:  Oh, gosh. You need to have a t-shirt made for everybody who ever worked as a SCADA integrator. Just put it on a t-shirt, let them walk in. Just because you can doesn’t mean you should.

Scott:  At the end of the day, can the controllers do their job? Do they have situational awareness? Do they have the data they need? Can they see what they need?

Russel:  Is the user interface clean, intuitive, consistent, low cognitive load to understand it? Pretty much anybody can get a SCADA tool and start throwing valves, and pipes, and stuff around, but to build a really clean UI requires good design skills and a very good understanding of how somebody operates.

Scott:  The guys building your SCADA system would do well to hang out with the controllers. See what they’re doing. See what they’re looking at.

Russel:  They have to translate, though.

Scott:  True.

Russel:  The controllers know what they need, but they don’t know how to ask a SCADA engineer for it. The SCADA engineers know what they can build, but they don’t know how to talk to a controller about it. I’ve been on both sides of that conversation. It’s just the nature of things. I don’t even mean that as an indictment. It’s just a very, very good thing to be aware of.

Scott:  When it’s done well, it’s an observational exercise. Watch what the guy’s doing. Ask him why he’s doing what he’s doing.

Russel:  You got to get into their head.

Scott:  It’s hard. It’s hard to do.

Russel:  Most controllers are very good at building a mental picture in their mind, or a mental algorithm in their mind about how everything’s working. The really good designers are able to get into that mind, understand it, and figure out how to take that and put it on a screen.

Scott:  The newer guys can get to that mental model really fast.

Russel:  Exactly right. Before we wrap this up, we should talk about what are all the things that are not SCADA that are required to have SCADA? Does that make sense? Did you like the question? [laughs]

Scott:  We need to decide what’s not SCADA. So, Measurement.

Russel:  IT support is not SCADA, right?

Scott:  True.

Russel:  But I have to have IT support to run a SCADA system because I got to have networks and switches and servers, and all that kind of firewalls and all that kind of stuff has to be there. It’s not SCADA, but you have to have it to have SCADA.

Scott:  It’s getting better. The big struggle is you have IT, and they manage their server farm a certain way. Their SCADA system servers, it’s a 24/7 always up system. Your IT tasks have to be done more deliberately with more care. It’s a lot better than they used to.

Russel:  It is getting better because there are a lot more IT groups that are getting exposure to 24/7. Here’s the distinction. I think you’re making a really excellent point.

If I’m working in business IT and I need to update the email server, I start that at seven or eight o’clock on a Friday evening. If it goes sideways on me, I have the whole weekend to recover so that Monday when everybody comes in, it’s in place, it’s working, and nobody notices I did the work.

In the SCADA world, if I need to make a major update to the SCADA system, I start that Tuesday morning after everybody gets through their morning shift change.

Scott:  Yeah, in the middle of the day.

Russel:  So that if something goes sideways, I have everybody there. I don’t need to call people out, and I can get it corrected before quitting time.

Scott:  You try to batch it, so that everything’s good overnight.

Russel:  Exactly right. In the perfect SCADA world, that control center operates 24/7, and the SCADA guy never gets called out.

Scott:  There you go.

Russel:  By the way, that world does not exist.

[laughter]

Russel:  This is very topical and timely — the other thing that’s not SCADA that’s required is cybersecurity. That is really a whole set of disciplines about maintaining an inventory of all the potential places you can be hacked from all the places you touch the Internet, all the equipment you have, all the firewalls, the rules on all the firewalls, and just on and on and on. In this day and age, particularly given the recent Colonial incident, that’s absolutely critical.

Scott:  Right. Then don’t forget about communications.

Russel:  Communications is SCADA.

Scott:  I mean, from a security perspective.

Russel:  Oh, right.

Scott:  I guess, it’s radio waves, unless you’re encrypting your radio traffic. Which some of the newer TCP mesh radio networks do. But, it’s open airwaves. It’s like your neighbor across the street.

Russel:  I know, for years, people used to say that it’s Modbus over the air, and nobody’s going to be able to hack that.

Scott:  Modbus is trivial.

Russel:  Well, Clint Bodungen has been on the podcast a number of times. It’s probably been 10 years ago now when he did this. He went to one of the ICS cybersecurity events. He demonstrated a “man in the middle attack” using radio and hacking Modbus.

He basically took a radio antenna and got between the host and the field device and told the field device to fill up and overflow to tank and told the host that the tank was doing fine. He did all of that. He did all of that from scratch in about 30 minutes, downloading some stuff off the dark web.

For a lot of people that was an eye-opener when he first did that, it is a good point. Particularly in pipelining, you really have to think about how you are protecting the edge from being a place that people can penetrate your network.

Scott:  That’s the downside of TCP/IP radios because it’s a full network connection. It’s not protocol-specific serial radio.

Russel:  I’ll tell you something else. That’s not SCADA. It’s probably required in most operating systems centers. Some people have tried to do it in SCADA, but it should never be done in SCADA, and that’s mapping.

Mapping, I could probably do a whole episode just on talking about why this is so. Maps and GIS are really resourceful and useful in a control center particularly in some kind of abnormal operation or emergency response. They should not be built into the SCADA system because it gets in the way.

Scott:  Day-to-day, it doesn’t help.

Russel:  Yes, it’s just not a convenient easy way to interact with data. Here’s the final question. What do you wish every manager knew about SCADA before you started a project?

Scott:  What do I wish before we started the project? [laughs] Because I’ve seen it not so often, if you’re starting a project, inventory what you have in the field. Inventory what communications you have. Inventory your protocols. Get an understanding of what data needs to be delivered outside of your SCADA network.

Russel:  All that’s true, Scott, and I would say a very detailed, stepwise plan of execution that is pretty deliberate about getting it turned on, getting it commissioned, getting it to normal operations.

Scott:  Your point-to-point operations, all those kinds of things, for sure.

Russel:  Yeah, exactly.

Scott: They’re required but even without regard to that, you still have to have it. If your controller has no confidence that the data he’s seeing is the data that’s happening, that’s a bad place to be.

Russel:  No doubt. No doubt. Man, this has been fun for me. [laughs] I don’t know how fun it was for the listeners, but certainly it was fun for me. How about you?

Scott:  It’s good. Stuff’s coming up the whole time you say, “Oh, I remember the project when…”

Russel:  We probably ought to do an episode of “what’s the worst thing that went wrong when doing a SCADA project?” We’ll change the names to protect the guilty.

Scott:  Then you have to say the craziest scariest things that happened to that turned out okay. I think we have to go with that. So far, knock on wood. It’s all come out okay in the end.

Russel:  I remember, oh gosh, this is probably been 15 years ago now. We were doing a number of projects, automating some gas storage facilities. We had hired a bunch of brand new engineers out of A&M, and we’re teaching them our way of doing SCADA.

When it got time to go and do the field commissioning of those SCADA systems, and we would take them out there, and they’re used to building these little screens with the little valves and all that kind of stuff.

They go out there, and these valves are 36-inch valves. [laughs] They’re running pressures up to 1440 psi. They get out there, and their eyes just get, “Oh man, I had no idea this is what we were doing.” Kind of fun.

Scott:  The first SCADA project I worked on, my boss at the time was very canny about these things. It was a water project. One of the rights of passage is to go monitor these big valves, I’m sorry, big pumps that are rated to fill up water towers. They’re large water pumps. Like, “Go make sure it’s running,” because it’s water and compressible, so there’s a pressure relief. When they start up, they throw water everywhere. Fortunately, I saw the other guy get it first, so I’m like, “Yeah, I know. That’s okay.” I got lucky. The other guy wasn’t. Luck of the draw. If they’d got me first, I would have been drenched.

Russel:  Those are the lessons you retain, my friend. I think we’ll end it right there.

Scott:  [laughs] There you go.

Russel:  All right, Scott. Hey, it’s been fun. Glad to have you back.

Scott:  All right. It’s been fun. I’ll talk to you all later.

Russel:  I hope you enjoyed this week’s episode of the Pipeliners Podcast and our conversation with Scott. Just a reminder before you go, you should register to win our customized Pipeliners Podcast YETI tumbler. Simply visit pipelinerspodcast.com/win to enter yourself in the drawing.

If you’d like to support the podcast, please leave us a review on Apple Podcast, Google Play, or on your smart device podcast app. You could find instructions at pipelinerspodcast.com.

[music]

Russel:  If you have ideas, questions, or topics you’d be interested in, please let me know on the Contact Us page at pipelinerspodcast.com or reach out to me on LinkedIn. Thanks for listening. I’ll talk to you next week.

Transcription by CastingWords