Have you ever wondered why, when people talk about cybersecurity, they always seem to be talking about data breaches, and they never seem to be talking about infrastructure and OT—that is, operational technology? Is it because the water supply and electric grid could never be attacked? If your answer is yes, stay tuned. If your answer is no, well, you have to admit it's a good question. And it's one we will explore in depth with my two guests.
Hello, my name is David Hechler. I'm a writer and editor at TAG Cyber. Welcome to a roundtable discussion called “Enough About Data Breaches. Let's Talk about OT Security.” Before I tell you more, let me introduce my guests, who I know will engage in a thought-provoking conversation. Joe Weiss is an expert on instrumentation, controls, and control system security. In the year 2000, he helped start the control system cybersecurity program for electric utilities around the country and beyond. The author of the book “Protecting Industrial Control Systems from Electronic Threats,” he is an ISA fellow, managing director of ISA99, a Ponemon Institute fellow, and an IEEE senior member. He speaks often at industry conferences, holds patents on instrumentation, control systems, and OT monitoring, and is a registered professional engineer in California. Mark Weatherford has held a variety of executive-level positions in the public and private sectors. Currently chief strategy officer at the National Cybersecurity Center, from 2011 to 2013 he was the first deputy undersecretary for cyber security at the U.S. Department of Homeland Security. He has been a global information security strategist at Booking Holdings, a principal at the Chertoff Group, chief security officer at the North American Electric Reliability Corporation, known as NERC, and chief information security officer for both the state of California and the state of Colorado. Thank you both for agreeing to be here.
Joe has been pointing out the dangers he sees in operational technology for a long time. He speaks with passion on this subject, as you will see. And at times he has sounded like a lone voice in the wilderness. Mark has had to deal with some of the same issues Joe will be talking about—and he's had to deal with them from the inside. I didn't invite them to have a debate. I wanted to bring together two experts who know a lot about this subject, have a lot of experience—and bring different perspectives. So we will have no lone voice in the Zoom wilderness today. I have no idea whether they will agree more often or disagree more often. That's not what's important to me. I believe they will shed light on a subject that does not receive the attention it deserves. And it's one that we ignore at our peril.
David Hechler: Let me start with you, Joe. What is the cyber security issue that concerns you the most?
Joe Weiss: The biggest problem is that control system cyber security is not being adequately addressed by senior executives. And if it isn't addressed from the top, it isn't going to happen. It's not going to flow downward. That's the big problem. The secondary problem, which is technical, is that these systems were never designed to be secure. And these systems were also designed with features that make them very vulnerable. And you cannot close those vulnerabilities off. So the key to operating the entire commercial industrial manufacturing and defense transportation infrastructure is based on systems that cannot be fully secure.
Hechler: Mark, do you share Joe's concerns?
Mark Weatherford: I do. I don't disagree with anything Joe just said. We still have a lack of acknowledgment at the executive levels of most organizations that security is a problem. You can ask any CEO in the country today if cyber is a priority for their companies, and they will all say yes. But when you pull back the covers, and you look at the resources that are being devoted to cyber security, it tells a different story. That's why I agree 100 percent with Joe. But is that our number one problem? The biggest challenge I see, and it's something that we've been talking about for a couple of decades, is supply chain. We are instrumenting and outfitting our critical infrastructures in the nation with technologies that we don't understand the provenance of. You could go to any company in the nation today and ask them who they're buying their products from. And they will give you very good answers for their tier one suppliers. But if you ask them where their tier one suppliers are being supplied from, their tier two, tier three, tier four, tier five suppliers, they have no idea. And that is a huge gap in our critical infrastructure.
Hechler: Mark, you understand Joe's views, because you two are well acquainted. Are there subjects on which you part company?
Weatherford: You know, Joe is not wrong in most of the things that he's talking about. I approach them from an operational or more pragmatic perspective. Some of the things that Joe is a proponent of I have to figure out as an operator and as a leader in an organization. How would I implement that? How would I buy that? How would I resource that? And that's the gap between Joe and me. And in fact, there are very few people who would disagree with Joe from a technology perspective, because he is light-years smarter than I am—will ever be—on the technology. But I have to implement solutions as an operator. And I think sometimes there is a gap between what is a great idea, and what is an implementable idea.
Hechler: Joe, before I ask you to respond to Mark, are there some common responses you hear when you present your views at conferences, or through the media, or in your blogs—comments or criticism that you have fielded?
Weiss: Number one, most of what I talk about that is dealing with the vulnerabilities at the lowest level—the sensors, the actuator drives—is new to most people. And that includes the engineers. Part of what's happened is when you use the word “cyber,” it's taken on a life of its own to mean networks. And so part of the issue, and this goes to where Mark was talking from, is there are two worlds that are coming together here. One is the operational world, the people responsible for pumps, valves, relays, this operational equipment. But in general, they have been kept out of cyber security. And even more, they often don't even believe it's their problem. Why? Because they believe cyber security is email and networks. And they're going, "How will this affect my pump or valve? Tell me why I should care." The flip side is—and this is the other problem—when cyber security is addressed, it's addressed to the CISO, the chief information security officer. The CISO doesn't own that pump, that valve, that heat exchanger. He owns the networks to them. That's why I said that until the COO, or the CEO makes the VP of operations equal to the CISO—this is just for control systems—we can't get there from here. The technology is not there to fully secure a control system today. Therefore, what's got to be done is with policies, procedures, training, but for both sides—the engineering side and the networking side. And that's where the gap is. That's where we need management to make sure both are in there.
Hechler: Now I'm going to ask you to respond to what Mark said about where he parts company with you.
Weiss: The point I'm getting at is, as an engineer, if you do a good job of engineering, you will be addressing many of the control system cyber issues. I don't believe you need a ton more money or a ton more of anything else to do this right. Part of it is getting people to understand what to do, because it's not that far away from being part of what their normal job should be.
Weatherford: I'm going take some liberty here. I think what Joe is talking about is the concept of security convergence. I know that's a very unsexy term. We've been talking about security convergence for decades. And I know Joe will agree with me on this: You can go to any facility in the world today, go to a manufacturing plant, into a control center, into an oil company and there are very distinct lines between the OT teams, and the IT teams, and even the physical security teams. And that made sense 20 or 30 years ago. It does not make sense in 2021. Our systems are converging. The mythology of an air-gapped system does not exist any longer. This goes back to Joe's very first point. CEOs have not made some really hard decisions and said, "Wait a minute, we need to have one security team with multiple disciplines within that security team." Because there's such a gap in the communications between these different groups. I can tell you that when I was at NERC, we would often find out about events that happened out in the field. Something happened at a substation. The company knew that an event had happened. But they didn't know if they should send a guy with a gun or a guy with a laptop. They knew that there had been some kind of event, but they didn't know the right kind of person to send. And because the physical security teams, and the OT teams, and the IT teams weren't communicating, there was no way for them to know what people to get out there to respond.
Hechler: Do you agree that IT vulnerabilities garner much more attention than OT vulnerabilities? Is there a good reason why that should be?
Weatherford: There's no good reason. But I can give you the reason. IT is much more prevalent. I mean, everyone understands what IT is, you know? The OT community is much smaller. And as Joe said, it's just been in the past decade or so that people even understood what the Modbus protocol was even about. And even today, you go to an IT person and start talking about some of the protocols and some of the OT technologies, IT people won't even know what you're talking about. So again, it's just the blending of the disciplines has not converged the way it should have.
Hechler: Joe, can you give us examples of OT attacks, or OT incidents that people probably have heard about? You’ve been blogging about this a lot, you've talked about the Oldsmar, Florida water incident, and how people jumped all over it as though nothing like it had happened before. But you know it has.
Weiss: Let me start off by explaining what I consider to be a cyber incident. Because there are a lot of misconceptions. In the IT world, the de facto definition is you're connected to the internet, you're running Windows, and somebody is trying to intentionally steal or manipulate your data. And de facto it's malicious. And in the IT world, you also have all kinds of forensics to be able to detect—it may not be right away, but to detect the fact that it's a cyber event. I use one of NIST's [the National Institute of Standards and Technology] definitions. They've got a bunch, but there's one that I use particularly: electronic communication between systems, or between people and systems, that affects confidentiality, integrity, or availability. The key to this definition is the word “malicious” is never used. As an engineer, I don't care if I know it's malicious if there's electronic communications involved. And it doesn't even need to be a classical network per se. The reason I titled my book "Protecting Industrial Control Systems from Electronic Threats"—notice it didn't say hacking, it said electronic threat—is that I have a number of cases where electromagnetic interference, radio frequency interference has taken out SCADA systems or blown up equipment. It doesn't have to be quote unquote “the 12-year-old pimply faced hacker” typing on a keyboard, trying to do something on Windows. Where I'm coming from is if you think about Stuxnet for a minute, people will automatically say, "Yes, there was a cyber attack."
Hechler: We're talking about the U.S.-Israeli attack on centrifuges in Iran.
Weiss: Yeah, it was the cyber attack on the centrifuges in Iran at the Natanz centrifuge facility. The point being that it was in the 2009, 2010 timeframe. It was not detected as cyber until July 2010. For a year, those centrifuges were tearing themselves apart, and everybody could hear them, they were screaming. But until that laptop was sent to Belarus, nobody had a clue it was cyber. They just assumed it was a systemic design flaw in the centrifuges. So where I'm coming from is a sophisticated hacker or attacker is going to make a cyber attack look like an equipment malfunction. In the control system world, we generally don't have the right cyber forensics, except the internet protocol layer. So for a lot of what you're talking about, there isn't forensics. In the control system world, if something happens you can't hide it. A train crashes, a pipe breaks, the lights go out, you can't hide it. What you can either hide or not know is if cyber played a role. That's where it gets very different than IT, where you know—again, it may take awhile to find out. You think about SolarWinds. It took a year or whatever. But when you saw it, you knew it was cyber. And much of what I have, there is not the classical, quote unquote, “network indication” that it was cyber. And one of the things I run into constantly is, "Gee, why do you call that a cyber attack?" Or "Why do you call it a cyber incident?" Because I didn't find any malware. Let me give you one final example of where this really comes in. And then I'll go to Florida. In 2007, the Idaho National Lab did the Aurora hardware demonstration. You know, there was a CNN tape on it. It was physically destroying a diesel generator. It's as if you put sticks of dynamite there.
Hechler: It was an experiment to intentionally demonstrate the capability of doing this.
Weiss: Yes. And by the way, there have been subsequent real Aurora incidents. That was the demonstration. But what I'm getting across here is in Wired magazine, there was an article that said 30 lines of code destroyed a 27-ton generator. That's wrong. There were zero lines of code that destroyed that 27-ton generator. Because this was, if you will, a physics attack. The only thing cyber did was to put that equipment into an operating regime where physics would destroy it. There was no malware whatsoever. That is very different than what IT or even quote unquote “network OT” views as a cyber security attack. Mark, if you want to say anything, I'll wait and then we'll go to the Florida situation.
Weatherford: This might be another area where I don't agree completely with Joe. Joe maintains a database of 1300 or so of these cyber incidents that most people don't agree were cyber incidents. And that is a big point of contention. And this one that he just mentioned—that there were zero lines of code—how do we know that, Joe? I mean, how do you know that that was not a cyber incident? I'm not saying that this database that Joe has, that these weren't incidents of some kind. But here's my position. Three people can barely maintain a secret. And when we have some of these big incidents that dozens and hundreds and thousands of people are aware of, more people would be talking about these things as cyber incidents if in fact they were really this type of cyber incident. So that's my response, Joe.
Hechler: Let’s get back to that water threat.
Weiss: I have over 100 control system cyber cases in water and wastewater. The first one was the Maroochy Shire case in Australia, where you had a disgruntled ex-system integrator who remotely opened a sewage discharge valve 46 times and dumped I think it was a million litres of sewage on the grounds of a Hyatt Regency in Australia. There was a case in Spencer, Massachusetts, in 2007, where they had a sodium hydroxide over-fortification and 140 people went to the hospital. They had to boil water for four days. Was it a control system cyber incident? This was putting a programmable logic controller [PLC] in manual operation. So it basically didn't stop at any of the alarms. That's one that really did happen and was a control system cyber incident. Oldsmar was not very sophisticated. But what happened was that this case was made public because the sheriff held a news conference.
Hechler: Is that unusual?
Weiss: Very. Most of the cases never see the light of day.
Hechler: Why is that? And why was this one the aberration?
Weiss: I can only speculate. My speculation is that the sheriff held a news conference. DHS and the FBI didn't get there in time to stop him from holding a news conference.
Weatherford: Well, now you're implying that they would have stopped him. And I'm not sure that's the case. I agree with Joe on this, though. There are probably a lot of incidents like this that happen, and that never get reported. Because somebody catches them early enough. And it gets—I’m not saying that this is a nonevent, but it ended up being a nonevent. An operator caught the issue and was able to remedy it before anything bad happened.
Hechler: Mark, from your perspective as someone who's been inside the government, can you talk about who investigates these kinds of OT cases? And how often they become public information? And why we don't hear about more of them?
Weatherford: I think it depends on the vertical of the critical infrastructure. Certainly in the electricity industry, which I'm most familiar with, the Department of Energy (DOE) and NERC coordinate on those events. I've been out so long, I can't remember the name of the report. Joe knows what it is. DOE requires utilities to file a report anytime there's a cyber event. Whether you think that the critical infrastructure protection standards for the electricity industry are good or bad, I don't think that anyone would argue that the electricity industry isn't better off today than they were before the CIP [critical infrastructure protection] standards. The problem is that the electricity industry is the only one of the 16 critical infrastructures that has any kind of federally mandated cyber security standards. To your point. that's a problem from a reporting perspective. In many cases, companies are not required to report cyber security incidents. That's one of the things in the current executive order that President Biden is considering right now: to require certain industries that service or supply the federal government to have mandatory reporting requirements after a security incident. So why doesn't that happen more often? Well, I think it's obvious. Bad publicity is not good for a public company. And so a lot of the cases that they're able to remediate quickly enough: no harm no foul, so no reporting occurs. Is it right? I don't know if it's right or not. To the other part of your question, the government gets involved in some of these. DHS gets involved in a lot of these cases, and DHS has a program specifically so that if a company is having a problem, has had an incident, they can call DHS, call US-CERT and report a problem. And US-CERT assures the confidentiality of that report, so that they can help them mitigate it. And it's been a very successful program. They're able to help remediate a lot of public sector security problems that they would never know about if they didn't have this vehicle for confidentiality.
Weiss: I'd like to mention one thing. And this actually goes back to what Mark first brought up about supply chain. The Executive Order 13920 came out because there was a Chinese-made transformer that had hardware backdoors preinstalled coming from China.
Weatherford: Joe, Is that a fact?
Weatherford: I've heard you say this. How do we know this? It hasn't been reported.
Weiss: Two things, without naming names. Number one, I was on a call with people who were physically at the substation where that transformer was, as it was being installed. These were the people—
Weatherford: Wait, wait, wait wait. The way I understand it, that transformer never left the dock in Houston.
Weiss: No, that's where you're wrong. That's what's missing. There were two transformers involved. The first transformer was installed in the WAPA [Western Area Power Administration] Ault substation, Mark, not far from you, outside of Denver. It was installed in August 2019. When WAPA was doing the site acceptance testing, the mechanical and electrical engineers found the extra electronics in that transformer.
Weatherford: I didn't know that. I have not heard this. How do we know this, Joe?
Weiss: I have pictures of both transformers—Ault and Houston. As a result of that, the next transformer that arrived at the Port of Houston in early 2020 was intercepted by DOE and taken to Sandia [National Laboratories]. There is a utility missing a transformer. It would have never, ever happened if DOE wasn't so concerned about what they found with the first. What’s missing is what DOE found at Sandia.
Weatherford: I need evidence, Joe. I need evidence. I mean, you saying it doesn't make it true.
Weiss: Mark, you were within the government. Go ask DOE. I've got, I can read you—I won't even mention the country—an email I got from one of our closest allies. From someone very senior. And it's saying, "I am hoping you can help me with something. Regarding the transformer issue you discuss, can you please tell me to what level that information is confirmed?" This is from one of the highest levels of one of our allies, because they have one of those transformers there. And I'll go on to read: "I, with our bulk power transmission organization, came up with hypothetical reasons for Executive Order 13920. Our conclusions were very close to (in several parts exactly) what you have in your slides. But for us, it was unconfirmed. Did you go through a similar process to us, i.e. brainstorming why such an order would exist? Or are you at the quote 'knowing stage' rather than the concluding stage?" Isn't that interesting?
Weatherford: Well, I think you just confirmed my point, Joe, and that is, if they don't know, we don't know. Maybe there's nothing to know. I mean—
Weiss: We have a utility missing a transformer. Mark, that has never, ever, ever happened. You don't buy a transformer like it without an absolute need to have it installed.
Weatherford: OK. I get it. Understand—100 percent agree.
Weiss: It's going to take another year to get another transformer. Now I want to bring up one other point. When you look at Executive Order 13920, they give a detailed list of all of the equipment that is in scope for Executive Order 13920. Every single item in that executive order is out of scope for NERC CIP. Every single thing in NERC CIP, and in the supply chain, is out of scope for the executive order. We have a problem here. This is a real, honest hardware implant. There are over 200 large Chinese electric transformers in our electric grids today. We have no idea how many of them have these hardware backdoors.
Weatherford: My response to that is that if, in fact, there is a problem with the transformer that's been sitting at Sandia for the past year, we could make an assumption that those other 200 transformers have been investigated. And so I want to qualify everything that I've said here. I don't say that this could not happen. In fact, this is that supply chain issue we've been talking about. It's something that we've been worried about forever. But where I do disagree here is that I think you're making assumptions that we can't validate. Just because Sandia—again, it's never happened before, but just because Sandia collected a transformer from the Port of Houston, it's sitting somewhere and they're looking at it, I don't think we can legitimately assign nefarious actions to that. It may be, but we're an evidence-based society.
Hechler: We have almost no time. Tell me what could and should be done—further steps bringing more attention to the security issue for OT. For example, Mark, do you think there should be reporting requirements for all kinds of cyber security events in OT systems that are not now required? Are there other reasons why our country doesn't have a focus on this? If we were in Ukraine, and the power had gone off, and everybody had experienced that, if something like that ever happened in our country, I don't think we'd be having this conversation.
Weatherford: I'm not sure I agree with you. Because I think we've proven over and over to ourselves that we're not smart enough to learn from our mistakes. But yes, I do think we need to have more reporting requirements for all critical infrastructures, things that society depends on for safety and security should have a higher level of oversight by the government. And I don't say that lightly. You know, I am not a regulatory guy. But we have we have seen over and over again that companies do not prioritize security when it comes down to: is it security or revenue? They almost 100 percent default to revenue and not security.
Hechler: The reporting requirements—can you spell them out? And when they report, when the infrastructure companies report incidents, however that's defined, should those reports be public information? Or should these be confidential?
Weatherford: Yeah, I have a little problem with public reporting. Listen, there are all kinds of unintended consequences for reporting. I don't say this lightly, and I think it's often used as an excuse that if we public-report it, then the bad guys will know what the vulnerability was, and they'll be able to do more of the bad stuff. But there's some legitimacy to that statement. So I think maybe within boundaries of reporting, one of the things that we've been trying to do for a long time is get a certain number of people within certain critical infrastructures to have security clearance so they could be briefed on some of these things. And that's a whole other ball of wax that the government has not solved yet. But I get a little bit anxious thinking about all these things being made public.
Weiss: I don't believe the details should be made public either. But where I come from is enough information so that quote, unquote “the good guys” know what to do to protect themselves.
Weatherford: Let me give you an example. And I hate to air dirty laundry, but in the electricity industry, we get about 10 percent of our electricity from Canada. When I was at NERC, I couldn't talk to our Canadian companies about classified information that directly affected them. That's the absurdity of the classified conversations that happen.
Weiss: It was the same thing with Aurora. Aurora was classified for official use only, which meant that the people in the field who didn't have clearances were not informed.
Weatherford: And that was even after it was on CNN, for crying out loud.
Hechler: Joe, if you had one suggestion to make that could help advance this issue and improve the way we handle it in this country, what would it be?
Weiss: You set up a system like what was set up in the U.K., where the government takes a backseat, and it gets the industry or those involved together and they share the information. It works. When I held my conference—I no longer have it, I sold it after 15 years—it was the only conference where people who actually had their control systems impacted by cyber would speak. Who were they? The engineers. Because they wanted to know if anybody else had seen the same thing they had. I had spent 15 years of effort. The whole point was the information sharing. The term cyber has caused more grief than you can shake a stick at.
Hechler: Well, the NCFTA [National Cyber-Forensics and Training Alliance] in Pittsburgh, and CISA, the Cybersecurity and Infrastructure Security Agency in D.C., are supposed to be doing this. I have heard lots of good things about the NCFTA. And I know that CISA has been trying to provide those kinds of services and has worked hard to facilitate communication. And the Solarium Commission wanted to pour a lot more money into CISA to help the cause. Do you think those two organizations can make a difference, if they're given more power and more resources?
Weiss: I'm only looking at incidents, not looking at vulnerabilities. And I even had this in my book. If the government is directly involved, industry is not going to share real incident information. Good, bad, or indifferent, they don't trust that that information will stay there. Part of what I'm saying is the government needs to provide the wherewithal to allow people to share the information. The engineers want to. The security people don't like it, because the engineers want to talk. They want to share the information. And it's this Catch 22. How do you share, and how much do you share? And to whom do you share?
Weatherford: Exactly right, Joe.
Hechler: I want to thank you both for agreeing to be here and for sharing your insights, and your disagreements, and your agreements. We didn't exactly have a debate. We didn't exactly have a love-fest. But I think there was something that our audience is going to learn from what happened here today.
Weatherford: We agree on far more than we disagree on.