Recently, one of our technical advisory board members told me that a CEO at a complementary technology company felt I was throwing rocks at passive solutions in the ICS space.
Nothing can be further from the truth. While we passionately believe that most companies will benefit from our Active OT protection solution, we do advocate a layered approach to security; it’s a familiar refrain to all the people with an enterprise background. I’ve been touting this for over 20 years: it takes a combination of people, process and technology.
A layered approach starts with human capital and training. From there, process and tools that can complement each other in allowing the overall solution set to identify, contain and mitigate any threat in the cleanest manner, with the lowest number of hops that the human capital has to make between solutions, usually results in the quickest resolutions with the fewest numbers of mistakes.
The ICS space that many of us find ourselves in is fascinating, challenging, frustrating and, for some, will be the most rewarding of our careers. I speak to you as someone who has been in the technology field since the early 80’s, so I bring the experience of watching several technology revolutions that were unexpected. I would like to relate some of those experiences and then apply them to our current market to give you some thoughts on the way forward in the ICS planning and protection space.
First, let me start off by reminding you I am the CEO of an ICS security company, Bayshore Networks. All of my competitors are younger, smarter and most likely better educated than me. This will be my 7th start up, first in the ICS space, and second as CEO or COO. What I bring is a ruthless pragmatism of what I have seen work, and experience in human behavior when responding to new technology.
Give me a few minutes to bore you with some experiences I have had in the tech industry in general, and I promise to tie it back to where we are today.
Remember Toshiba introducing the T1000 and T3100 laptops in the mid 1980’s? It was all the rage for several years. I worked in that division, which was the Dot Matrix Printer group. Yes indeed, it was the printer division that introduced laptop computers to the masses. A well thought out product mapped to the times? Nope. The laptop was kicking around Japan as a science project, and couldn’t find a home.
That Dot Matrix Printer division was actually started 3 years prior as a word processing solutions group based on the, wait for it, CPM standard. Well, that launched as a thud. And the dot matrix printer that took that group from $0 to $50M in sales in 30 months, was also a fluke. In desperation before the division was shut down, the GM found a 24-pin bidirectional printer created for a very specific application, and saw its ability to compete with the standard daisy wheel printer of the day. It wasn’t this long planned out product process to introduce 24 pin printers, it was another accident.
I remember sitting in the office of the GM of that group, John Rehfeld, who had steered the division from a flop to a moderately successful printer division. He had brought the laptops back from a trip to Japan, and was huddling with his senior team about whether to have the division introduce the product or not. There was great hesitation in the room. You see, these guys were all in their mid to late 50’s (younger than I today), and had almost lost their jobs a couple of years ago with the CPM fiasco. In those days, down in Orange County there were few, if any, other tech jobs, and they were afraid of stepping on another land mine. When John completed the circle of discussions, he looked at me (I was a 27-year-old Sales Analyst who kept the VP Sales in line) and he said, “What do you think”? I pointed at the T3100 which had a liquid crystal plasma screen and 10 MB hard drive and said, “I think we will be selling 10K units a month of those, and 15K units a month of the T1000”, a floppy based unit. The room exploded in laughter with these senior guys waving their arms motioning me way. Once the noise died down, John asked me why, and my answer was simple: “You think you’re going to sell 200 a month of these to Engineers going between meetings, but your admins are all fighting over the few prototypes of the hard drive units because of the ergonomics of the size, they don’t have to have a huge desktop computer on their desks, or at their feet.” The benefit was technology creating ease of use for the user, and allowing the completing of assigned tasks easier.
To his credit, John did push that division into the laptop business, and it became history. In less than 18 months it had grown to almost $500M a year. Think about it, a division that almost went bust in less than four years was the hottest item in high tech, it was mostly an accident, and it almost didn’t happen because of fear. But it succeeded for one reason, it solved an immediate ease of use problem for the masses with low risk. Keep that in mind.
Fast forward to 2002, and I was the VP of Sales for Counterpane Internet Security. Some of you will remember them as a pioneer in SIEM development, correlation techniques, alerting and mitigation. When I started with Counterpane, Morgan Stanley had just invested in the company. I flew to New York to pitch their CSO on becoming a customer, sending their logs to our SOC for correlation and alerting. NO WAY they told me; we know we invested in you, but we will never send our logs out of our network. All the shops in that vertical told me the same thing. Three years later, they were all doing it. Why? Because, they realized that although they all had their own SOCs, and with the threat Intelligence of the day, the bigger risk was going down the street with blinders on. You could see your network, and only your network, but you don’t see the attack coming at 100 miles an hour down a side street that is about to end your life. What a good MSSP provided them was the hope that one of the MSSP’s other customers was the zero-day attack victim, and the MSSP was quick enough to alert you of the incident and how to mitigate. Again, fear was overcome by the knowledge gained that extraction of logs could be done securely creating the ease of use to getting to a satisfactory solution once basic security fears were overcome, and adding significant value to a company’s security posture.
Now we are in this incredible space of OT/Critical Infrastructure. It’s important, people can and do get hurt when things go wrong. Plant managers have been concerned with two things in the past, and two things only, safety and uptime, and that is how they are compensated.
But the digitization of the plant and critical infrasture calls out for a robust layered approach to this unique environment. I joke about that famous chart that shows something like 50-60 security products that the average Enterprise CISO deals with, and say the Plant Manager maybe has one, a firewall, and she doesn’t even want that!
I don’t know Dawn Cappelli, VP Global Security & CISO at Rockwell, but I saw a great interview with her, which was posted on LinkedIn. I am going to try and paraphrase what she indicated, that as a CISO she would love to be able to map her current security providers over to her OT environment - reduction of fear and increased ease of use - but she knew that wouldn’t be completely possible. What she did know was that the paradigm of a layered security approach would be transported to the OT space. I couldn’t agree more.
Today, ICS security is the hottest space in the security technology market. Many of the leading players are out the Israeli Intelligence Corps Unit 8200 responsible for collecting signal intelligence (SIGINT). They are the best and brightest of the best and brightest. But that doesn’t mean they are not guilty of some form of group think.
The ICS space today is dominated by Passive Asset Management companies that inform you of what you have in your network, create a baseline, detect violations against that baseline, and then optionally send an instruction to a perimeter firewall for rule changes.
All of these companies have their selling points, what they claim to do better than the others, and it’s not for me to comment one way or the other about them. But, the basic premise of the solutions are the same.
The thinking, of course, is that you can only do “passive enforcement” inside a perimeter firewall because no one would ever want to enforce a policy below that point, right? I mean, no one in his or her right mind, correct?
I want to tie this long message back to confronting fear of the unknown, and addressing it with a solution that provides the ability for teams to do their jobs better and more effectively.
I was on a panel at the AGC event alongside RSA this year with many of my very capable competitors. When I asked when they went into a claimed air gapped network if their asset management software found things going on that the plant manager had no idea about, they all jumped up at the opportunity to agree that their technology uncovered all kinds of unknown sins going on behind the firewall.
That’s the point of having Active Protection.
Most of what they found was functions that the plant manager NEVER wanted to happen, great, set a policy and enforce that. You don’t want that workstation talking to that HMI, you don’t want this data register value for this furnace to exceed this threshold under any command, no problem.
You see claims you never want to enforce a policy inside a perimeter firewall in an OT network is rumor and myth propagated by well-intentioned people out of fear of the unknown. They can’t do it, so they discourage people from wanting it.
When using Active OT, I tell people we aren’t here to stop your plant, we are here to make sure it doesn’t blow up.
I also want to point out that the two most active penetration devices into critical infrastructure are here on my desk. My cell phone and my laptop, both of which gain access to critical infrasture when I am walking though most plants regardless of any perimeter firewall or passive detection tools.
I don’t allow negative selling by my team; we don’t throw rocks. Besides the fact that your parents wouldn’t want you to do that, I never have believed that it engenders you to the customer you are trying to influence.
We tell people what we do and why; it’s up to them to decide. We offer, at our core, a policy engine that essentially operates as an IPS. Yes, you can just use it in alerting mode, but the customers who have been using it the longest have gone into in line policy enforcement.
Why, because we have been able to create a huge reduction in noise level by using Active OT solutions so the few OT/SOC analysts you have can focus on the alerts that truly need human intervention. We reduced the fear and increased the ease of use of security, just like the admins wanting a laptop 30+ years ago, or CSO’s starting to share logs for correlated analysis 15 years ago. This pattern in tech keeps repeating itself every 15 years or so, you just need to have been around long enough to have recognized it before.
I do believe in human capital, good training, good threat intelligence, and having a great service provider to help you through the rough spots. A great playbook to deal with incidents is a must, and there are a few platforms out there today that fit these bills, Dragos comes to mind, and there will be others eventually, it’s the capitalist system at work.
But in the heat of an attack, in milliseconds when someone is attempting to break in, or has broken into your network, you want to limit damage at the point of the attack, you need to use tools to respond in electronic time, not only in human time. Think about that.
What do these ramblings mean?
- The ICS space is hot with a lot of charismatic CEOs taking center stage. They are all the brightest of the bright, well-educated and passionate about the space. But they all have taken money (just like we have), and have investors. No one is pure.
- The axiom of no active protection in an ICS environment will most likely not survive capable products that create a combination of workload reduction and increased effectiveness of an organization’s security staff. Lean into the thought process.
- Common sense dictates the following:
- Not all companies have the human capital, or can afford or retain the human capital to solely go the passive solution route.
- Many installations in the ICS world are remote, and have no onsite human capital for attention, and thus need some type of active protection.
- The paradigm of a layered approach on the enterprise side works for a reason. We don’t advocate using our active solution as a standalone panacea. We recommend a well thought-out defensive posture that includes:
- First and foremost, training and a playbook of what to do when things go wrong. Think of it as being the Captain of an airliner when trouble hits, you go to your checklist.
- Perimeter and segmented plant protection with a combination of firewall and IPS devices with endpoint protection thrown in.
- Threat intelligence. If you really don’t have the capability to utilize this then a great MSSP. If you do have the capability, take some of the free feeds from DHS as well as one of the paid services.
- Be honest with your capabilities, and if you can’t deal with this on a day to day basis get a good service provider, they are worth their weight in gold IF they have an ICS practice.
A great layered approach with training, support and passive as well as active tools, will provide the best way forward to ensure that you meet your safety and uptime goals. No one company has the complete answer.
There is no such thing, in most instances, of an air gapped network, that is the reality.
Pitch for Bayshore: Take a hard look at the modular approach we have taken in creating vertical solutions off our policy engine that can stand alone, or integrate together as a comprehensive solution for communication, identifying assets, detection, protection, all on one pane of glass.
Beacon : Secure Remote Access
Compass : Unidirectional software based data diode
Lighthouse : Asset Management, Detection, Alerting and Policy Enforcement
Lighthouse Scada : Use in either remote, or low staffed networks, operating completely without touch
If you have been able to read this article, first, thank your first-grade teacher!
Finally, if you did read this post, then you are in an industry that is paying you well. Regardless of what you think of my thoughts above, if just two of you this weekend does one of the following my efforts are worthwhile:
- Give a 50-pound bag of food to a local pet rescue
- Go to your local SPCA and take the dog who has been there the longest out for a walk
- Take a coat, or some books, or toys down to your local homeless shelter
- Offer to tutor a child, it can be done online, my wife does this 5 nights a week. If you want us to show you how, send me a message on LinkedIn
Enjoy the ride in this space, the opportunity to be in a space growing like this does not come your way very often, take it form a guy who has been doing this for over 35 years. Enjoy the moment.