Unsolicited Response

Maggie Morganti joined Dale Peterson on the show to discuss how International Humanitarian Law (IHL), which almost all countries subscribe to, would treat cyber attacks on critical infrastructure.

• The terms distinction and proportionality (the latter is very important and is on the proportion of civilian to military damage) and how they affect what is permissible under IHL
• How sabotage like Stuxnet or Triton is treated by IHL, and the concept of behind enemy lines - Does IHL mean anything when the most capable players do not agree or exhibit restrictions?
• And a lot more

This is a hard issue and important topic, and Maggie’s master’s thesis is the basis of the discussion.

Preparation and Persistence Paper 

Video of Dale's comparison of cyber conflict situation to pre World War I cult of the offensive 

Part 1: Awareness of Purdue Level 0 and 1 (In)Security

Part 2: Properly Prioritizing Level 0 and Level 1 Security

In this third and final article in my Level 0 / Level 1 security series the focus is on the appropriate security controls.

The security concern with sensors is that the sensor data will be incorrect and lead to incorrect control decisions. Sensors fail for a variety of reasons unrelated to a cyber attack, so this is not a new issue. However, an attacker with engineering skills and automation skills is more likely to know what type of false data could lead to high consequence control decision errors. A simple example would be spoofing the data so the Operator and logic thinks everything is operating normally, when in fact the process is entering a bad state.

Bad sensor data could be injected at the sensor itself (Level 0), communication networks between sensor and PLC (Level 1), at the PLC, communications between the PLC and the Level 2 computers, or in the ICS applications at Level 2. As noted in Part 2, the exposure to a cyber attack is greatest where the device or network has an IP stack.

Ideally we would like to have authentication of the source and sensor data integrity along each step of this communication path, and hopefully we will eventually get to having this. In the meantime, the solution where the risk of false Level 0 sensor data is unacceptable is process variable anomaly detection (PVAD) on reported sensor data.

The best example to date of this is GE's Digital Ghost, originally shown at S4x19. GE had a digital twin of a GE turbine and the control system. After training the twin with data from operations, GE was able to identify when specific sensor data did not make sense based on the state of the process reported by other sensors. Digital Ghost then calculated what the nonsensical sensor value should be and sent that to the actual control system where it could be considered by the Operator or automatically corrected in the HMI and system.

Importantly this solution deals with bad sensor data regardless of the cause, sensor failure, cyber attack or other.

The GE example is in some ways the simplest one. GE makes the physical product, the control system, often deploys the solution, and has a somewhat standard deployment. It's why they had digital twins for these turbines before the term digital twin existed. The growing benefits of digital twins is leading to exponential growth in their deployments across vendors and integrators, and the ability for this data to be used for PVAD is another benefit.

The other method for detecting sensor data errors today is to have a separate Level 0 monitoring network and compare the sensor data reported up to Level 2 with the data received on the Level 0 monitoring network. Vendors such as SIGA OT Solutions, Cynalytica, Fortiphyd, Mission Secure and others are offering this. Importantly, some are also touting PVAD capabilities which obviates the need for this new monitoring.

The cost of deploying and monitoring a second network for something with the lowest exposure to cyber attackers is difficult to justify from an efficient risk reduction criterion. I have been interested in the possibility of monitoring only a small percentage, perhaps 5% or less, of the Level 0 sensors through a separate network. Could machine learning identify what 5% of the sensors could detect a Level 0 cyber attack? It likely wouldn't be as simple as selecting the most critical 5% of the sensors. I imagine it would be sensors distributed over the process and with certain correlation results related to high consequence events. This is a good research project.

As noted in Part 2, new sensors with an IP stack should have the same security controls as listed in the actuators below.

Actuators are the end point that demonstrates the insecure by design problem. They will do whatever they are told to do, within their operationally deployed capabilities, regardless of the source of the command.

There is no authentication of the source of the command. There is no authentication of the integrity of the command. This is what the security controls need to change.

(Remember from Part 2, adding security controls to new or legacy serial interfaced actuators is deferred, look at it again in 3 to 5 years.) These recommendations apply to new actuators with an IP stack and are the minimal set.

1. Signed Firmware / Secure Boot - This is to prevent simple DoS / brick attacks with a corrupted firmware upload, as well as more sophisticated attacks that actually put attack code or hooks in the firmware. The beauty of this control is it requires no user action. (And no this would not have stopped the SolarWinds incident or anything else that has compromised the vendor's firmware creation and issuing process).
2. An administrator role and login capability where over the network actuator administration is possible.
3. Authentication of any control commands - Until recently this would have required a vendor to come up with a proprietary solution. Now there is CIP Secure, Modbus/TCP Security, and other ICS and IIoT protocol security. For better or worse, most of the security consists of wrapping the protocol in TLS. This does require some PKI / key management to be effective. At a minimum the vendors should start with something simple, and less than perfect, where part of the commissioning is to put an asset owner signed certificate on each actuator.

The wrap-it-in-TLS decision does add complexity to the Level 0 security problem. If Level 1 is doing anything more than forwarding packets, it will need to decrypt the wrapped packet, and then potentially encrypt it again to send on to Level 2 or other Level 1 devices.

Of course there are many more security controls that could be specified and could be helpful. Others who leap from insecure by design to secure by design lean heavily on the importance of security development lifecycle (SDL) requirements. The fuzz testing of the protocol stack's that began in the '00 decade helped a great deal with protocol stack robustness and will only help the vendor with the product lifecycle. This list of three security controls is the bare minimum in terms of addressing the insecure by design problem.

New PLC's are Priority 1 in the Level 0 / Level 1 security decision tree for adding security. Upgrades to high and medium consequence legacy PLC's are Priorities 2 and 3. The security controls listed for actuators are required for PLC's as well. In addition, they should have security event logs and the ability to store and forward these event logs.

Nice to have, premium options for these devices include:

• A deep packet inspection firewall to restrict access to the device from Levels 1 and 2. This is the Tofino, M-Guard, OTfuse or similar integrated into the Ethernet card or somewhere else on the PLC. The only thing stopping this option is the business case.
• PLC endpoint protection - this is an area ripe for research. What is the equivalent of anti-virus for a Level 1 device. We have had proposed sessions for S4, that were pulled back, that would look at the logic / program upload for attack code before loading it.

This three part article series can be summarized as follows:

1. The ICS Security Community understands that Level 0 and almost all Level 1 devices lack authentication. Access sensor data can be modified, and control commands that reach the device will be accepted.
2. The risk of the lack of authentication varies at Level 0 and Level 1 based on the exposure and capabilities of the device. While we would like to have cyber security throughout the entire ICS, it is important to prioritize efforts where we will achieve the most efficient risk reduction.
3. Process variable anomaly detection (PVAD) is the most effective way in the short and medium term to detect and address bad sensor data.
4. Authentication of the firmware, administrative actions, and control commands are the most important security controls to add to the Level 1 and Level 0 devices in the decision tree specified priority order.

Subscribe to my ICS Security: Friday News & Notes email.

Jason Christopher is the lead author of the new paper: Industrial Cyber Risk Management. Dale Peterson interviews Jason on this episode of the Unsolicited Response show. They discuss

• How industrial cyber risk fits into the business risk management strategy
• The proposed addition of resilience to the industrial cyber risk equation
• How much an asset owner needs to know about threat to perform effective cyber risk management (and how often it changes)
• When quantitative risk management will be feasible for industrial cyber risk
• The paper's proposed double heat map.

Industrial Cyber Risk Management whitepaper discussed in this episode

Subscribe to my ICS Security: Friday News Notes email.

We have resolved the issue on whether the ICS security community knows that almost all Purdue Reference Model Level 0 and Level 1 devices, and the protocols that communicate with them, lack authentication. They know this. The next question is what to do about it from an OT / ICS risk management perspective. I'll break the answer into two parts. This article will cover efficient risk reduction prioritization, and next week's article will cover the recommended security controls.

In a perfect world with unlimited resources, all Level 0 and Level 1 devices would have a set of security controls. New devices would come with the security controls and deployed devices would be upgraded. Since resources are limited and ICS cybersecurity risk reduction options are plentiful, deciding the priority of risk reduction actions is important.

This is similar to evaluating the risk reduction provided in applying security patches in ICS. It is a good security practice to apply all security patches that mitigate vulnerabilities. However as shown in ICS-Patch: What To Patch When In ICS, there is a large variance in the risk reduction achieved in various asset / patch pairs. The small percentage of patches that result in significant risk reduction should be applied asap, and the large percentage of patches that result in almost no risk reduction should be deferred and applied primarily when needed to keep the product in a supported state.

Similar to ICS-Patch, a decision tree is a good way to look at the prioritization of securing Level 0 and Level 1 devices, see diagram below.

Exposure is the most important factor in determining ICS risk reduction. It was the first decision point in ICS-Patch, and it's the first decision point in determining the risk reduction achieved in securing Level 0 and Level 1 devices. How easy will it be for an attacker to access, and therefore be able to compromise, the device or the communication to and from the device?

Factor: Exposure... Level 0 or Level 1?

Securing Level 1 provides a leveraged security point with significantly more efficient risk reduction than securing Level 0. The PLC, RTU, Controller or other device at Level 1 typically communicates with many sensors and actuators. It can be a perimeter security device that stops or limits attacks from reaching the Level 0 device from untrusted networks.

Falsified or bad sensor data is often given as a reason to secure Level 0. Yes, a compromised Level 0 sensor could send back incorrect data that could lead to an incorrect and consequential control action or inaction. However, a compromised Level 1 device connected to that sensor could also provide falsified or bad sensor data for that sensor. The compromised Level 1 device could additionally falsify sensor data for all other sensors it is connected to, send rogue commands to all actuators it is connected to, and alter any of its process logic.

Prioritizing Level 1 security over Level 0 security is similar to prioritizing a firewall to limit enterprise and Internet communication from reaching the ICS over securing the Level 1 devices.

Factor: Exposure... Ethernet Port / IP Stack?

If a Level 0 or Level 1 device has an Ethernet port and an IP protocol stack, it is much more likely to be attacked and compromised through cyber means than if it does not.

There are two reasons for this. First, the IP stack makes it much easier to route attacks to the device. As we know the air gap is almost always a myth. IP based networks are connected to share information, and this provides a potential attack path to the device if it has an IP address. Second, most of the attack tools, as well as almost all of the attacks and attack attempts to date have tried to compromise ICS cyber assets via this interface.

Can a Level 0 or Level 1 device with an "analog" or "serial" interface be accessed? Yes. It could be reached from an IP network through a serial-to-ethernet gateway, and this gateway is a better, again from an efficient risk reduction standpoint, place to put cybersecurity. This is the location where you should apply your security controls, whether it be through an industrial security gateway or other means.

And yes, a serial Level 0 device can be accessed by physically connecting to the serial network. This often, but not always, requires close physical proximity to the device and physical attacks on the device or process are typically easier than cyber attacks with this access.

This will be a contentious point with some, and I'm not arguing against security in all devices. The point is we will achieve greater risk reduction by securing the devices in each level that have an IP address than securing those that do not have an IP address, and they should be prioritized.

Factor: New or Legacy

I used the Will Rogers quote "if you find yourself in a hole, stop digging" in a recent article on the legacy system problem. Any new devices with an IP address should have security. At Level 1 there are a small, but growing list of devices with the basic security functions that will be described in next week's article.

If the choices available for purchase lack security then you should either insure there is an acceptable upgrade path to add security or plan on a much shorter lifecycle than usual, such as 2 to 5 years rather than decades.

Which only leaves the question of what to do with legacy devices with an IP address.

Factor: Impact

Legacy Level 0 devices with an IP address fall into the defer category. Even looking 5 years out it is hard to find a case where an efficient risk reduction approach would lead to replacing or adding on security to these devices.

Legacy Level 1 devices with an IP address should be upgraded or replaced in the order of the impact of compromise of their availability and integrity. In general, Level 1 devices that are involved in control, changing the physical component, should be prioritized over Level 1 devices that are connected solely to sensors. There are a number of ways to detect bad sensor data. That said, there are examples where bad sensor data would be difficult to detect and would lead to high consequence actions, and this would lead to PLC's associated with these sensors being prioritized.

The decision tree in this article prioritizes the order to add security in the Level 0 / Level 1 category to address the insecure by design problem. It does not describe when this issue should be addressed in relation to all of the other potential ICS cyber risk reduction activities. It is easy to identify activities that provide more and less risk reduction than addressing the Level 0 or Level 1 security issue. Some examples:

Greater Risk Reduction Than Addressing Level 0/1 Insecure By Design

• Enterprise / ICS Security Perimeter
• Two-factor authentication for remote access
• Ability to recover minimum required ICS cyber assets
• Large scale consequence reduction

Less Risk Reduction Than Addressing Level 0/1 Insecure By Design

• Majority of security patching inside the ICS security perimeter (Defer category in ICS-Patch)
• Individual Operator accounts in a 24/7 control room
• ICS application security controls at Level 2.

Next Week: Appropriate Security Controls for Level 0 and Level 1

Subscribe to my ICS Security: Friday News & Notes email.

This is a slightly edited version of the LinkedIn Live and YouTube Live show Dale Peterson recorded on March 17th. Dale begans talking about the Level 0 issue and was joined partway into the conversation by Ron Fabela. They talk about the awareness of insecure by design that exists in almost all Level 0 devices, the risk related to this, how asset owners are and should be looking at this, and more.

Then the last ten minutes Dale talks about his two favorite Incident Response Tabletop Exercises and what you should expect when you do a TTX.

Solving a problem typically begins with awareness that there is a problem. Back at S4x12 a group of researchers under the Project Basecamp banner demonstrated that most PLC's (Purdue Level 1 devices) were both insecure by design and ridden with exploitable bugs, as well how an attacker could leverage these issues. Nine years later I believe the issue is known in the ICS security community, even if it is just beginning to be solved.

Similarly, ICSsec pioneer Joe Weiss has been on a content blitz the last three years pointing out that Purdue Level 0 devices, sensors and actuators, do not have any security. Primarily with the example that false process sensor data could be presented to the control system. He has asserted that the ICS security community does not know this, most recently in a post SANS ICS Security Summit article.

Joe and I had a back and forth on LinkedIn on whether this misconception was present in the S4 audience, SANS ICS Security Summit audience, and ICS security community in general. My belief was lack of security at Levels 0 and 1 was common knowledge in the ICSsec community and the only disagreements are the prioritization of addressing this issue amongst the large number of issues and how it should be addressed. We tried to answer this disagreement with a poll.

The same poll in my Friday Newsletter had 147 responses with 98% answering False. It is likely that there is some overlap in the respondents to the two polls. 97% is clear evidence that the ICS security community understands there is not security at Level 0, and I'm certain they also know it is exceedingly rare at Level 1.

This does not mean there is no awareness problem. Security professionals coming in from the IT security world often do not know and are surprised to learn this. The bigger problem is when asset owner executives are unaware of this because it can lead to spending a lot of money and effort on good practice security controls that have little impact on risk. It is incumbent on Operations and OT Security to inform executives of risks and the best risk reduction options.

What I'm still uncertain of is how well known the insecure-by-design Level 0/1 problem is in the ICS involved engineering community. I'd like to believe that similar to the ICS security community this is common knowledge. I don't have an audience of engineers who are not involved in ICS security. Any readers with the right audience want to run a poll?

Next Week: The Solution To Level 0 / Level 1 Lack Of Security Problem

Subscribe to my ICS Security: Friday News & Notes email.

Cplane.ai and ExxonMobil recently completed a pilot project showing how orchestration could simplify the deployment of a complex, multi-vendor system. This was actually a test of both Orchestration and the Open Process Automation (OPA) Forum's work.

John Casey of Cplane.ai joined Dale Peterson on the show to explain exactly what the pilot project did and how orchestration could be useful in the Operation and Evolve phases of a ICS lifecycle. Orchestration is common in the enterprise, particularly in large telecom, and new to the ICS world.

John helps Dale grapple with the concept, and there is a bit at the end about the industrial edge.

• Dale's interview with Steve Bitar on OPA
• Cplane.ai video on the pilot project
• Cplane.ai white paper on the pilot project

The large amount of insecure legacy ICS and long ICS lifetimes mean we will need to live with this security risk for years / decades. We can argue about how long it should take to replace the deployed insecure-by-design ICS, but there is no disagreement that it is a huge problem. A big hole. Which is why it is so disappointing that we keep digging.

This was brought to mind again in a tweet from Joe Weiss's session at the SANS ICS Security Summit last week.

The key is that less sentence correctly pointing out that almost all systems deployed today add to the "legacy system" problem because they still have insecure-by-design PLC's / controllers and are using ICS protocols lacking authentication.

Back in 2013 in my S4 introduction (see video clip below), I bemoaned the fact we have been hearing it will take decades to address the legacy system security problem in ICS every year since I was first involved back in 2000. By 2013, we had made virtually no progress in dealing with insecure-by-design Level 1 devices or unauthenticated ICS protocols. We were still decades away from solving it, and the problem had gotten much larger with more "legacy systems" being installed over those 13 years.

The theme of S4x13 was NOW!, and the tag line was "If not us, who? If not now, when?"

https://youtu.be/bZLbm7J2E8o

It's now eight years after the NOW! themed S4x13 event, and we can look at what has occurred over those eight years optimistically or pessimistically.

The pessimist's side is easier. Over those eight years 99%+ of the ICS deployed have insecure-by-design PLC's/Level 1 devices and use unauthenticated ICS protocols. Access inside the perimeter = compromise only limited by the engineering and automation skills of the attacker, and the capabilities of the Level 0 connected devices. We have increased the 'legacy system' problem with eight years of ICS deployments. We are still digging that hole.

The optimist's side is some of the Level 1 device vendors and some of the ICS protocol groups have addressed the problem. There are now encrypted and authenticated versions of many ICS protocols. There are also now PLC's that have signed firmware, secure boot, support for secure ICS protocols, and authentication of operation and administrative functions.

Is it perfect? Of course it isn't. In some cases these PLC's carry with them a lot of legacy code. It's analogous to the Microsoft challenge after Bill Gates' Secure Computing memo. Yes, there can be a lot of improvement in the short run, and there is still a multi-year grind until that old legacy code is replaced.

Some of the key vendors have invested in development to have, at a minimum, a non-insecure-by-design offering. And more are near release. Now it is the asset owners' turn. The asset owners have to show they want to move away from insecure-by-design systems and reduce the associated ICS cyber risk.

Is circa 2021 when we stop digging the hole? Stop increasing the "legacy system" problem?

It is too soon to tell, but early signs show very little uptake to the available security capabilities. Asset owners aren't asking for them, integrators are designing them in, and vendors aren't pushing them. Features such as signed firmware do not require asset owner action and will be a clear win, but the secure protocols and user / device authentication do. They add complexity to the project and ongoing operation. The features often make the product more expensive as well.

The answer is likely, if security is deployed at all, to be sector and asset owner size specific. There may be a sector, such as large petrochemical, that may adopt this move away from insecure-by-design while other sectors continue with the status quo. Even this limited progress would be a big step forward as we have seen other ICS security practices trickle down from more security conscious sectors to the less security conscious sectors.

If the asset owners don't purchase and deploy the more secure versions, then there is little impetus for a for-profit vendor to spend resources developing, marketing and supporting security features.

The other option would be for the regulators to step in and require ICS being sold into certain sectors have certain security features. This would be difficult to do well, and it's a whole other article.

Bryan Owen of OSIsoft joins Dale to discuss all that when on in February. Top stories include:

- Attacks and Outages: Oldsmar, Solarwinds fallout, ERCOT/Texas and China's efforts on the Indian power sector

- Ruben Santamarta's research on IoT Software Development Kits

- What does GE backing out of the ICS security product market mean.

Plus wins, fails and predictions.

I recently stumbled upon a McKinsey article from October 2019 that more elegantly, in McKinsey speak, made the argument against "cyber hygiene" than I do.