William T. (Tim) Shaw
PhD, CISSP / CIEH / CPT

In spite of regulatory pressures and government mandates and reading about cyber attacks and data breaches in the news almost daily there are still a surprising number of industrial facilities that have not implemented an effective cyber security program to protect their critical digital assets. Just as worrisome are the plants where they think they have provided adequate protections; but in reality their digital assets are actually still vulnerable. IT consultants and product vendors are willing to provide all sorts of advice and cyber toys (for a price); and yet time passes and industrial facilities continue to be inadequately protected. Is there any logical reason why this is so?

I have worked with many industries ranging from pharmaceuticals and refining to power generation and steel production and over the years I have been involved with cyber security efforts at a good number of such industrial facilities. I will be the first to admit that establishing and maintaining adequate cyber security in an industrial setting is MUCH more complex and difficult than for a typical IT operation. To an outside observer it may not be obvious why this should be true. After all computers are computers and networks are networks so why not just have the corporate IT staff just show up one day, loaded-down with security appliances, and cyber-secure the plant? In a moment I will try to explain why that usually won’t work, but the frightening thing (to me at least) is that many corporate IT departments would/do actually agree with that proposition.

Now don’t misunderstand me, I am not excusing the owners of critical industrial facilities from their obligation to protect their critical digital assets from cyberattacks. Their lawyers and insurance underwriters should explain to them the risks associated with such a failure in governance. No, I am merely pointing out that establishing adequate cyber security is harder than it might seem due to a number of factors that are unique to industrial facilities, be they continuous or BATCH processes or even discrete manufacturing plants.

• Not every digital asset/system in a facility needs the same level of cyber protection, and many don’t need any cyber protections at all, the trick is in knowing which is which. Unlike an IT operation where you have servers, PCs and network elements (all of which are ‘computers’) an industrial facility has a range of digital assets including smart instruments and control elements, digital protective relays, PLCs, digital MCCs, digital analyzers, data acquisition systems, workstations and so forth all the way up to large and complex DCS and SCADA systems. I have seen plants where the I&C group identified several hundred (or more) digital assets but had no idea of how to go about classifying them by their importance to plant operations and safety and their need for cyber protections (or lack thereof). Faced with what seemed to be a manpower intensive (and hugely expensive) task of protecting everything they had identified, some groups just choose a wait-and-see (and hope nothing happens) posture.
• A one-size-fits-all approach to cyber security (such as many IT organizations use) merely leads to unnecessary complexity and frustration in an industrial facility. In the IT world there are not that many types of digital assets and many IT organizations standardize on a handful of software and hardware platforms, so it is possible to devise a cyber security approach that standardizes the technical controls and countermeasures to be applied to all digital assets. An example of this is the government’s FISMA cyber security standard for their IT systems which can be met by applying the NIST SP 800-53 standard and its huge list of cyber security measures.
  
  Also, most IT departments regularly update their assets every few years to keep them current with computer and networking technologies. In an industrial facility taking such an approach usually ends up creating a massive (and totally useless) paperwork nightmare where I&C personnel have to waste time documenting all the ‘exceptions’ to the standard policy – such as why they could not (and still have not) installed firewall software on the PLC that controls the steam generator and why they have not been ‘patching’ the digital trend recorder in the main control room. Also a lot of digital IACS and I&C equipment in industrial facilities is quite old and not even close to being up-to-date with current computer and networking technologies and so often incompatible with standard IT cyber security technologies.
• Many of the automation systems in operating facilities are required to run continuously 24/7 until such time as the plant reaches a scheduled shutdown or outage – which might not be for several years. Many security approaches, such as installing patches and updating (or eliminating) applications and installing specialized countermeasures such as a host-based intrusion detection system require at least rebooting of the systems to which they are applied, if not actually temporarily requiring removing those systems from operation. This is usually not a big deal in the IT world but, due to potential adverse safety and production impacts they may not be possible to implement in a plant environment until the next outage. Adding to the problem, during an outage the plant personnel are usually busy supporting the plant/process/ automation changes in order to get the facility back up and into operation again and have little time left to support cyber security activities. Some of the more complex automation systems may be fully redundant, and so it would seem obvious that you could do your cyber security work on the standby portions of the system and then switch over and work on the primary portions. Sounds good on paper, but in reality only a plant with a low-risk, stable process, would likely consider taking the risk. Making redundancy just work as advertised has always been a challenge to system vendors and few plants would want to mess with a properly functioning automation system.
   
• Many of the plant systems are categorized as being ‘legacy’ and have little if any vendor support (and spare parts have to be found on EBAY) and thus are considered as ‘no touch’ (following the ancient wisdom which recommends that “if it ain’t broke, don’t fix it!”). Such orphan systems may be running on hardware and operating system platforms that are semi-modern and might even use Ethernet networking. But no one at the plant has in depth technical knowledge of the system and IT would probably laugh (or put a lot of ‘lol’s and smiley emoticons in their email response) if asked to support the system (and would probably ask why it hasn’t just been replaced with something modern?) I feel sure that the plant manager can probably explain to them the reason that has not happened better than I ever could.
• IT strategies for convenience may actually degrade plant cyber security. Corporate IT people don’t generally like to go to industrial facilities. They (plants, not IT folks) are dirty and smelly and loud and things occasionally go boom. So a typical IT approach is to provide site support and administration of networks and cyber security countermeasures via remote access. Establishing a network/cyber DMZ (a term borrowed from the Korean War and the no-man’s-land established between the armed forces of the north and south) is one way of providing remote access to systems. In a DMZ strategy you place selected systems into a special, restricted LAN segment that is ‘visible’ from both the ‘inside’ (other plant systems and users) and ‘outside’ (corporate IT and hackers). You plan on the fact that systems in the DMZ are visible to bad guys and will be attacked and you strip them down to the basics (something called ‘hardening’) and put monitoring measures in the DMZ with them. The DMZ provides an early warning system of sorts and delays an attacker thus giving the plant the chance to respond by taking actions ranging from totally shutting off the external connection or at least making changes to the plant’s external firewall. But I have been at plants where everything except the drinking fountain and their DCS was placed into their DMZ, just so corporate IT could support them remotely. Most of these systems in the DMZ were not even given any special protections against cyberattack. The focus seemed to be on making life convenient for the IT folks. The plant was wide open for a cyberattack because of this, but they actually thought they were cyber secure. Their only true cyber protection came from the fact that the plant DCS was somewhat old and obsolete. Mind you, I understand why many corporations have had to resort to remote support, keeping IT folks at each plant is expensive and sending people to site takes time. But there are ways to make remote access secure; putting everything into a DMZ is not one of them.
   
• Many of the digital assets and systems have no test/support environment on which changes can be tested. As has already been mentioned a lot of digital assets in an industrial facility are legacy. The plant may have originally purchased enough spare parts to build a test and support environment for such a system, but by now, because the vendor either doesn’t exist or no longer supports the system, that support environment has been cannibalized for spare parts or for parts to expand the original system. This means that the only way to test a change, a patch, a new application is on the production system. Most plants would really, really resist the suggestion to put untested changes onto a running production system. This is less of a safety issue in a discrete manufacturing environment or even a BATCH production environment; but it is still not in any way a practice I would personally recommend.
   
• Many digital assets in an industrial facility are not ‘computers’ (even though they contain microprocessors and executable program code) and most (if not all) conventional cyber countermeasures can’t be applied to many of them. In fact many of these digital assets do not need to be protected against cyberattack. Many low-functionality digital assets are actually nearly immune to cyber compromise but, not knowing this, efforts are often made to protect them anyway. Part of the problem here are the vendors/manufacturers. They publish spec sheets and manuals that give a lot of information about their products, but almost never provide information about the cyber vulnerabilities (or security) of their products – possibly because they don’t know? I sat on the phone with a vendor’s engineer one day asking question after question in order to finally determine that indeed, their device would not be vulnerable to an infected USB ‘thumb drive’ being inserted due to limits in their USB port support software. It would have been nice to find that data in one of their manuals. Vendor documentation rarely tells you if their ‘smart’ device uses ROM, EEPROM or flash memory to hold their program code; yet knowing this can tell you if a device is essentially immune to any form of cyberattack (and therefore requires no protections). IT folks generally don’t have to think about such issues since everything they deal with tends to be a computer with a bootstrap BIOS program, lots of RAM, possibly a file system and hard drive, Ethernet ports and a COTS operating system – and that pretty much guarantees that the device is vulnerable to some form of cyberattack, and will need to be protected. I have seen far too many plant personnel wasting time and money trying to figure out how to put cyber security protections around a digital asset that actually didn’t need any and which wasn’t getting any benefit from the effort.
• IACS often use industrial protocols and vendor-proprietary protocols that are unknown to IT products. Most of the early local area networking protocols devised prior to the widespread acceptance of Ethernet (and eventually TCP/ IP) have been ported as ‘ISO-Layer 7’ (application layer) protocols that can be transported using Ethernet-TCP/ IP networking. Many DCS systems have been converted to use Ethernet, and possibly TCP/IP, in place of their older, vendor-proprietary networking technologies. But the protocols and message types used by IACS and I&C devices are not always supported by commonly-used IT protective and detective mechanisms such as firewalls and NIDS (network intrusion detection/protection systems). It’s not that such mechanisms don’t ‘see’ such message traffic. After all it is carried in Ethernet frames and possibly IP datagrams. The problem is that those tools either can’t interpret, or have a limited capability to interpret, and decode such message traffic and thus determine which of those messages are allowed and which might be malicious/ unauthorized. IT personnel usually know nothing about those ‘industrial protocols,’ and it might not be their fault; if it’s a vendor’s proprietary system messages they might not actually be documented anywhere.
   
• Older plant networks may still incorporate legacy communications and LAN technologies that pre-date Ethernet and TCP/IP. First and even second generation DCS systems mostly used vendor-designed LAN hardware and software. Early PLC based systems used vendor-designed LAN hardware and software. A lot of that stuff is still out there running in plants. Also communications interconnectivity between digital systems and devices originally was established using low-speed, minimal-functionality, point-to-point (or multipoint) ‘serial’ communication circuits with a range of legacy industrial protocols. This was particularly true with SCADA systems. There are few if any products available to monitor or secure such communications. On the other hand it may be overkill to attempt to do so based on the potential (limited) consequences of such communications being compromised.
   
• Some industrial facilities are highly unlikely to be the focus of a targeted cyberattack but still move forward on implementing excessive cyber security protections out of fear and ignorance. The government, in the form of the department of homeland security, has tried to identify industry segments that are important to the nation and economy and thus are much more likely to be a target for cyber terrorists. Facilities that make glass, dog food and shoes (ok, we don’t have shoe plants… so maybe toothpaste) are not going to be grabbing headlines if they are shut down, and probably no one would be killed in the process and the economy would be largely unaffected. This does not mean that an ad hoc cyberattack could not occur. Someone can always unknowingly bring an infected laptop into work and connected it to a plant network. But protecting against that kind of attack does not require the levels of protection and detection as would say, a nuclear power plant or a large refinery, where an adversary would likely target them and be willing to expend significant resources in an effort to create a headline-catching catastrophe. One of the challenges with industrial facilities is picking the appropriate level of cyber security based on cyberattack risk and likelihood. Most IT organizations assume that all of their assets are important (after all the business couldn’t keep running without them) and therefore need to be given equal protection. On the other hand individual industrial plants, and specific areas/ units/trains within those plants, need to be assessed to determine how much (if any) protection is adequate and how much risk is tolerable and cost-justified.
   
• Some industrial facilities believe that their physical security measures (fences, gates, video surveillance and guards) provide adequate cyber protection for critical digital assets, especially since they probably also believe those assets to be ‘isolated’ and thus impossible to attack, and therefore don’t think they need anything further. The definition of the term ‘isolated’ would seem to be clear; you can look it up in a dictionary (anyone remember those?). I thought I understood the term, and I have had plant personnel swear up and down that a critical system was ‘isolated.’ Except of course when Engineering Lead Fred X needs to dial into it from home (or connect to it across the Internet) during an emergency. Oh, and of course when the system vendor needs remote access to diagnose a problem and download a ‘fix’. And of course any of the technicians can walk up to the system and connect a laptop or insert a CD at any time, because all of them are totally trustworthy. Possibly we need a new term such as ‘usually isolated’ or ‘somewhat isolated’ or even ‘hardly isolated’ in order to more accurately describe the actual status of such systems? Maybe the plant management team is unaware of the Stuxnet attack on the really well isolated Iranian fuel enrichment facility? Maybe they are unaware that hackers can locate and attack remote access connections, both dial-in and cross-network? Maybe they are unaware that insider sabotage is one of the leading (and growing) causes of unplanned plant outages and plant system failures – and we are not talking about radicalized followers of ISIS. Most malicious insiders are just Bob from the instrument shop who got a crappy review, was just informed that he has to pay more for his medical insurance next year and who got no cost of living raise this year, and he feels like a little payback is in order. (My apologies to any actual ‘Bob’ who works in the instrument shop.)

So, establishing and maintaining a cyber security program at an industrial facility is challenging. Not impossible, but definitely challenging. There are a number of factors that need to be considered that are outside the realm of expertise and experience of most IT organizations (as well as most plant I&C organization). But it is not impossible and it is important and many industrial plants HAVE implemented cyber security programs. There have been cyberattacks on industrial facilities, both domestically and internationally, and there is no reason to believe this will stop. Fortunately none have resulted in major catastrophes (just a few minor ones). There are actually strategies and approaches for every one of the challenges that I listed above, many that don’t cost a fortune and some that are quite innovative…..but that will have to be left to a future column.
 

About the Author

Dr. Shaw is a Certified Information Systems Security Professional (CISSP), a Certified Ethical Hacker (C|EH) a Certified Penetration Tester (CPT) and has been active in designing and installing industrial automation for more than 35 years. He is the author of Computer Control of BATCH Processes and CYBERSECURITY for SCADA Systems and co-author of the latest revision of Industrial Data Communications. Shaw is a prolific writer of papers and articles on a wide range of technical topics and has also contributed to several other books. Shaw has also developed, and is also an instructor for, a number of ISA courses and he also teaches on-line courses for the University of Kansas continuing education program. He is currently Principal & Senior Consultant for Cyber SECurity Consulting, a consultancy practice focused on industrial automation cyber security and technologies. Inquiries, comments or questions regarding the contents of this column and/or other security-related topics can be emailed to timshaw4@verizon.net.