Recently I overheard a couple of supposedly-knowledgeable cyber security experts discussing the amazing (to them) range of microprocessor-based devices they were running into in a power generating station where they were working with the electric utility to achieve compliance with the ever-evolving NERC CIP requirements. One of them was lamenting that far too many of the NIST 800-53 recommended cyber security controls had not been applied, and that the plant automation staff kept claiming that it was not possible to implement them. The other was responding that it was probably okay in some cases since the devices had their program code in ROM so malware could not infect them.

Look at All the Funny Little Boxes!
Well, both of those two experts were somewhat correct but also very, very wrong. I have used this column in the past to reflect on the general lack of knowledge among cyber security practitioners in the area of industrial automation. Specifically, I see an amazing lack of knowledge about technologies such as SCADA, DCS and PLC-based systems, in addition to commonly used industrial protocols such as EtherNET/IP, Modbus/ TCP and ProfiNET. Even more glaring, based on the overheard conversation with which I opened this column, is the lack of knowledge about commonly used “smart” instrumentation and devices such as protective relays, PID controllers, digital trend recorders, smart annunciator panels and the like. I realize that IT personnel don’t run into that kind of stuff all that often, if ever, in their careers. But if you don’t know about that stuff, and claim to be able to properly assess and establish adequate cyber security protections in an industrial facility, you are kidding yourself (and misleading your clients.)

I have personally witnessed heated arguments about smart transmitters (HART, in this case, operating in “analog” mode) serving as possible platforms for attacking a control system. The assumption being that the transmitter could somehow become infected with malware (an amazing and mission-impossible-worthy feat in and of itself) and then attack and compromise the associated control system via the wired connection between the two. Now I don’t expect an IT person to particularly know what an analog-to-digital converter circuit is and exactly how it works, but I do expect people who purport to be “experts” to do the necessary research and investigation when they run into something they don’t understand, rather than to jump to far fetched conclusions with no basis in scientific or engineering reality. One thing that I take away from this sort of experience is that fact that IT education is badly lacking in providing the practitioners with a wide ranging understanding of the various ways in which computer-based devices can be constructed. Not everything that contains a microprocessor is a Microsoft Windows PC or server. It would be useful for IT folks to be exposed to the many ways that microprocessors are embedded into “smart” devices today, from the washing machine that can “phone home” to request a service call to a doorbell that calls your cell phone and initiates a Skype-like video chat session. But it is not enough that they know such things are being built; they need to be told the cyber security implications of such designs.

Going back to that conversation on which I was eavesdropping, one of the parties asserted that it was okay not to have added cyber protections for some of the smart devices since their code was in ROM and thus the devices could not be infected with malware. To begin with, that assertion/assumption could be wrong. Fewer and fewer devices today use ROM memory for much since it is impossible to make updates or patches to code “burned” into ROM without taking the device apart. Today it is more common to have a bootstrap loader in ROM that takes a compressed program image from flash memory (possibly/often stored on removable media) and decompresses/writes it into RAM for execution. In such a design you can overwrite the contents of flash, in the field, to provide a software (firmware) update. In such a design the code is actually running in RAM and thus it can potentially be overwritten with malware. Mind you that task is way more difficult than might be apparent because:

1. You need a detailed understanding of the physical memory locations of the program code you want to overwrite
2. The code for the device is very hardware platform/ CPU specific
3. There needs to be extra RAM available in which to insert the bulk of the malware
4. The source code for the device is vendor proprietary and not generally available for analysis unlike Windows and Linux
5. The device is most likely not running a C.O.T.S. multitasking operating system (and probably does not have a file system), and
6. Any change you make must not interrupt the normal/ expected functions of the device unless all you want from injecting malware is to disable the device or modify its functionality in a manner that will quickly become obvious.

Of course, another factor that needs to be considered is possible access to the device’s configuration settings. With a lot of smart devices, the functionality of the device can be drastically modified through the changing of configuration settings. (I differentiate configuration settings from operational settings: the former are usually set during installation and commissioning and rarely if ever need to be altered – think PID tuning constants. The latter may be altered quite often as part of normal operations – think PID setpoint changes.) I don’t need to inject malware into a smart device to degrade or disrupt its functionality if I can achieve the same thing by modifying some setting values. A good example of this would be a digital protective relay. Today most such devices come with a broad number of built-in relaying functions and you turn the device into an under-frequency relay or an over-current relay by selecting the protective function(s) you want it to perform using configuration settings. Many modern smart devices support communication connectivity, including Ethernet-TCP/IP network connectivity. In such an arrangement the device may “expose” (make available for reading and possibly writing/changing) its configuration settings through an industrial protocol such as EtherNET/IP or Modbus/TCP and it might also “expose” them via an integral web server running on the device and through SNMP (standard network management protocol) access. However, these setting are made available for modification. If the mechanisms, or means for accessing them, are not adequately secure than I can maliciously impact the device by making such changes. So clearly the second expert in the overheard discussion was not taking that possibility into account and would not have looked into protecting those configuration access mechanisms and pathways.

I mentioned earlier that another problem with many cyber security experts is that they are woefully lacking in their understanding of the communication protocols used in industrial automation applications. If they have any familiarity at all it is usually only with the “modern” variations of older, vendor-proprietary protocols, the ones that have been ported onto an Ethernet-TCP/IP platform as an application layer (ISO/ OSI layer 7) protocol. These are the kinds of protocols that you have some chance of inspecting and monitoring using modern firewalls and network intrusion detection (NIDS) technologies. But of course, to do a good job of this you do need to understand the details of the protocol at the application level, not just the TCP/UDP and IP levels. And if you have one of the legacy variations of these protocols, running on a proprietary LAN such as the Allen-Bradley Datahighway-plus®, then all bets are off since no off-the-shelf IT solutions exist for monitoring and controlling message traffic on such a LAN.

Of course, the argument can be strongly made that a cyber-attacker probably can’t do much in that case either, unless they get physical access to the LAN, or (more likely) to one of the attached computers. (And if an attacker is that far inside your plant a cyber attack is probably the least of your worries.) For those old legacy industrial LANs, and for “serial” (asynchronous, low-speed) point-to-point or point-to-multipoint communication links, the only available strategy is to physically protect the communications media and system components from unauthorized physical access. In some limited cases (e.g. a point-to-point “serial” link), you might also consider link encryption if you feel that you can’t physically secure the communication circuit.

The challenge, when discussing such legacy communications technology with many IT-knowledgeable cyber security experts, is that they think the same issues apply to these technologies as do to an Ethernet-TCP/IP-based communication scheme. I have had endless arguments with such experts concerning buffer overflow attacks and why they won’t happen with something like “serial” Modbus (Modbus-RTU) protocol. In an IP network, with a multitasking operating system, a program that needs to receive data from another program allocates memory space for that data and then asks the network services part of the operating system to receive a message and put it in the allocated memory area (and is then put to “sleep” by the operating system.)

IP presumes that the sender is going to send a valid message that won’t exceed the allocated space and essentially trusts the sender to do the right thing. Of course, hackers find such situations and use them to send too much data, which overwrites program memory (thus changing the program instructions to do something the hacker wants.) When the operating system then lets the receiving program resume executing the damage has been done. With the “serial” Modbus (Modbus-RTU) protocol the node receiving a Modbus message receives each message byte individually and verifies that the byte value is proper/expected. If they byte does not make sense (e.g. it is supposed to be the Modbus “command” and its value represents a command that the receiving node does not support), the node flushes what has been received to this point and starts waiting to see the start of a new message. A buffer overflow is just not possible.

Likewise, it is not possible to use Modbus-RTU protocol to create a file, write a program into that file, and have the program executed. First because (standard) Modbus doesn’t even envision the concept of files and programs, second because no such commands exist in the Modbus command set and finally, because the “coils” and “registers” Modbus can read and write are not usually mapped to executable program memory. IT experts usually presume that any “smart” device will have a command prompt and command line interpreter of some type that enables such activities. It isn’t always easy to convince those experts that a “serial” Modbus communications link does not provide such access. This is not to say that a serial Modbus communication link might not offer a target for cyber compromise. If a field device speaks Modbus protocol and controls plant equipment, then injecting forged messages addressed to the field device could cause such equipment to fail. To do this, the attacker still needs to get physical access to the communication circuit or to the “Master” node. So now we are back to physically protecting the communication circuit and system elements or employing link encryption to prevent such an event.

For the Ethernet-TCP/IP based industrial protocols, you DO need to think about cyber security issues because it is possible to both “sniff” and “spoof” such message traffic and because most such protocols do not have intrinsic cyber security mechanisms (e.g. end-point authentication, encryption, etc.) Having said that you also need to be realistic and practical; if your network consists of a few controllers and operator HMIs on an isolated LAN, all located in a locked (or otherwise access controlled) room, then you have a highly constrained cyber security issue where physical access and use of occasionally-connected portable digital devices and use of computer-readable storage media pose the main threat. But if your plant network is extensive and contains many devices and computers you really ought to be taking steps to ensure their cyber security. This is where lack of knowledge of industrial protocols can be an issue.

A good way to protect a critical digital system or device from cyber attack is to place it behind a well-implemented firewall. A firewall allows message traffic you have designated to pass through and blocks all other message traffic. You make this happen by writing “rules” (an ACL – access control list) that describe the criteria of permitted messages. You can say they must come from a specific set of IP addresses, that they must be going to port 20000, that they must be using TCP transport. But all of that is easy enough to fake/spoof using readily available hacker (and penetration testing) tools. But, if you are knowledgeable about the specific industrial protocols you are using, you can extend your rule criteria to the level of the protocol itself. In Modbus/TCP terms that could mean specifying that only specific commands be permitted and only if they reference a specified set of register addresses. In EtherNET/IP terms that could mean that only specific object types and instances are permitted and only if specific methods/commands are being issued to those objects. By having rules down to that level of detail it makes it much more difficult for an attacker to spoof/ fake your message traffic. It also increases the likelihood of detecting efforts to forge messages. But, if a cyber security expert doesn’t know about the details of your industrial protocols it is unlikely that they could be of assistance in enhancing your firewalls.

In most instances, once a vendor goes to the point of adding Ethernet and TCP/IP communications to their device, they don’t limit their efforts to merely implementing one (or more) of the application-layer industrial protocols. They usually add additional network-enabled features to enhance their product offering. It is common to see vendors implement several basic IP “services” such as embedding some form of web server that provides a GUI (graphical user interface), preferably using HTTPS (encrypted/ authenticated) access. They also often implement a command-line interpreter such as TELNET (insecure) or SSH (secure shell – secure), and they may support other functions such as (T)FTP upload/download file access to firmware and/or device data and even SNMP access to critical settings, as I mentioned earlier in this article. The bad news is that these services open the device up to a range of possible cyber attacks. The good news is that most IT cyber security experts really DO know how to protect these kinds of services with appropriate firewall rules.

A good general-purpose rule of thumb in cyber security is that convenience and cyber security are opposing forces. Increasing one generally decreases the other. All of those added “services” make it much more convenient to remotely (across a network) administer and support these devices, but it also makes them far more vulnerable to cyber attack and compromise. In an industrial facility most devices and systems are installed, commissioned and acceptance tested and then rarely, if ever, messed with unless something breaks. Even installing security patches and updates is often frowned upon since history has proven time and time again that you have a greater-than-zero chance that something will go wrong when you are making such changes. So, in a plant environment, because of the infrequency of changes, the need for device/ system administration services is also infrequent and thus not a major burden, even if said administration has to be performed locally to the device/system rather than remotely. In this case, it is possible to consider disabling the services running on a device/system which are needed for remote administration (e.g. Telnet), but which also provide a potential attack pathway for an adversary. This reduces the device’s or systems ”attack surface,” thus improving their cyber security. But that sort of thinking runs contrary to conventional IT wisdom and so your IT cyber security experts probably won’t suggest that approach.

I have written in the past about successful collaborations between OT and IT groups where the domain knowledge of both is merged and results in a far better cyber security end-result. And there is training available on cyber security for Industrial Automation and IT cyber security experts would benefit from getting such training if they are going to be providing cyber security consulting services to industrial clients. A CISSP, or one of the many variations, does not equip a cyber security expert to work in the industrial world. I have often thought that there needs to be a specific certification for industrial automation cyber security. But that will have to be the subject matter for a future column.
 

About the Author

(William) Tim Shaw (Ph.D., CISSP, C|EH, CPT) has been active in industrial automation for more than 40 years and is the author of Computer Control of BATCH Processes and CYBERSECURITY for SCADA Systems and co-author of Industrial Data Communications. Shaw has contributed to several other books and is a prolific writer of papers and articles on a range of technical topics. He has been directly involved in the development of several DCS and SCADA system products and regularly teaches courses for the ISA and the University of Kansas on a range of topics from cyber security to process automation and basic process instrumentation and measurement. Inquiries or comments about this column may be directed to Tim at timshaw4@verizon.net.