Not too long ago I had the opportunity to engage with a number of utilities struggling to attain full compliance with the NERC CIP standards. Part of that involvement included visiting their generating facilities and looking over the way they were protecting their critical plant systems. It also included reviewing the security for their EMS/SCADA systems. The effort invariably involved dealing with plant and corporate IT personnel because the evolution of automation systems – particularly over the past decade – has blurred the distinction between those systems and what we think of as ‘business’ systems. This could be a good thing or a bad thing, depending on how far it goes. This column will discuss the impact of these ‘hybrid-ized’ automation systems on security and the fact that when it comes to networks, sometimes you really don’t know who you’re talking to… – Tim
William T. (Tim) Shaw
PhD, CISSP
From a historical perspective, the development of computer-based automation is somewhat separate from that of the development of computer-based technology for general business applications. These two markets may have indeed shared some hardware platforms, but the computer suppliers involved did not offer the specialized real-time networking, I/O hardware, software or HMI technologies needed to create control systems at the time. Thus, DCS and SCADA vendors either developed their own tools internally or cobbled together bits and pieces of whatever they could find to create solutions.
For that reason, a person learning “IT” – Information Technology, that is – would not have been familiar with much of the technology in a computer-based automation system. But considering the eventual domination of Intel and Microsoft and the widespread adoption of Ethernet and TCP/IP networking in the control world, today’s DCS and SCADA systems look a
lot like business systems. And, because of that transition, today even a traditional IT person would be quite familiar with much of the underlying technology.
That is both a blessing and a curse. A very significant reason why our current automation systems are vulnerable to cyber attack is because they incorporate such a large amount of conventional IT/business technology. Moreover, IT people in more and more organizations are being called upon to support automation systems as a cost savings measure. This is perhaps more prevalent with SCADA systems than with DCS systems so far, but contemporary IT philosophies and strategies are making their way into both environments.
IT organizations are used to creating fully interconnected and integrated corporate networks, where it is possible to access any system or computer from anywhere on that network, and where centralized servers and support are the most efficient way of performing their mission and provide rapid response to problems. As one would expect, they (IT personnel) would tend to promote those same strategies when asked to support other, seemingly similar, computer-based systems. Both DCS and SCADA systems of any recent vintage – certainly those installed and commissioned within the last decade – will have undergone some amount of “hybridization” with traditional IT systems.
During the past year I’ve had the opportunity to closely study a few systems where corporate IT had exerted significant influence over their evolution, which yielded some rather interesting observations. For example, an application engineer or process engineer sitting down at a system workstation for either type of automation system (i.e., SCADA or DCS), used to “log in” and have his/her usage rights validated (or rejected) directly by that system. But in a highly hybridized system, those same engineers may have their user login passed over a corporate network to – and validated by – a centralized corporate authentication server, such as a Microsoft Active Directory server that is managed by the IT department.
This is very convenient because it allows a single repository for user access rights information and credentials across all corporate systems. If the automation systems incorporate MS-Windows workstations or MS-Windows- or Linux-based servers, it is quite possible that those computers are being patched and having their virus scanning software updated from another corporate server that is being managed by the IT department. These automation systems may also receive time synchronization from a corporate network time-server, and they may depend on yet another corporate Domain Name Server (DNS) to be able to locate other systems within the corporate network with which communication is required.
These various corporate servers may be physically located in the same plant facility, corporate facility or, they could physically be anywhere else on the corporate network, since many organizations have linked their corporate networks and plant/system networks together. The result is very convenient, very efficient and very much in line with accepted IT strategies. But from a cyber security perspective – and especially a NERC CIP perspective – this is also very problematic.
NERC-CIP requirements call for special protections, both physical and electronic, to be applied to critical systems, which are included under NERC’s “critical cyber assets” terminology. But when a critical system has some functional dependence on other systems (i.e., systems that may or may not be afforded the same level of security), the critical system has been inherently – and perhaps seriously – compromised with additional, exploitable vulnerabilities. That is, if communications between automation systems and such servers traverse corporate networks without suitable communication protections, they are implicitly vulnerable to any number of network-based cyber attacks. Not only can message traffic can be intercepted, falsified or blocked, but if these servers are attacked and disabled, the functions they provide will be unavailable to the critical systems unless some level of redundancy or backup is provided.
NERC CIP standards take a somewhat conventional view of SCADA/EMS systems and also plant automation systems. The conceptual model is of stand-alone, (mostly) autonomous systems, with a small number of data exchange interconnections that might be – but probably aren’t – critical to the essential functioning of the system. But, through hybridization with corporate IT systems, a modern SCADA/EMS or plant DCS system might rely on one or more distributed servers scattered around an insecure corporate communications infrastructure. This makes it very difficult to establish and protect an electronic security perimeter (ESP).
The NERC CIP concept of protecting access points through the ESP is based on the assumption of an attacker trying to gain access to the critical system or infect it with malware through the access points. But with highly hybridized systems, an attacker need not attempt to reach the critical system at all. When corporate IT systems become essential to a critical system, one would simply attack the corporate system(s), which frequently would not have the same level of security protection as the critical system. Of course, a utility could solve this problem by declaring all of the various corporate servers as being within and ESP and PSP (physical security perimeter) and giving them all of the security controls outlined in the CIP standards. Notably, this would mean appropriately monitoring and defending each of the access points – every one of those connections to the entire corporate network, which could be exceedingly complex to say nothing of the time and cost issues.
The obvious solution to these problems is actually quite simple: Move those distributed functions back into the critical systems themselves. In other words, return them to the stand-alone, autonomous model envisioned by NERC. Most DCS and SCADA systems already have the ability to support those functions, even if it means adding another computing platform within the ESP. This makes supporting those systems less convenient, but eliminates a very severe and potentially dangerous (yet largely invisible) set of vulnerabilities.
Don’t misunderstand my point here; IT people have a lot to teach SCADA and DCS system engineering and support personnel about cyber security and network protection. And unfortunately, we can’t simply turn back the clock to the days before plant-to-boardroom network connectivity. But there remains a basic difference between ‘best-practice’ policies and procedures for IT and those for industrial automation. In general, it really boils down to addressing safety issues and the very different consequences between having an IT server compromised and having a SCADA/EMS or plant DCS system compromised. Clearly, there needs to be an ongoing dialog and knowledge exchange by and between these departments and disciplines, but that will have to be the subject of a future column… – Tim
About the Author
William T. “Tim” Shaw (PhD, CISSP) has been active in industrial automation for more than 30 years and is the author of Computer Control of BATCH Processes and CYBERSECURITY for SCADA Systems. Tim has contributed to several other books and is a prolific writer and presenter on a range of technical topics. He is currently a senior security consultant for SecuriCon, an information security solutions firm, based in Alexandria, Virginia. Tim has been directly involved in the development of several DCS and SCADA system products and regularly teaches courses for ISA (International Society of Automation) on various topics. Inquiries or comments about this column may be directed to Tim at Tim@electricenergyonline.com.
