Watching the NERC Critical Infrastructure Protection (CIP) standards evolve over the past few years, and seeing all of the ongoing discusssions in the various on-line blogs about what may or may not meet with NERC compliance, leaves me with the belief that there are still many issues that have not been properly or adequately addressed. When the ‘rubber meets the road’ during the implementation of a cyber security program, that is when you run into the issues that don’t seem to fit into the strictures of standards developed by well-intended people who don’t do this for a living. We also continue to see the collisions between what corporate IT thinks is a best practice and what NERC thinks is a better practice. Let’s look at some examples.
William T. (Tim) Shaw
PhD, CISSP / CIEH / CPT
One of the common practices in big corporate IT is to outsource manpower-intensive activities in order to minimize ‘head count’ (a cute accounting term for staffing level) and possibly to move liability and risk to an outside organization (a strategy called ‘blame-shifting’). A common example is the outsourcing of the IT help desk (irritating at times, but not necessarily a CIP compliance issue).
A more pertinent and growing example is outsourcing the 24/7 monitoring of the corporate network and the security mechanisms such as the firewalls, critical servers, host intrusion detection systems (HIDS) and network intrusion detection systems (NIDS). Why not let low-cost, but well educated and technically proficient, people in Bangalore watch over your critical infrastructure? You can set up a virtual private network (VPN) connection over the Internet and route all the alarms, alerts, and logs to their attention and inspection and at half the cost of staffing this function yourself. Makes good business sense, but does it make good CIP sense?
Recall that the basic purpose for producing the CIPs was to get electric utilities/entities to provide adequate cyber (and physical) security for their critical cyber assets (CCAs) in order to protect them from attack. A basic tenet of that requirement is the establishment of an electronic security perimeter (ESP). Although there have been lots of arguments over what creating an ESP really means and what constitutes a penetration of that perimeter, I would hope that everyone would agree that the intent is to prevent unauthorized, uncontrolled, and unmonitored (and potentially malicious) communications between CCAs and ‘external’ systems. A major aspect of creating an ESP is the placement, configuration, monitoring, and maintenance of boundary protective devices, i.e. firewalls, at all ESP penetration/ access points. Let’s ignore asynchronous, serial, low-speed [non-IP, non-Ethernet] communications for the moment as they present a different challenge and may not count. Let’s also ignore wireless communications (cellular, IEEE 802.11 and Bluetooth) for the moment as they provide a whole different range of special challenges.
CIP-002-5 defines bulk electric system (BES) cyber system ‘Associated Cyber Assets’ and includes Electronic Access Control or Monitoring Systems (EACMS) in that category. IDS/IPS and ESP boundary firewalls would seem to fall under that definition and therefore they would need to be protected against compromise. The CIPs address requirements for verifying the trustworthiness of your personnel – especially those who are involved in the use, support, and maintenance of the CCAs. The CIPs also address the basic physical security requirements to protect as well as to limit and control physical access to the CCAs. The problem with out-sourcing the monitoring, and possibly the administration, of your network architecture and protective/detective assets, especially to a foreign organization (particularly in a country where a lot of people don’t like the U.S.), is that you can’t guarantee that they meet the same level of cyber and physical security as you do. Problems can arise even if you ‘include language in contracts that requires adherence to the Responsible Entity’s Interactive Remote Access Controls’, as stated in CIP-003-5 requirement R1, paragraph 1.2.
Have you ever signed a contract with a vendor and then discovered that they failed to actually live up to the all of the elements of that contract and associated P.O., at least as you interpreted them? I know that I have, and they were mostly US firms. You might provide yourself with some legal/regulatory blame-shifting by including appropriate contract language, but in reality you really can’t be certain that you are not opening up a very dangerous attack pathway directly into your ESP and your CCAs.
If you do decide to do this anyway you must open up a bidirectional access point in your ESP in order for it to work. Even if that connection is VPN protected, and there is no known attack methodology for compromising that connection today, that doesn’t mean that one won’t be discovered in the future. Lots of cyber security mechanisms believed to be unbreakable in the past have fallen to technology advances and new technologies – just ask the IEEE folks who came up with wired equivalent privacy (WEP) or the Government folks who gave us DES.
Another common corporate IT practice is the centralization of user authentication and security policy management, often through the deployment of technologies such as Microsoft’s Active Directory© (AD) servers. These are typically not located within your ESP. This strategy allows corporate IT to ‘push out’ policy changes, including user account changes/removal to all of the computers and servers across the corporate WAN and to centrally manage user ID/account verification. That is very efficient from an IT manpower point of view, but if this approach is integrated into EMS/SCADA systems or plant DCS systems then you potentially create a cross-WAN, ESP boundary crossing communication requirement essential to the function of those automation systems, which are ‘inside’ your ESP. Those AD servers may or may not have adequate cyber and physical security and thus may not meet the requirements of being inside a separate, but linked ESP. Corporate IT usually dislikes allowing such things to be out of their control, but it is possible to have a duplicate AD server positioned within your ESP and then all you have to deal with is the periodic updates sent to this server from the master AD server. That kind of message traffic is actually pretty easy to run through an ESP access point firewall and then monitor with a NIDS. But you have to talk corporate IT into letting you set things up that way, which goes against their world view of best practices.
To take things even further, corporate IT has embraced the use of virtualization and thin-client architectures so they have control over the applications you are allowed to use and so they can rapidly provision additional virtual desktop PCs and application servers. It always seems as if they never plan for enough server and network bandwidth (probably as a cost saving measure) and so the responsiveness of your personal thin client can vary dramatically through the day/week. Of course that is because the workload on an IT server will peak and drop as people first arrive at the start of the day, take coffee and lunch breaks and then head home at end of the day. IT often plans their server processing power and network bandwidth to support the median or average load, thus things are slow first thing in the morning and speed up over lunch.
Having been forced to endure working in this kind of ‘IT paradise’ for several years, I have often been reminded of why using centralized computer control was dumped when distributed control architectures were eventually developed. I have lately seen electric utilities where the SCADA/EMS group was placed under the management of the IT organization. As a result the next SCADA system upgrade/replacement was also a conversion to a thin-client architecture with the servers being controlled by IT and the control room operators of the transmission system being given web browser access to their system – essentially treating the SCADA/EMS system as just another corporate application. The servers were physically owned, managed, and controlled by IT. Again, this makes it very complicated to establish an ESP – let alone a physical security perimeter (PSP) – particularly when a good portion of the CCA(s) that need to be protected are ‘virtual’.
Oftentimes, a gaping hole is created in the in the cyber security/electronic perimeter of many organizations due to the incautious/insecure use of portable electronic devices, such as laptops and tablets and the use of removable/portable computer media like the ubiquitous USB ‘thumb drive’ or DVD/CD platter. I can’t remember the last time I was in a SCADA system or DCS system control or computer room and didn’t see a laptop PC, USB thumb drive, cell phone or CD/DVD platter (not to mention less-often seen things like MP3 players and digital cameras). Because critical systems are often isolated (air gapped) or given very strictly controlled and limited communications connectivity to networks and other systems (at least one would hope so) the only reasonable way to transport patches, software updates, configuration files, collected data and the like is via portable media or portable devices that support file systems and storage. The problem is that these same media and devices can also be used to deliver malware; as the Iranians found out when Stuxnet was introduced into their control systems.
So, on one hand it is almost essential to be able to use portable devices and media with your critical automation systems and digital assets, but on the other hand this can lead to self-inflicted cyber attacks if you don’t do it properly. Is there a safe way to use these things?
First it’s important to recognize that there is a major difference between passive media, such as a tape, a floppy disk, a CD/DVD platter or a dumb USB storage device and an electronic device that has a processor, a file system and field-alterable stored programming that might include a laptop PC. Passive media has no ability to actively hide its contents from virus scanning tools. If known malware is loaded on passive media it is probably going to be identified via a comprehensive malware scan. I say ‘probably’ because some evaluation testing performed on several of the more popular virus scanning utilities showed that the detection of more recent malware is not 100 percent reliable. The reliability gets much better when several scanning packages are used and the packages are kept current with vendor updates. The secret is to run the scanning tools on a bare-bones, well-hardened, and properly patched stand-alone system, just in case there is dangerous malware on the media that attempts to compromise the scanning system itself!
On the other hand a severely compromised device, such as a laptop PC that has been infected by a modern rootkit, has the ability to hide malware from virus scanning tools, even making entire directories invisible, so that the likelihood of successful malware detection is low to non-existent.
In-between those two extremes are less-capable electronic devices with differing levels of vulnerability to malware infection and being able to be used as a delivery mechanism. Modern cell phones are probably the most dangerous devices in this group, particularly because of Bluetooth wireless networking support. But any device with USB connectivity and a mountable file system has the potential for being used for malware delivery. You normally have very few options for performing malware scanning on such devices. Worse, if connected to a Microsoft Windows computer that has universal plug-n-play services enabled (the default setting in older Windows distributions) such a device can install a rootkit in the form of a device driver.
You have a reasonable chance of avoiding infection if you establish and maintain good procedures for media scanning and use only passive media. You can even allow portable devices to be connected to your CCAs if you establish strict procedures to ensure that such devices don’t leave your control and are not connected to insecure networks, like the Internet, or to systems outside of your control. There are some best practices for this and this is one case where corporate IT may actually have the ‘right stuff’ to keep you protected and your ESP secure.
Of course it is possible to find strategies that can both satisfy the corporate IT folks and meet the NERC CIP specifications, but that requires that IT be willing to compromise and accept that their best practices are not always industrial automation best practices. It may also require taking an approach that isn’t the lowest cost way to do things. Getting management buy-in may be the biggest stumbling block of all, especially if they see the CIPs as just a bunch of stupid regulations that produce wasted time, effort, and funding to protect against a threat that isn’t real. There are ways to explain the cost-benefit justification of reasonable cyber security, but that will have to be the subject matter for a future column.
About the Author
Dr. Shaw is a Certified Information Systems Security Professional (CISSP), a Certified Ethical Hacker (C|EH) a Certified Penetration Tester (CPT) and has been active in designing and installing industrial automation for more than 35 years. He is the author of Computer Control of BATCH Processes and CYBERSECURITY for SCADA Systems. Shaw is a prolific writer of papers and articles on a wide range of technical topics and has also contributed to several other books. He has also developed, and is an instructor for, a number of ISA courses. Dr. Shaw is currently Principal & Senior Consultant for Cyber SECurity Consulting, a consultancy practice focused on industrial automation security and technologies. Inquiries, comments or questions regarding the contents of this column and/or other security-related topics can be emailed to timshaw4@verizon.net
