William T. (Tim) Shaw
PhD, CISSP

We all buy stuff and expect the stuff we buy to be safe, reliable and meet our expectations of quality and performance. No one is ever happy to buy a product and then discover that it is faulty – or worse – unsafe. In business we have the same basic requirements; we just deal with a totally different set of ‘stuff’. The set of suppliers and vendors we deal with and their sub-suppliers and sub-vendors (i.e., the people from whom we procure our automation systems and support services and other business-related stuff), are broadly called our “supply chain”.

One way an attacker can get to our automation systems is to attack them in the supply chain rather than waiting until we have acquired them and our cyber defenses are all in place, and then, trying to attack them from afar. There have already been several instances of computer-based products – laser printers and ‘smart’ electric meters come to mind – that turned out to have secret malware and spyware embedded in their outsourced software or firmware. It is highly probable that more of this is happening as software development is subcontracted to low-cost, but questionably trustworthy, foreign sources.

Supply chain protection means establishing processes and procedures that reduce/eliminate the possibility of something like that happening to you. No one (probably including your automation vendor) wants to discover that the SCADA, DCS or ESD system they provided came with the malware ‘factory installed’. Your procurement process ought to be addressing vendor cyber security requirements, including how your vendors test their products and how they manage their software development and subcontracting.

As a former supplier, I must admit to being guilty of incorporating clandestine user accounts and ‘back doors’ into our systems for our own convenience. That way, when a customer called in a panic because they had screwed things up to where they couldn’t even log in any more, it was really handy to have a secret entrance that allowed us to remotely fix the damage they had inflicted. But of course, that means that there are now lots of former employees who know about those back doors and secret passwords. I’m glad that they’re all trustworthy individuals, and I’m even happier that most of those older systems have been replaced through the years.

You might think that the practice of including secret ‘back doors’ in control systems is bizarre and something that no reputable vendor would ever do, but I suggest having a frank discussion with YOUR automation vendors – this practice is far more common than most people think. Part of supply chain protection is ensuring that no such secret “open-sesame” access is built into your automation equipment.

Most of us understand that in the last two decades – and particularly in the last decade – our computer-based automation systems, whether SCADA, DCS, MES or ESD, have all migrated onto the same commercially-available hardware, operating systems, networking and communications platforms as use by conventional IT systems. In fact, we now hear discussions about “Cloud” SCADA systems and the use of virtualization in these kinds of systems. Yes, when automation systems were highly proprietary they were much less likely to be subject to cyber attack. But they also cost ten times as much and became technologically obsolete in short order, making support and maintenance a major issue.

So, this technology migration has had both good and bad outcomes. The basic fact is that our legacy computer-based automation systems are vulnerable to cyber attacks, not only because they have become far more standardized, but also because they are based on a mountain of software written in languages like ‘C’ that have integral weaknesses and security flaws. Hey, who knew?

As a person who has written a lot of software through the years – much of it in ‘C’ – I’m also guilty of making the assumption that no one would intentionally try to screw up my programs and cause them to malfunction. Of course, when I wrote programs that interacted with humans I knew that human error had to be considered, so I included some basic validity checking. But I never typed random gibberish as user input just to see how the program would respond (okay, maybe just a bit, but certainly not to any great extent!) The fact is, if my program’s input was coming from another computer or a local program, I just assumed that the input would be provided as specified.

I also presumed that the standard functions and libraries that were part of the programming language were properly designed and thoroughly tested and could be used with confidence. Boy, was I stupid – or, maybe just naive? One of the reasons that you get inundated with security patches and updates on nearly a daily basis is that vendors, researchers and the ‘bad guys’ are constantly looking through that software mountain and finding a treasure trove of design and/or coding flaws to exploit. There is actually an international black market for selling newly discovered (called ‘zero day’) flaws to the highest bidder.

The field of Software Assurance is the study of how to manage the design, development, testing, management, delivery and deployment lifecycle of software such that the software performs exactly as expected. This means no surprises – intentional or accidental – under a full range of environmental conditions, including intentional malicious attacks. In other words, the software is as free from exploitable flaws as is possible and designed to withstand and survive attacks without compromising system security.

Please understand that I am simplifying a very complex field into a few sentences, but this is the gist of the concept. The Department of Homeland Security (DHS) has done an excellent job of collecting a set of ‘best practices’ and creating a common body of knowledge on this subject, in cooperation with various educational and research institutions. And, they have created a set of quite useful pocket references that are freely available on their website: https:// buildsecurityin.us-cert.gov/swa/).

One significant outcome out of this study of software assurance is a set of coding/programming practices that greatly reduces the possibility of exploitable flaws. There are also ‘secure’ versions of language function libraries as well as utility programs that can check software for violations of these proper coding practices. One would hope that all vendors whose products are, or contain, software have adopted secure coding practices. One would also hope that they have programs in place to go back and remediate the flaws in their legacy code. Part of supply chain protection deals with the insistence that your vendors address cyber security in their product development processes and overall lifecycle – and require their software subcontractors to do the same.

Another aspect of software assurance is having procedures and processes that guarantee the executable code you receive from a vendor was generated from the approved source code and, that no unauthorized or undocumented additions or modifications were made to the source code along the way. Your vendor ought to have processes in place – a chain of custody if you will – that allows them to be totally confident that the ‘run-able’ code they provide to you can be directly associated with the approved source code and support libraries.

Something else that often happens with complex automation systems is that vendor field support and maintenance personnel often end up practically living at your site during system installation and commissioning. You get to know these folks and start to treat them like trusted employees and even give them an unescorted, all-access key card. Then at some later point in time you need some support and someone else shows up from that same vendor organization and you just assume that they will be as trustworthy as the personnel who used to support your system. But how do you really know? If you don’t know how your vendor vets their personnel and ensures their trustworthiness, you really can’t be sure that these new support personnel ought to be given free and unsupervised access to your plant and critical systems. Unfortunately, the notion that people with malicious intentions could work their way into third-party support organizations is not as far fetched as you might think.

Another aspect of supply chain protection is taking steps to verify that vendor/contractor personnel are acceptably trustworthy, either by requiring that your suppliers perform some basic level of background, criminal and financial checks, or performing these checks yourself prior to allowing supplier personnel into your facility. That is, although you would not be very likely to let a complete stranger enter your computer/ control room facilities, when a person shows up wearing a shirt that has your automation vendor’s logo on it, you probably don’t ask any – or at least not totally adequate – questions about who they are. Instead, many of you probably just lead them into your computer room or control center. That would be fine as long as you can be sure that your vendor is taking their personnel security seriously (an important part of supply chain protection) and ensuring the trustworthiness of their personnel. And even if they do, it would still be a good idea to call for a credential check before allowing them into your critical facilities.

A final but equally important aspect of supply chain protection is performing adequate factory and site acceptance testing on your automation systems and ensuring that such testing goes beyond merely doing a functional and performance test to also include security testing. It is always much more cost effective, and less disruptive, to fix problems at the factory rather than on site. Your acceptance process ought to include sufficient cyber security testing and validation aspects so that you can feel reasonably secure. There are good recommendations out there about the kinds of cyber security testing that you should require from your automation vendor. But that will be subject matter for a future column. – Tim

About the Author

Dr. Shaw is a Certified Information Systems Security Professional (CISSP) and has been active in industrial automation for more than 30 years. He is the author of Computer Control of BATCH Processes and CYBERSECURITY for SCADA Systems. Shaw is a prolific writer of papers and articles on a wide range of technical topics and has also contributed to several other books. He is currently Principal & Senior Consultant for Cyber SECurity Consulting, a consultancy practice focused on industrial automation security and technologies. Inquiries, comments or questions regarding the contents of this column and/or other security-related topics can be emailed to Tim@electricenergyonline.com.