As these separate developments have interacted over the past few decades, those who defend against cyber attacks have cycled through the same trial-and-error phase. There has been limited application of lessons learned from the experiences of others, either from a lack of information sharing or because it was (incorrectly) believed that past experiences did not translate into best practices. However, the nature of the threat has grown steadily, both in potency and in malice, and the time has come where re-learning old lessons is not an option.

In the 1980s, hackers had to work hard and possess unique skills and tools just to gain what is now considered public information. This resulted in a small number of dedicated, knowledgeable hackers, as opposed to the vast number of unskilled hackers encountered today. In the ‘80s, reliable sources of information were few and far between; public access to the Internet was nonexistent, the few subscription-based information services that had numerous points of presence (i.e., local phone numbers where one could dial in via modem) were devoid of the kind of rich knowledge we now take for granted, and the cost of long distance was much higher than it is today.

A key component of hacking was ”phreaking” – the hacking of telephone infrastructure. The number of useful bulletin board systems was so few, and the cost of dialing into them remotely so expensive, that getting free long distance was a crucial component of a hacker’s existence. A common occurrence was for a hacker to gain control of a system, only to immediately notify the administrator of that system of how they gained access and how to prevent it in the future. A major purpose of the exercise was exploration and learning… not destruction.

Eventually, the tools developed by this first group of hackers became available to others. The advent of public Internet access and the World Wide Web resulted in a situation where knowledge became far more accessible. Thus, came the dawn of the “script kiddie,” a type of hacker who had to rely on scripts and tools created by other more talented hackers, as the means to compromise systems.

As a result, the number of attackers grew exponentially while their innate skill level declined. Website defacements for nothing more than bragging rights became the most common form of compromise, resulting in embarrassment and the expense of a cleanup, but relatively little real damage. It was eventually recognized that good network design and maintenance practice, combined with diligent patching, was sufficient to reduce the impact of the script kiddies to little more than background noise.

In the end, all security decisions are economic decisions, and in the most recent evolutionary wave of hacking it has become clear that this rule applies to the attacker as well. Criminal organizations have become a significant force in the hacking scene, seeking to turn a profit through extortion or outright theft. This threat combines the advanced technical expertise of the first-generation hacker with a malicious intent that far exceeds that of the worst script kiddie. With the addition of the latest in offensive technologies (e.g., botnets, kernel-hooking and memory-resident rootkits, application-level attacks, etc.), an increased number of resources available to criminal enterprises, and the ability to perform large-scale coordinated efforts controlled by a multi-level organization, and the current state of threat becomes truly frightening.

Against this backdrop, consider the trend of interconnectivity among critical systems, and how far that trend has progressed.

Systems that used to run as islanded networks (or without any form of a network) are now connected, albeit indirectly, to the Internet. Even more challenging, in the power industry the dawn of Smart Grid technologies promises to vastly accelerate that rate of interconnectivity, creating links in ways that present ever-growing challenges for those who wish to protect the critical grid infrastructure that powers the world.

Consider a customer-facing web portal that connects to a back-end database, which in turn, is connected to a billing system. That billing system contains private data about customers and interacts with the head-end of an AMI (Advanced Metering Infrastructure) deployment spanning millions of meters, each with a remote disconnect feature. It is not hard to imagine how a sophisticated attacker would attempt to breach the security of the web portal, compromise the database and set disconnection flags for every meter, particularly in view of the breaches that have compromised millions of credit card accounts. The resulting outage would shed load on a scale far more severe than any other incident that has occurred since the first power line was strung.

The good news is that none of this is new. Other industries have had to face this kind of threat, and worse, and there are lessons to be learned from all of their efforts. The same mistakes need not be repeated by the power industry as more modern control systems are implemented and utilities go forward with new, network-reliant technologies like AMI.

The financial industry provides a comparable example for the ways in which they automate interaction with the general public. Automated teller machines – or ATMs – provide cash, give balance information and accept deposits from the public. Those machines, in turn, need the ability to interact with the account information of not only that bank’s customers, but also the customers of all other banks in near real-time.

As such, they are indirectly connected to the global banking system, even as they stand out in the public in shopping
malls, dark nightclubs, and grocery stores. At the other end of the environment, online banking provides a web-based interface where customers can pay bills, transfer funds, and even open new accounts, while having direct access to privacy-sensitive data.

The banking system has always been targeted by all manner of criminal enterprises from the most short-sighted robber to large-scale crime syndicates. While the industry seems monolithic and unchanging (much as the power industry seems to outsiders, even today), it has had to deal with dramatic changes from the shift in check processing that resulted from the “Check 21 Act” to the advent of e-commerce and commonplace use of debit cards to pay for items as menial as a cup of coffee, requiring greater and more immediate interconnectivity. All of these provide opportunities for exploitation by an attacker for direct monetary gain, and all of them have been attacked for just that purpose. As with the power industry, there is heavy regulation and oversight to ensure the availability, stability, security and governance of banks.

Financial institutions like banks learned the lessons of IT security first, beginning with how to think like an attacker when securing an environment. They were the first industry to hire consultants for this purpose, relying upon hackers to help them find and fix flaws in their defenses. Fortunately, the use of “Black Hats” (i.e., former hackers, sometimes also called “crackers”) is no longer necessary to accomplish the same goal. Other methods and techniques, from integrating security operations into the change control process to careful network design to control over where and how data is kept, all evolved first in the banking and financial industry as well.

But the most striking characteristic common to these institutions that became secure without disrupting opera­tions, was that they openly embraced the task of incor­porating security into their environment and processes.

To a subs­tantial degree, that work needs to be done in the power industry, and the sooner it is tackled, the better it
tends to be performed and the less pain is felt from the changes that occur. This becomes especially true when regulatory standards come into play, incurring specific deadlines. The sooner the process of seeking compliance with those standards begins, the more time there will be to involve all stakeholders and make sound decisions.

One of the most overlooked emerging security practices, however, is arguably the most important. Security of products and systems at the application level is the next place where the most positive effects can be realized, after the implementation of basic best practices. Security needs to be a driver in the design process for all vendors moving forward. It can be considered another form of reliability, only against a much nastier form of outage, resulting from undetected and/or undefended security breaches.

Nowhere is this aspect of security going to be more important than in AMI. Unlike the traditional islanded networks of past control systems, AMI involves a significant amount of infrastructure that is accessible to the general public. Wireless communications are ubiquitous, and there is no way to add security measures after the fact.

There is not – and likely never will be – a firewall, antivirus, or intrusion prevention solution that can be added to an AMI meter or collector. Even if such a solution were created, the cost of it would be prohibitive when dispersed over the millions of meters of a comprehensive solution, and management would be a nightmare. The only line of defense is that which resides within the metering solution itself; if the products in use are insecure, then that insecurity inexorably translates to the larger infrastructure.

At the Black Hat Briefings in 2009, a security researcher stood up and disclosed a long list of security vulnerabilities in AMI solutions. The specific vendors were not named, but there were four in total that were assessed, and the impression given was that none of them lacked glaring flaws. The crowning achievement of the security testing was the creation of a worm that was able to spread from meter to meter, wirelessly, until the entire infrastructure was under the control of the attacker. Some of the problems manifested in software flaws, while others were related to hardware or architectural design.

The latter two are more insidious, because they cannot be fixed with a simple software update or patch. In such cases, the cost of fixing the security holes is equivalent to, or greater than, the cost of the initial AMI rollout, as every meter must be physically upgraded, modified or replaced. This is precisely the kind of situation that every industry has faced previously, and only addressed after years of incidents eventually forced vendors to make secure design a critical part of their development processes. Unfortunately, metering implementations historically tend to last for decades, which is a sobering thought considering how long that gives attackers to find and exploit flaws.

Conclusion
For decades, security has been an afterthought that later became accepted as a needed and necessary component of any major environment or system. In the past, the process of trial and error that led to that evolution has been able to move forward without significant impact, but every iteration of the cycle has brought a higher and higher cost to the errors. With AMI in particular and Smart Grid technologies in general, there is no flexibility for a trial and error approach to the security challenge being faced today.

Meters are in fixed locations as are substations and transformers; thus, one cannot change the layout of the power grid simply because it’s safer from a networking perspective when insecure products are in use. Moreover, the impact of such an incident would be enormous, far beyond what is acceptable or in some cases, imaginable. The public outrage at an incident that could be attributed to an attack would result in legislation far more stringent than anything yet seen by the power industry, but ironically, that legislation would probably do little to improve security.

Today the stakes are higher than ever, the threats are more dangerous than before, and the alternatives to integrating security into our power infrastructure are few. The best time to act is now, before the vulnerabilities and flaws that already exist become locked into our national power delivery and distribution infrastructure.