There has been much written about security recently – and will probably (rightfully) continue to be for along time to come. These days, security is top of mind in many aspects of our daily lives, whether it involves shipping, travel, information management or – for those of us in the utility space – critical infrastructure protection. By contrast, security was hardly ever mentioned prior to the September 11th attacks, except perhaps among special interest groups already focused on security as a business. This dramatic shift illustrates just how much our thinking about security has changed in less than a decade.

Although some would say that seven years seems like a long time for contemplation, it really isn’t in the traditional utility vernacular. Indeed, it’s probably just enough time to get our arms around a challenge of such huge proportions and potentially ubiquitous consequences. Indeed, besides being a major issue itself, security is also caught up in the transformation of the grid – commonly referred to as the Smart Grid Initiative – from a relatively passive, 1-way network into a highly intelligent 2-way, self-healing architecture of which security is a fundamentally vital part.

It is also well known that the realization of current and future SGI goals and objectives will require an unprecedented level of capital-intensive infrastructure investment by virtually every utility from the smallest to the largest load-serving entities. There are so many different dimensions of security these days that it’s hard to even know where to begin.

Panel of Experts Speaks Out On Security…

Ernest Rakaczky has played an active role within the Process Control arena for over 31 years. He is currently the Principal Security Architect for Invensys Process Systems (www.ips.invensys.com) and a key member of the Control Security Team in this position.

Rakaczky participates in the efforts under way at ISA within SP99, NIST within PCSRF, MSMUG and plays an active role in the various Security initiatives with DOE, DHS, INL, NRC, IAEA, Process Control Systems Forum (PCSF) and Sandia Labs, most recently being appointed to the PCSF Governing Board as the control vendor community representative.

He is a founding member to the Canadian Industrial Cyber Security Council and was most recently appointed by Public Safety Canada to chair an active working group to define the Cyber Security Requirements for the Canadian Critical Infrastructure. With the formation of the ISA Security Compliance Institute, ISCI, has been elected as the Marketing Chair of the initial Governing Board.

Jonathan Pollet, VP of North American Operations at Industrial Defender, Inc. (www.industrialdefender.com), brings a blended history of more than ten years of experience in supervisory control and data acquisition systems (SCADA), distributed control systems (DCS) and cyber-security solutions in both disciplines to the company. In recent years, Pollet has led combined physical security and cyber security teams on over 100 SCADA and DCS vulnerability assessments for critical infrastructure facilities.

Stephen Rubin, Longwatch (www.Longwatch.com) President & CEO, has over 30 years experience in the software industry. Rubin was the founder and CEO of Intellution, Incorporated, a worldwide leader in the development and application of process control software for personal computers. Elected a Fellow of the International Systems and Automation Society (ISA), he is a graduate of the Worcester Polytechnic Institute where he also serves as a member of the WPI Board of Trustees.

Anthony Clem is a Senior Security Architect for the Hewlett Packard Americas Security Practice (www.HP.com). Anthony has focused on compliance consulting for HP for the past eight years in the retail, financial, and energy markets for SOX, PCI and NERC-CIP. Anthony has over twelve years in security experience and 15 years experience in IT. He previously worked in security for U.S. government agencies and was also involved in building early Internet banking architectures.

Andrew Wright holds a Ph.D. in Computer Science from Rice University. Dr. Wright is the Chief Technical Officer for N-Dimension Solutions (www.n-dimension.com), responsible for technical product strategy and direction. He is also working with IEEE working group 1711 to make AGA-12 an IEEE standard, with Idaho National Lab to develop best practices for securing industrial control networks, and with ISA’s SP99 Working Group 4 on secure control system requirements. He has published over 20 technical papers and has 16 years of experience in industrial research and development.
Prior to joining n-Dimension Solutions, Dr. Wright was a Technical Leader in Cisco’s Critical Infrastructure Assurance Group (CIAG), where he developed cyber security solutions for critical infrastructure, and particularly for Industrial Control Systems and SCADA. He established the Cisco Secure Control Systems lab in Austin TX, was the key architect of the AGA-12 serial SCADA encryption protocol, and was a founding developer of CVSS, the Common Vulnerability Scoring System.

Deryk Yuill received his Bachelor’s degree in Electrical Engineering from the University of British Columbia in 1984. He has spent his career in a variety of product development and management roles in the telecommunications and utility automation industries. Deryk joined Bow Networks (www.bownetworks.com) in December 2001, where he serves as Vice President of Technology and is responsible for the definition, sales and marketing of the Company’s substation communications, security and data integration products.

In 2006, the North American Electric Reliability Corporation (NERC) adopted a set of critical infrastructure protection (CIP) standards with their primary mission being to protect the nation’s bulk power system against cyber attacks that could potentially disrupt the provisioning and operation of the electric power grid. As Chairman Kelliher’s comments (below) suggest, security and smart grid initiatives are inextricably intertwined.

In January 2008, these NERC-CIP standards were approved by the Federal Energy Regulatory Commission (FERC), making them mandatory and enforceable with significant sanctions and penalties for non-compliance. These standards establish the minimum requirements for cyber-security protection and are a good framework and foundation to build a more solid protection against cyber-security breaches. Like all current Cyber Security Guidelines/Practices/Standards there is a common set of requirements in the implementation and management of a successful cyber-security program.

With fines up to $1 million per day – per NERC CIP violation – electric utilities must be prepared to support these compliance requirements.

Dr. Andrew Wright, Chief Technology Officer for n-Dimension Solutions, is quick to point out that there is no “silver bullet” to address all cyber security problems – a view that is shared by substantially all of the security experts we interviewed for this article. For many organizations with critical infrastructure to protect – presently focused on utilities involved generation or transmission of energy at the BES (Bulk Electric Supply) level, which has traditionally been set at or above 100 kV – achieving the level of cyber-security protection required by NERC-CIP can be a daunting task.

Figure 1 illlustrates the correlation between NERC-CIP compliance requirements and the established ISO/IEC standard. The overall implementation of the cyber-security program will require a very strong collaboration across all standards and elements within the operational environment, but perhaps even more important, long-term success will depend on the awareness, understanding, acceptance and adaptation to the new set of behaviors that any successful program will require.

“Keeping in mind that there is no one single product that can meet all the stringent requirements imposed by the NERC-CIP standards, an organization should not base their cyber-security protection solely on a single device – or class/category of devices – such as firewalls. Comprehensive cyber-security is achieved through a combination of physical, technological and human elements that must work together to arrive at a complete ‘best practice’ solution,” according to Wright.

Based on their research, NERC has stated that the Top 10 Cyber-security Vulnerabilities facing the industry are:

1. Inadequate policies, procedures & culture
2. Insufficient defense mechanisms
3. Lack of control at remote access points
4. System admin mechanisms
5. Wireless networks/communications
6. Shared communications channels
7. Lack of tools, forensic and audit methods
8. Installation of inappropriate applications
9. Unauthenticated control systems command and control data
10. Inadequately managed, designed, or implemented critical support infrastructure

“Our company and our partners know the industry and know the challenges that an operator faces,” Wright continued. “The provisioning of a wide range of cyber-security solutions in both products and services is to accomplish one objective; that is, to assist critical infrastructure organizations of any size anywhere to achieve the highest standards in cyber security and conform to industry regulations.”

Industrial Defender’s Jonathan Pollet agrees with Wright that there is no panacea solution when it comes to cyber-security: “A truly secure Smart Grid should defend itself at multiple points throughout the system and should use active defense systems like firewalls or universal threat management (UTM) devices to actively stop attacks at the touch points. Also, Intrusion Detection and Intrusion Prevention technology should backup the firewalls and UTM devices to add another layer of protection.

All devices and system components should create security events and logs with the logs centrally collected for event correlation, incident response, forensics and audit trail. Core system components should have redundancy so that system continues to work, even while under attack. And to prevent any fraudulent activities, the system should use strong encryption and authentication methodology.

“Open systems are now commonplace,” Pollet notes, “but when it comes to open systems utilities should rethink the ‘open’ model where all meters can be read by everyone and data shared openly. If the utilities own the system, they also own the risk. So before opening up the system, it is prudent to consider a model where participants push data out on a prescribed basis using a secure protocol.”

Pollet also stresses that it is far easier to design security into the system up front: “We need to think about basic architecture and security standards such as ISA99 (refer to Figure 2) and NERC-CIP well ahead of the implementation curve,” Pollet warns.

Ernie Rakaczky of Invensys Process Systems reminds us that Pollet’s view regarding Smart Grid impact on existing connectivity models also extends to legacy installations of critical control systems. “Over the past decade, suppliers of critical control systems have made tremendous efforts to ensure they will operate in an environment that is open, interoperable and continue to take huge steps in defining a more secure operating environment for these systems,” Rakaczky points out.

“The reality is that within our current grid infrastructure we have an operating environment of control systems that could easily date back a decade or more. We are now facing one of our first big challenges in creating a Smart Grid environment, so as we begin to put our modernization plans into place those plans must also include the modernization of these critical control systems,” Rakaczky warns, “for it is only through that modernization will we be able to take advantage of the full requirements and benefits of the Smart Grid.”

Indeed, one of the big issues that will have to be faced is the traditional 15- to 20-year life cycle estimate for control systems. That figure will probably have to be shortened since an increasing number of elements within these systems – such as the embedded operating system software upon which these systems are based – will have abbreviated life cycles as they adapt to rising numbers and types of security threats as well as for various other reasons. Moreover, it will be critical that all existing control systems be supported for all security issues for as long as these systems are installed and operating.

Rakaczky also agrees with Pollet when it comes to the secure transfer of information across critical control system networks: “The other key element for the control systems within the Smart Grid environment will be the ability to more effectively manage the overall cyber-security profiles of their interconnected networks and the protective technologies that will be in-place (i.e., policies, procedures, firewalls, IDS/IPS, etc.). If not already, this will quickly become a full-time (24x7x365) management requirement,” Rakaczky warns.

“And unlike today – where this environment is usually predictable, repetitive and fairly simplistic – requirements dictated by the new Smart Grid environment will likely create obstacles not unlike those encountered in banking, information management and other business environments where this characteristic of network change is quite prevalent. Determination of whom and/or what has authorized access; what category, class and type of information is needed; and which devices within the networks should be accessible must all be assessed and must all be decided well in advance of access being granted,” Rakaczky continues.

“From the outset, the key to success will be to start building a fully functional security operation center (SOC) for each Smart Grid operation that will have a full cyber-security responsibility for the actual control devices, systems and networks as well as the overall security infrastructure,” advises Rakaczky.

Steve Rubin, President of Longwatch, notes that important steps need to be taken to secure physical assets as well as those in cyberspace. Moreover, Rubin envisions emerging intelligent grid initiatives promising greater system reliability, uptime, safety and security. Achieving incremental improvements, reducing operating expenses and minimizing capital expenditures, while extending the value of installed infrastructure are but a few of the many objectives that Rubin believes will be logical outcomes of the transformation to an intelligent grid.

“Integrated, widespread video can deliver much of the information needed to support and help achieve those objectives,” Rubin states. “Recent advances in video technology and software make it possible to transmit digital video over a power plant’s existing network, such as Ethernet or wireless, and to put the video images on HMI/SCADA workstations in the control room,” says Rubin. “This technology allows high-resolution video to be stored at the remote site for up to 30 days, and low-resolution ‘video clips’ can be sent to the HMI/SCADA system whenever an event occurs. Each clip can also be configured to show footage before, during and after the event. And, the operator can switch to a live video feed at any time, and pan and zoom the camera remotely,” Rubin explains.

Rubin goes on to say: “Some plants are already using digital video clips to monitor multiple sites using a pre-scheduled scan period, presenting the operator with updated still images of each remote site. Up to 24 video images can be put on a single screen and updated at speeds up to once per minute, depending on the bandwidth available. By using the existing plant network, as many cameras as needed can be installed around a given facility – be it a substation, generating plant or other type of asset installation – without the need for dedicated cable, with all video from that location easily brought into the HMI/SCADA system for display.”

“Intruders can still be detected by conventional devices such as door switches, motion detectors or other types of sensors, but the video software can also detect an intruder entering through analysis of the video image itself. Then, after security has been alerted and the police called, high-resolution video can be downloaded from the remote site and used as evidence in criminal proceedings,” Rubin notes.

Equally important, video can be used by operators to monitor power equipment, investigate the cause of a control problem, or verify that procedures such as startup or shutdown are being carried out properly. For example, if a problem occurs at a remote substation, technicians will know whether they need a shotgun or a toolbox to fix the problem. With video on HMI screens, operators can see what is happening anywhere in the facility, 24/7/365.

Deryk Yuill, VP of Technology at Bow Networks, also focuses much of his attention on securing the installed base. “It’s natural that much of the discussion around cyber-security involves new technology,” says Yuill. “These discussions are useful to have, but the massive installed base of equipment is frequently neglected, as a practical matter. It will take years – if not decades – before even a substantial portion of legacy installations can be upgraded or replaced, as Ernie Pointed out,” Yuill agreed.

“In talking to a large number of utilities about NERC-CIP compliance, it has been interesting to observe their response to it and the two main threats that accompany it,” Yuill continues. “The primary threat is that of a cyber-attack, whether accidental or malicious. The secondary threat is that of fines, which can be levied on non-compliant utilities.”

“There is quite a variety of attitudes about this, Yuill says, ranging from those who are primarily concerned by the security threat to those appearing to be more concerned by the threat of punitive fines for non-compliance. The good news,” says Yuill, “is that there are reasonable ways of securing most of the systems utilities have in service today.”

“Utilities that are committed to security and focus on these fundamentals should find that cyber-security is an attainable goal,” Yuill continues. “I believe that for a utility that has built a good security program, the compliance burden can be substantially minimized. Furthermore, as the bar is raised on NERC-CIP compliance – a virtual certainty – those utilities that have focused on security first will be best prepared to adapt to the more stringent requirements.”

HP’s Anthony Clem believes that after achieving security compliance, utilities must document security controls extensively to prove compliance. “NERC-CIP mandates require security event monitoring, incident alerting, forensic analysis and event data retention,” Clem contends. “Therefore, it is no longer practical or operationally feasible to assemble and store records manually. That’s because utility companies will likely need to manage many documents to meet compliance and collect log files from multiple applications and servers. With sanctions, significant fines, more frequent audits, and increased federal oversight, can any utility afford the ramifications of non-compliance?” Clem asks.

Besides acting as an additional line of defense against cyber-security threats, a complete security solution should also help automate compliance tasks, which can be tedious and time-consuming.

“As others have already said, the energy and utilities industry is experiencing unprecedented pressures to transform the way it delivers energy and interacts with its customers,” Clem observes.

“Going forward, the Smart Grid will increasingly rely on advanced technologies, which besides bringing better power management and automation to the meter, new services and business opportunities for utilities and industry alike will also be created. The Smart Grid will manage and distribute electricity and operational information through an extensible, reliable, digitally managed network. This fully two-way communication environment will deliver asset optimization and efficiency opportunities for utilities,” Clem explains.

The new expanded network also brings emergent risks associated with advanced technologies. Subsequently, utility providers must ensure that appropriate measures are in place to protect the extensive information flow and control signals intrinsic to the Smart Grid. Protecting both the operations/control network and enterprise network is paramount, as sophisticated cyber attacks are on the rise and increasingly targeted toward critical infrastructures.

A Final Note…
Of course, there are many ways to approach security, but it seems clear that most if not all of the experts agree that adoption of a comprehensive, holistic approach that embraces both new and legacy installations as well as both cyber and physical security is the easiest, fastest and least expensive way to achieve compliance.

Complying with the tide of mandates creates greater overhead expenses and data-management headaches. Therefore, to address these new cyber-security requirements, it is imperative that an overall security approach not only address the compliance standards, but also leverage the tools, processes and investments made to support the broadest and most comprehensive security vision possible.

[These and related topics will be addressed in Part 2 of this article, appearing in our Jan/Feb 2009 issue.]