Cyber-attacks against critical infrastructure sectors in general and the energy sector in particular have escalated at an alarming rate in recent years. Within the energy sector, municipally and cooperatively owned utilities are among the most vulnerable targets because they have the same systems and front-page appeal as large investor owned utilities (IOUs) but frequently lack the financial and human capital needed to implement large and lengthy enterprise level security projects. By necessity, public power and cooperative utilities have had to find a different approach because doing nothing was not a viable option – not only for their own security but also for the larger grid to which they are connected. By focusing on their strengths including a more nimble decision process and interconnected regional networks some municipally and cooperatively owned utilities are implementing innovative solutions that could have the larger IOUs rethinking their approach.
This article reinforces the cyber threat confronting the energy industry and focuses on topics including: current trends, category vulnerabilities and why the solution for cooperatives and municipal utilities needs to differ from those available to large IOUs. Finally the article spotlights a cyber-security project currently underway at a cooperatively owned G&T where some of these initiatives are being implemented. For security reasons the name of the utility is not disclosed.
A large and growing problem
The Industrial Control Systems Cyber Emergency Response Team is the department of Homeland Security division that responds to cyber-attacks against critical infrastructure installations. In 2009 they responded to nine reported incidents.1 Since then, they have become much busier.
Last year ICS-CERT responded to 198 cyber-attacks against critical infrastructure targets including electric and water utilities, pipelines and nuclear facilities.2 These attacks represented an increase of more than 50 percent versus the previous year. In 2013 that figure is on track to double. In the first half of the fiscal year (October 2012 to May 2013), ICS-CERT responded to more than 200 incidents, exceeding the 12-month total from the previous year.3 A disproportionate number of these incidents continued to originate from the energy sector.
In the 2012 report, 41 percent of the reported incidents targeted the energy sector. By the mid-year 2013 report the energy sector accounted for 53 percent of the incidents. While the report does not explicitly state why the majority of all incidents targeted the energy sector, security experts frequently cite size and response hurdles as contributing factors.
The attack surface of the grid is simultaneously becoming larger and more vulnerable. The grid’s vast network of interconnected generation, transmission and distribution systems creates an attack surface that ranks second only to the internet in scope. As the grid continues to grow through increased interconnectivity and interoperability it is also becoming much less secure. Remote assets like sub-stations, previously islanded from enterprise information systems, are being brought into SCADA environments without adequately hardening systems against the vulnerabilities these new additions create. Despite the increased risk, defensive initiatives by utilities have been hampered by cultural, organizational and resource issues.
RESPONSE CHALLENGES
A Culture of Compliance
Utilities are both legally and culturally driven by regulatory requirements and standards. On the plus side of the equation, this culture of compliance promotes uniform policies that create an industry-wide level of quality and reliability. On the negative side it can discourage independent initiatives that become necessary when the problems are progressing faster than the policies. This latter situation is confronting utilities who are struggling to span the divide between compliance and real security.
Part of the problem is with the policy makers. A report issued by the Government Accountability Office (GAO) stated that: Aspects of the current regulatory environment make it difficult to ensure the cyber security of smart grid systems.4 The report specifically noted the concern about the ability of regulatory bodies to respond to evolving cyber security threats. This concern has been echoed by elected officials, security agencies and regulatory bodies themselves.
Former FERC chairman Wellinghoff frequently addressed this topic including the following statement which points to the lack of an actionable command and control structure.
“My intent is to simply have somebody be put in charge,” Wellinghoff said in response to a question from The Hill at the Platts Energy Podium in Washington, D.C. “And the person who’s put in charge has the authority to tell those entities that are responsible for the infrastructure of the immediate threat and vulnerability, and to order them to do something if necessary to mitigate that threat or vulnerability.”
“That’s my really, bottom-line message. It doesn’t have to be FERC.”5
While it is clear that the regulatory structure needs to improve, utilities also bear some of the responsibility themselves for the slow pace of change. The same GAO report referenced above makes it clear that both utilities and oversight agencies need to progress beyond mere compliance.
Utilities are focusing on regulatory compliance instead of comprehensive security. The existing federal and state regulatory environment creates a culture within the utility industry of focusing on compliance with cyber security requirements, instead of a culture focused on achieving comprehensive and effective cyber security. Specifically, experts told us that utilities focus on achieving minimum regulatory requirements rather than designing a comprehensive approach to system security. In addition, one expert stated that security requirements are inherently incomplete, and having a culture that views the security problem as being solved once those requirements are met will leave an organization vulnerable to cyber-attack. Consequently, without a comprehensive approach to security, utilities leave themselves open to unnecessary risk.
But even utilities that are willing to step beyond mere compliance measures still have to wrestle with implementation issues including organizational accountability gaps that have significant cyber-security ramifications.
The IT / OT Gap
Historically, the responsibilities of utility information technology (IT) departments were confined to the servers and computer systems that housed customer data, billing information and other digitally stored data. At the same time operational groups concerned themselves with the performance, maintenance and reliability of assets. For the most part, the virtual concerns of the IT group did not intersect with Operations’ concerns for physical assets. Today, the operation and management of physical assets is increasingly being managed through virtual systems and as a result departmental lines have blurred. IT groups struggle with the introduction of disparate and often unsecured assets into the information technology environment. Operational groups find themselves in need of support from IT groups that do not operate on the 24x7x365 schedule that operational staffs do. This Informational Technology / Operational Technology (OT) gap has not escaped the notice of hackers.
At Black Hat USA 2013, a hacker conference in Las Vegas, one of the most popular sessions highlighted a hacker’s ability to gain control over utility systems through remotely deployed assets. The following is an excerpt from the description of the session from the conference’s website:6
In this presentation, we review the most commonly implemented key distribution schemes, their weaknesses, and how vendors can more effectively align their designs with key distribution solutions. We also demonstrate some attacks that exploit key distribution vulnerabilities, which we recently discovered in every wireless device developed over the past few years by three leading industrial wireless automation solution providers. These devices are widely used by many energy, oil, water, nuclear, natural gas, and refined petroleum companies.
An untrusted user or group within a 40-mile range could read from and inject data into these devices using radio frequency (RF) transceivers. A remotely and wirelessly exploitable memory corruption bug could disable all the sensor nodes and forever shut down an entire facility. When sensors and transmitters are attacked, remote sensor measurements on which critical decisions are made can be modified. This can lead to unexpected, harmful, and dangerous consequences.
Security issues associated with this increased interconnectivity between IT and operations are starting to get attention from the energy industry and cyber-security vendors alike.
In a recent Public Power Daily article7 APPA President and CEO Mark Crisson advised member utilities to determine if their Operations Departments shared any systems with enterprise or information technology departments to be sure that “…one doesn’t provide a backdoor to the other.”
Large, enterprise IT consultancies are also aware of the IT/OT gap and have added terms like ‘Operational Technology’ and ‘IT / OT Convergence’ to their utility targeted offerings.
Resource Limitations
While IT/OT offerings from large, enterprise IT consultancies is a step in the right direction, it is not a panacea. The high cost puts these solutions beyond the reach of all but the largest IOUs.
Large consultative cyber-security projects can last a year or longer with costs that can run into seven digits for fees alone. Based on industry standards, these types of enterprise level cyber-security engagements are estimated at 15 percent of cost of the total IT system. This puts them out of the reach of most municipal and co-op utilities.
A Surprising Trend
Based on the cultural, organizational and especially resources challenges articulated thus far it would be easy to assume that larger IOUs are more secure than municipally and cooperatively owned utilities.
Surprisingly this may not be the case.
While municipally and cooperatively owned utilities may not have the resources of larger IOUs they make up for it by being less resistant to new solutions and a more nimble decision making process. This observation comes from first-hand experience dealing with all three types of utilities and is reinforced by data from a recent congressional report.8
In January 2013 Representatives Edward J. Markey and Henry A. Waxman requested information from utilities including 150 IOUs, municipally-owned utilities, rural electric cooperatives and federal entities. In addition to revealing that some of the utilities were under ‘a constant state of attack,’ the report concluded that: Most utilities only comply with mandatory cyber-security standards, and have not implemented voluntary NERC recommendations.
For example 91 percent of IOUs, 83 percent of municipally- or cooperatively-owned utilities reported that they were compliant with mandatory measures related to the Stuxnet virus. By contrast, 21 percent of IOUs and 44 percent of municipally- or cooperatively-owned utilities reported compliance with the voluntary measures. Clearly, the non-mandatory response by all utility categories could have been better, but it was interesting to note that despite a resource disadvantage, the voluntary security response by municipally and cooperatively owned utilities was much stronger.
Municipally and cooperatively owned utilities may be more willing to adopt non-mandatory security measures because they are more closely tied into the fabric of the communities they serve. They take great pride in service delivery; many of their CEOs / General Managers will say: “my neighbor is my owner, and my aim is to provide the best possible service.” It would be stretching the facts to definitively state that these results mean that municipally and cooperatively owned utilities are more secure than IOUs, even within this limited data set. It does, however, suggest that the municipally and cooperatively owned utilities are more willing to go beyond mere compliance and seek outside resources to help them achieve real security.
One example of a proactive approach to cyber-security is the work N-Dimension Solutions, a cyber-security firm specializing in critical infrastructure assets is doing with a large cooperatively owned Generation and Transmission (G&T) utility in the southwestern part of the United States. The utility is not identified for security reasons, but N-Dimension has a long track record of working with both municipally and cooperatively owned utilities and related associations.
“The American Public Power Association is working hard to educate public power utilities on the cyber security risks to their operations” said Jeff Haas, Vice President, Membership and IT, American Public Power Association. “N-Dimension has assisted APPA develop tools and deliver information to improve the cyber security posture of public power systems”.
A cooperatively owned utility case study
The utility in this case study is a tax exempt, consumer-owned public utility that provides low cost, reliable electric service for its rural distribution cooperative members. Its member systems serve retail consumers located across a large geographical footprint.
The members are interconnected through a private Wide Area Network (WAN) using AT&T Multi-Protocol Label System (MPLS) routers. This network is private in the sense that data on the network is isolated from non-network members. The data itself, however, was not encrypted.
Nothing about the network was out of compliance with NERC CIP standards including the fact that the data was not encrypted. Nevertheless, when the G&T upgraded its AMI metering system, they also took the opportunity to improve the level of security on their network.
For several years, N-Dimension Solutions had been working on cyber-security solutions specifically designed for municipally and cooperatively owned utilities. In contrast to the costly, all-consuming engagements of large technology consulting practices, the more specialized company focused on providing a product based solution that was modular, extensible, and most importantly affordable.
Through a close working relationship with both the APPA and the NRECA, N-Dimension had already contributed to the development of a significant number of the cyber-security guidelines, educational and training materials that each service organization provides to its respective members.
“Cyber-security for utilities requires a more complex solution than just securing IT systems especially for our Generation & Transmission cooperatively owned facilities,” said Craig Miller, CTO NRECA. “N-Dimension provides cyber security offers that are uniquely suited to protect the operational assets of G&Ts and their co-op members”
After reviewing the utility’s systems, N-Dimension identified a number of potential attack vectors including:
- SCADA and AMI data collected from member SCADA systems over a private IP-based WAN leased from a Telco
- Security of this WAN requires 100% security of each and every router and link in the network, which includes a diversity of devices both owned by the telco and leased from local providers
- Physical access to any of the routers or links in the WAN
- Varying security of member SCADA systems on the WAN, including direct connections of those systems to member corporate networks and the public Internet
A larger concern than the specific threat vectors was the fact that an attack on any individual point of access opened the entire system to a cyber-attack. Based on this vulnerability simple objectives were established.
- Protect G&T systems from compromised member systems
- Protect non-compromised members from a compromised member
The solution to meet these objectives required four key capabilities:
- Data encryption
- Firewalls
- Intrusion detection
- Dynamic routing
The first three capabilities were already incorporated in N-Dimension’s n-platform,™ a comprehensive cyber security unified threat management (UTM) software platform, designed specifically for protecting critical infrastructure that can be deployed across entire operational networks. The fourth capability was developed specifically to address this utilities configuration.
The solution uses point-to-point SSL VPNs to encrypt traffic across the private WAN between cooperative member SCADA systems and the G&T SCADA system. Encrypting traffic across the WAN ensures that the security of this traffic cannot be compromised by an attack against the Telco network. Traffic between member and G&T SCADA systems typically traverses 10 to 20 routers, and encrypting the traffic removes all links and all but the two endpoint routers as potential points of attack.
In addition to data encryption firewall rules were established to restrict the types of traffic allowed to traverse the SSL VPNs to specific protocols. Unlike a typical corporate network, the number of protocols that need to be allowed across this network of VPNs is very small. The allowed protocols are essentially just ICCP, SCADA synchronization, and AMI. Restricting the types of traffic allowed across the network significantly narrows the types of attacks available to an adversary to only those that exploit weaknesses in the allowed protocols. Previously, any type of attack would have been conveyed by the WAN from a compromised member system to the G&T network. NetBIOS and other undesirable traffic originating from member systems had previously been found on the G&T network, and are now blocked by these firewalls.
To guard against attacks that bypass the above protections, Intrusion Detection Systems (IDS) were implemented at each of the cooperative members and at the G&T to monitor traffic received from the VPNs and detect potential attacks and attack indicators. Tuning IDS systems on corporate networks is recognized as a difficult problem, but since the firewalls severely restrict the types of traffic that can be transported across the VPNs, tuning these IDS sensors is significantly simpler. The IDS alerts from the individual sensors are aggregated into an Event and Log maintained in the n-Central monitoring system run by the G&T, to provide network-wide awareness of threats.
Due to the broad geographical nature of the G&T and its members outages of individual communications links across the WAN are common. With the above technologies in place, using the public Internet connections as a backup communications resource was a natural extension. The solution uses SSL VPNs across the Internet to provide backup channels, and dynamic routing to automatically switch from the SSL links across the private WAN to SSL links across the Internet when there are connectivity issues with the WAN. Previously the G&T suffered approximately one SCADA communications outage per month, but with backup Internet communications, outages of SCADA data have all but disappeared. Outages now occur only when a member’s Internet is provided by the same Telco that provides the WAN, or shares the same physical infrastructure.
All of these measures were implemented in close cooperation with the G&T’s IT and Operational departments. The latter consideration was especially important because of the mission critical nature of the operational systems.
Moreover the project was more product focused than consulting based which fit the financial and human resource parameters of the cooperatively owned utility. Industry standard costs for cyber-security projects range from 15 to 20 percent of the facilities overall IT system cost. This project was closer to the five percent range.
It bears repeating that these measures were not required to comply with mandatory cyber security standards. They were motivated by the G&Ts recognition of the escalating threat of cyber-attacks and a genuine desire to ensure reliability, revenue assurance and risk mitigation for its member/owners. Moreover, the solution implemented was tailored to the specific needs of the G&T enterprise and operational systems. In our experience this bias toward effective action is characteristic of both municipally and cooperatively owned utilities who just might be leading the way into a more secure future for the smart grid.
About the Author
Doug Westlund is an expert on the role of Operational Technology in the development of defense-in-depth cyber protection for critical infrastructure assets and is a regular speaker and presenter of cyber security topics in the energy sector at industry conferences across North America.
He co-founded N-Dimension Solutions in 2002 because cyber security solutions designed to protect enterprise systems did not have the Operational Technology rigor to protect critical operational assets from sabotage. Since then, N-Dimension has grown into a leading cyber security solutions provider for the critical infrastructure segments. As CEO, Doug is primarily focused on developing and implementing N-Dimension’s strategic plan and business development activities such as developing N-Dimension’s eco-system of strategic partners.
Prior to N-Dimension, Doug was a SCADA Engineer with Valmet Automation, a Business Development Manager with Motorola Information Systems, and a Vice President with AT&T Canada. Doug holds an Engineering degree in process control, an MBA, and is a licensed Professional Engineer.
References
1 http://ics-cert.us-cert.gov/sites/default/files/documents/ICS-CERT%20Incident%20Response%Summary%20(2009-2011).pdf
2 http://ics-cert.us-cert.gov/sites/default/files/Monitors/ICS-CERT_Monitor_Oct-Dec2012.pdf
3 http://ics-cert.us-cert.gov/sites/default/files/Monitors/ICS-CERT_Monitor_Apr-Jun2013.pdf
4 GAO-11-117
5 http://thehill.com/blogs/e2-wire/e2-wire/241347-obama-official-feds-need-more-cybersecurity-oversight-for-electric-grid#ixzz2hbry3pqO
6 http://www.blackhat.com/us-13/briefings.html#Apa
7 Public Power Daily, October 23, 2013 “Cyber and physical security plan is key to risk management, Crisson says”
8 http://democrats.energycommerce.house.gov/sites/default/files/documents/Report-Electric-Grid-Vulnerability-2013-5-21.pdf
