Today’s electric utility substations and distribution feeders contain and interact with a variety of assets, including substation computers, intelligent electronic devices (IEDs) and software applications. They are serviced by utility personnel who must access certain systems and IEDs but are not permitted to log into others. Permissions may vary depending on the organization, individual roles, training, skillset and authorizations.

Managing access control is complicated, especially when assets number in the hundreds of thousands, as is the case with electric utility substations and distribution feeders. Adding distributed energy resources (DERs) — which may be owned and operated by third parties — to electric utility distribution networks further complicates securing assets and access control authorization.

Comprehensive access control requires more than simple communication network protection. Otherwise, it’s all too easy for malware to make its way through protected communication network connections, even those secured by firewalls, via transient systems such as laptops used as human-machine interfaces (HMIs) by service technicians, remote access tunnels or other means — with devastating consequences. Protecting the distribution network is even more challenging and complex as hundreds of thousands of sensors from various manufacturers are deployed to automate various functions or provide deeper operational awareness within the grid.

Focusing solely on communication network security means a hack on any asset inside the utility’s electric distribution network can cause a contagion that could compromise the entire utility. Utilities must protect and manage every application and interaction to protect their entire infrastructure. This can only be done if security is as distributed as the assets and applications themselves and based on a higher-level understanding of access control and authorization.

The article describes how utilities can use a security fabric that touches every application, device and HMI in their electric distribution network, from core-to-edge, to protect the entire system from unauthorized access, rogue devices, malware and other threats. The protection provided by a security fabric is effective even if the attempted intrusion comes from within the security perimeter.

Electric utility application security requirements

Electric utility distribution networks are composed of devices and software applications that cooperate with each other and humans in real-time. With DERs and grid edge devices added, electric distribution utilities may have hundreds of thousands of intelligent electronic devices (IEDs). Just one compromised component may create the potential for contagion. Security from core-to-edge, despite inherent challenges, is an absolute necessity.

Assets and applications in a utility’s distribution network are deployed in the field, which may span thousands of square miles. Some assets lack innate security capabilities and reliable connectivity. In addition to securing devices and applications, communication between them must also be secured. Security from core-to-edge is an absolute necessity.

Core-to-edge security can be implemented using a security fabric to centralize security policy creation while decentralizing security policy enforcement throughout the distribution network. This is a departure from the traditional centralized patchwork of disparate security tools commonly deployed in utility distribution networks.

Key requirements for electric utility distribution network’s security fabric include:

• Authenticating and authorizing all applications, devices, and humans
• Securing access to all applications, devices, and HMIs
• Securing all interactions, whether core-to-edge or edge-to-edge, between all applications, devices, and humans
• Securing data both in transit as well as at rest at the edge and in the core

The following section describes a model security fabric architecture that forms the foundation for meeting these requirements.

Model architecture for electric utility application security

Decentralized operation is inherent to electric utility distribution networks. Substation computers, intelligent electronic devices (IEDs) and other assets are spread out through the utility’s service territory. Interactions between them are distributed as well. As a result, the architecture for electric utility distribution security enforcement must be distributed, too. The reasons include:

• Scale: Distribution systems can contain hundreds of thousands of IEDs and other distributed assets. Centralizing computation, communication, control and security for such large systems is cost-prohibi¬tive and simply not feasible.
• Many-to-many data exchange: Field operations require many-to-many data exchange to scale and to provide real-time reaction to changes. All these data exchanges must be secured.
• Access by multiple entities: Decentralized system operation enables cooperation between multiple entities, (e.g., an electric utility, its suppliers and its contractors). Decentralized security eases the task of ensuring that each participant accesses only permitted equipment and systems, enabling data and system owners to define their sharing and access policies.
• Decentralization avoids single-points-of-securi¬ty-failure: Field assets are, by nature, exposed. Distributing security data and enforcement avoids contagion in the event of a localized compromise.

The model architecture (see Figure 1) for a security fabric to protect electric utility distribution networks includes four main elements:

• A policy manager that sets security policy, and manages the security fabric
• Brokers that interface to corporate IT systems, e.g., Active Directory or certificate servers, and nodes (see below)
• Nodes that authenticate and authorize interactions in-field throughout the entire electric utility distribu¬tion network
• Enforcement points that provide filtering to protect IEDs and other assets that lack innate password or secure protocol protection

The role of each of these elements is explained in the following sections.

Policy manager

A policy manager enables human operators to administer the security fabric in a highly automated fashion. It is usually deployed in a data or operations center.

Using a policy manager, administrators can centrally set the policies to be applied by the security fabric. These policies are propagated throughout the system by automatically and accurately replicating security parameters across all devices, applications and humans in an electric utility distribution network. Administrators can create groups of devices, applications and humans, prescribing how these groups interact.

The policy manager also ensures compliance with emerging regulations and policies for critical infrastructure security including NERC-CIP. It automates the gathering, recording and preservation of critical historical information.

The policy manager does not sit in the security fabric’s data or control plane. If it fails or loses communication with the security fabric, the fabric will continue to operate fully, without interruption.

Brokers

Brokers provide an interface between nodes and corporate IT systems. They also propagate security policies set by the policy manager to nodes in the field. Brokers usually reside in control/operations centers. However, brokers may be installed in the field, if security services such as domain controllers are based in the operational network.

Brokers interface with security services such as LDAP, Active Directory, trusted certificate authorities (CAs), etc., but can also provide user and device directory services and host LDAP internally.

To provide maximum reliability for the security fabric, the brokers must enforce existing security policies if communication with the policy manager is interrupted.

Nodes

Nodes securely store identity and policy information, as well as some operational data, and authenticate and authorize interactions between applications, devices and humans. They are installed at selected locations throughout the electric utility’s distribution network and can run on ruggedized computers, commercially available industrial IoT gateways, communication network devices such as routers and switches –– and lightweight computing devices such as the Raspberry Pi 3. Because they consume few resources from their host systems, they can run on existing computing and communication networking equipment in the field and operate independently of central services (e.g. policy manager and brokers). A node should need less than 100 MB of memory and be able to operate on narrowband connections with as little as 10 kbps of bandwidth.

Enforcement points

Enforcement points (EPs) enable role-based access control and single sign-on to secure intelligent electronic devices (IEDs) and other assets that lack innate password or secure protocol protection.

Using an enforcement point to secure assets in the field gives administrators visibility into and control of all access attempts, whether successful or not. Enforcement points ensure that assets are isolated individually — if compromised — protecting the rest of the electric utility distribution and permitting other assets to continue operating normally.

Figure 1: Model Electric Utility Distribution Network Security Architecture
 

Information flows bi-directionally through the security fabric. Identity and policy information, e.g., authorized device and user lists and privileges, flows from brokers (at the system core) to nodes and optional nodes (at the system edge). Operational information, such as asset inventories and device profiles, flows in both directions. 

Services to secure electric utility applications

A security fabric provides numerous services to safeguard applications, devices and HMIs in an electric utility’s distribution networks. Many of the key services are described in this section.

Enforcing role-based access control (RBAC)

The security fabric enforces role-based access control (RBAC) based on the managed identity of the user, regardless of whether the device supports managed passwords, local passwords, or no passwords at all.

For devices with passwords, the security fabric enforces RBAC by acting as an application proxy, ensuring only trusted devices can interact with applications. The security fabric automatically sets and stores device passwords, which are complex, hidden and can be regularly rotated per policy defined by the broker.

The security fabric also supports inline filtering, which can enforce RBAC for all devices regardless of their ability to support passwords or other access control mechanisms. Only applications, devices and humans that are authorized to interact will be able to communicate; all others are flagged and reported. For humans, the security fabric grants only the access that policy allows, restricting access to authorized applications and devices in specified geographic areas.

This access control enforcement goes beyond what can be done at the network layer. Even if a hacker obtains network-level access, the fabric’s policies ensure they cannot access networked operational systems and cause them to malfunction or be hacked in turn.

Discovering applications, devices and humans

The fabric discovers and inventories devices, applications and humans when they request connection to the electric utility’s distribution network. Login attempts are monitored and discovery is conducted at the edge based on passive, network-based scanning and active scanning for selected protocols, such as DNP3, Modbus and IEC 61850. Discovery at the edge, ensures unauthorized assets are blocked from accessing any distribution network components.

Creating trust for applications, devices and humans

The security fabric creates trust by working with other systems including:

• Directory and authentication services such as Active Directory, LDAP and RADIUS
• Trusted certificate authorities
• Asset management systems such as those from SAP

For each application and device, the security fabric stores certificate information, if available, and a fingerprint, containing information such as hardware, software and firmware identifiers, typical memory use, configuration and register settings and filesystem hashes. Fingerprints enhance certificates and create trust for applications and devices that lack them. The fabric allows only applications and devices with a valid certificate (if available) and matching fingerprint to join the system.

The fabric rechecks fingerprints and passwords to detect compromised applications and devices. An unauthorized fingerprint change indicates the application or device has been tampered with, and the fabric isolates the compromised asset. If an old password is used, the security fabric reports the attempted intrusion.

Tracking changes, documenting compliance

The fabric automatically tracks changes in application and device inventories and profiles, creating audit logs, documenting compliance with evolving regulations and standards for critical infrastructure security, including NERC-CIP.

Safeguarding data privacy and integrity across multiple parties

The security fabric safeguards the authenticity, privacy and integrity of data shared by multiple parties, e.g., different suppliers and contractors, in an electric utility’s distribution network. The data’s originator or owner controls access policies for the data, which the fabric replicates and enforces across the multi-party operation.

Using a security fabric to protects DERs: Threats and solutions identified by major U.S. utility

When planning to integrate DERs into their electric distribution network, a leading U.S. utility identified specific threats they would need to mitigate. They also determined that a security fabric would be the best way to defend against these threats. Examples of the threats and solutions that were identified appear in this section.

Unauthorized access to DERs

Real-time automation controllers (RTACs) are used to collect information from devices in DER installations, store it and provide it to monitor and telemetry systems. They also execute logic based on the data they collect and send commands to the intelligent electronic devices (IEDs) at DER sites. RTACs use industrial protocols such as Modbus, DNP3 and IEC 61850.

Unfortunately, many of these protocols lack security. For example, a Modbus slave will respond to any valid write request from a Modbus master. This makes the Modbus slaves in the DERs vulnerable to unauthorized access attacks. For example, an unauthorized device or application could reset the set point for a power inverter to disrupt the power balance of the DER.

Using a security fabric, an enforcement point prevents Modbus commands from unauthorized devices from reaching the power inverter while allowing authentic traffic to proceed unhindered. (See Figure 2)

Additionally, via the policy manager, the enforcement point will notify the human operator of the attempted attack by the rogue device.
 

Figure 2: Security Fabric Prevents Unauthorized Commands from Reaching DERs
 
 

Isolating a malfunctioning or suspect DER

When a remote device in a DER sends incorrect data, perhaps because an attacker compromised it to send data at variance with the actual power output to unbalance the DER, it is difficult to exclude that device’s data from the data stream. It may require reprogramming the governing RTAC, a burdensome process.

A security fabric will remove the suspect device from the active data stream and notify the human operator that inaccurate data is being blocked. The operator can then check the device for malfunctioning or tampering. (See Figure 3)
 

Figure 3: Isolating a Malfunctioning or Suspect DER
 
 

Detecting DER tampering

Protocols such as Modbus do not uniquely identify devices. Many devices used in DERs do not have tamper alarms. An attacker could replace a Modbus slave with a faux device. The faux device can attack the system by sending seemingly correct data that does not accurately reflect the power output of the device or masks the fact that the device has been removed.

A security fabric can detect such an attack by constantly monitoring the communication link between the node or enforcement point and each device in a DER. When communication with a trusted device stops or is interrupted, the fabric alerts the human operator that the event should be investigated.

Summary

Securing electric distribution networks requires that utilities look beyond communication network security. They must protect and manage every application and interaction to protect their entire infrastructure. Otherwise, a hack on any device inside the electric distribution network can cause a contagion that compromises the entire utility.

A security fabric that touches every application and asset in a utility’s operational network, from core-to-edge, provides an essential foundation to secure interactions between applications, devices, humans and data. The protection provided by a security fabric is effective even if the attempted intrusion comes from within the security perimeter. By providing policy enforcement plus logging, the security fabric can also automatically manage and document compliance with emerging cybersecurity regulations and standards for electric utilities.

Advancing beyond traditional security models, the proposed security fabric distributes authentication and policy enforcement across the network of devices, to ensure security at scale. The security fabric supports many-to-many communication, secures access to even legacy utility systems that lack inherent security capabilities and underpins continuous operation even in the face of intermittent connectivity.