There is a big change on the horizon that will impact every utility in the United States, and ultimately worldwide. Telecommunications service providers (what we typically refer to as ‘the carriers’) are phasing out circuit-switched phone service in favor of more advanced IP/Ethernet-based offerings. This change is a natural evolution of carrier networks toward more efficient technologies, which has been underway for more than a decade. Carriers have no choice, but to migrate to the new networks and standards.
However, a substantial concern that this network evolution presents is that these are the legacy networks that have been used by the utilities for decades. They are used for managing electrical grids; transporting data from substations and other nodes on the grid; and supporting critical applications such as the high voltage circuit breakers that protect the grid (teleprotection), and Supervisory Control and Data Acquisition (SCADA). Unfortunately, evolving to a new network also means finding alternative ways to address performance requirements of these systems, which is not as simple as switching from one data service to another – modern IP/Ethernet technologies are fundamentally different from traditional circuit-switched networks.
Though replacing electrical equipment such as Remote Terminal Units (RTUs) will likely be unnecessary, much of a utility’s grid communications equipment will no longer be compatible with the new services and will need to be re-engineered and replaced. Replacing this equipment is expensive and highly disruptive to utility operations, so the migration needs to be managed carefully to avoid unnecessary cost, complexity and risk.
For decades, circuit-switched carrier networks have been providing a reliable, deterministic, and relatively low-cost way to manage the transmission and distribution grids. The shift by the carriers away from the analog and Time Division Multiplexing (TDM) technologies – which utilities have relied on to manage their transmission and distribution grids – to packet-based networks is a consequence of rapidly shrinking demand for traditional home telephone service. It is an organic, market-driven pressure on the carriers that is fueling this transition and it cannot be stopped.
The reality is that carriers are no longer generating sufficient revenue from their current circuit-switched networks to continue operating them and are continuing to lose customer interest in paying for them, despite the fact that utilities are depending on those networks for mission-critical applications. Instead, carriers are focused on building out their broadband and mobile services revenue and must modernize their networks accordingly.
Carriers have not been shy about their plans. AT&T, for instance, recently distributed a ‘withdrawal matrix’ that made clear where and when analog, TDM and Frame Relay services would no longer be available. They shared plans to end support for all non-Ethernet access channels (such as DS0, T1, T3, OC-3, OC-12 and OC-48), all non- Ethernet private lines and Ethernet private lines slower than 600 Mbps, as well as, existing teleconferencing services and toll-free features. Required time-frames for withdrawal are as short as 120 days, which means that utilities need to be making plans well before they receive a sunset notification. AT&T plans to complete the transition – across its entire network – no later than 2020.
As a complication, most utilities source a percentage of their TDM and analog data services through multiple carriers, all of which have different timelines and locations for the transition, so a thorough and far-reaching plan is required by the utility to insure that communications services to the substation are not disrupted by the disparity in scheduling by the different carriers.
Also, as the move toward IP/Ethernet is universal, it is similarly affecting the carriers’ ‘ecosystem’ of suppliers and is similarly causing huge shifts in the supply chain for TDM equipment. Equipment manufacturers have for the most part stopped developing TDM equipment, and instead are focused on delivering IP-based solutions. At some point not too far in the future, this equipment will not only be impossible to get, but it will become difficult to maintain.
Finding technicians who are familiar with the analog and TDM technologies in order to repair them is becoming a major challenge as technical training programs focus on newer technologies. As a result of the shortage of both equipment and maintenance personnel, older networks are breaking down with greater and greater frequency.
Utilities are already in a situation where they experience reduced service hours and longer wait times to schedule service, and in some cases are refused service altogether, particularly when asking for new TDM circuits or renewing existing leases. In addition, they are also seeing financial disincentives to continuing with TDM connections. As the operational costs to maintain these systems increases, so does the cost to lease these services. This means the costs to the utility will begin to rise dramatically as these services become less and less profitable for carriers to continue providing.
What’s next?
Utilities cannot continue to rely on the TDM and analog circuit technologies that they have historically depended upon. Unlike ‘Smart Grid’ which has largely been a voluntary evolution of the electrical grid, the sunset of TDM and analog circuits are not within the control of the utility, so doing nothing is no longer one of the options. Carriers have already started to send notices that service will be discontinued in certain service areas.
As Connie Durcsak, president and CEO of the Utilities Telecom Council (UTC) details the evolution, “‘Flying like confetti’ is how one utility described their TDM and analog circuits termination notices from their commercial carrier providers. With carriers ending decades of support to utilities through the phase-out of TDM and analog circuits, utilities are left scrambling to identify replacement solutions. Given the vast number of circuits at risk, this task is not insignificant. It will be a costly and resource intensive exercise. And any replacement solutions will need to be identified quickly to assure continuity of service of SCADA, protective relaying, and other mission-critical applications. Certainly, wireless technologies will be a solution of choice for many remote areas or in cases where rights-of-way are challenging. However, the lack of available licensed spectrum prioritized for utilities and other critical infrastructure providers is a real concern. UTC is working with utilities and their technology partners to ensure that utilities have the information they need to weigh their options fully and to ensure that policy makers understand the impact of this situation on the nation’s energy and water resources.”
While describing the termination notices as ‘flying like confetti’ might be a bit dramatic, it sends a very clear message that the tipping point on these services for the carriers has been reached, the transition process is underway, and this process is certain to accelerate quickly.
The reliability of electric service is at risk and as a result utilities need to develop a transition plan, ideally one that does not involve a dramatic increase in costs. Unfortunately, upgrading to higher bandwidth, IP-based solutions offered by carriers will almost certainly result in an increase in monthly operations and maintenance costs.
Why? The options offered by carriers tend to be much more expensive, such as fiber access (where it’s available), IP over copper, or wireless cellular broadband (which is relatively unproven for mission-critical services). The fixed network service options, fiber and copper-based, typically cost anywhere from 4 to 10 times what existing TDM-based circuits cost. Moreover, they provide far more capacity, at much greater cost, than what is actually needed by most current utility applications. Worse yet, utilities may be required to pay substantial amounts to the carrier to install fiber or upgrade the copper infrastructure serving their substations and other locations.
For many utilities, the alternative offer by their carrier is likely to be a dedicated IP/Ethernet link that could cost up to $1,500 a month, in comparison to the $100 to $300 a month charge for their existing circuit connections. Spread this cost delta across a large service area and the increase in operations expense becomes fairly dramatic.
Of course, lower cost wireless solutions are being offered and these will certainly meet many utility data communications requirements. They are not, however, as predictable nor as reliable as the traditional services they are intended to replace. However, they provide service to delay tolerant and low-value utility assets. That said, consider the following scenario:
Hypothetical Transmission & Distribution Utility, ‘Universal Electric Service Group’ depends on circuit-switched DS0s and sub-rate Frame Relay connectivity to monitor the Dynamic Line Rating (DLR) sensors that are measuring the heat load and sag of a high-voltage transmission line. Recently, the carrier serving Universal gave them a 120-day notice of service sunset and offered either a multi-megabit bandwidth fiber-based Ethernet service or a low-cost cellular service as the replacement options. Without the Public Utility Commission’s approval for an increase to the rate base, the utility could not afford, and did not have sufficient time, to extend their private fiber and microwave network infrastructure, nor was it able to afford to implement a reliable fixed broadband digital connection from the carrier, so Universal scrambled to replace the service with the carrier’s cellular data service, having no other cost-effective means to remotely monitor the line.
A few months later, a localized power outage due to a thunderstorm caused a loss of power to the carrier’s cellular tower serving the DLR sensor. After 4 hours the batteries failed in the cellular tower and the neighboring cell towers became congested with consumer traffic as consumer phones switched over to the other nearby towers that still had power. This caused the transmission line’s DLR sensor to be unable to get a cellular connection which left the line unmonitored by Universal’s transmission management system for an extended period of time. The transmission line then overheated due to increased electrical load from neighboring areas as residents returned home from work, plugged in their Hybrid Electric Vehicles, and turned on air conditioners and pool pumps in areas not affected by the local outage. This caused the line to sag into an untrimmed tree and short to ground. This in turn tripped the differential protection breakers at the nearest two substations, creating a surge of electrical load on neighboring transmission lines which in turn tripped their breakers from the additional load. Finally, and in a matter of just a few minutes, it ultimately cascaded across the grid causing wide area outages including significant areas in several neighboring states.
Sound implausible? Although the root cause is a little different (communications failure vs. lack of line sensor), this scenario has played out before, in 2003, when a failure blacked out large sections of the bulk electric grid from Ohio to New York and into Canada.
It is possible, of course, for utilities to establish their own private TDM network to support applications such as SCADA, teleprotection, security, substation voice and others; although, there will be fewer and fewer vendors available to support these networks as suppliers naturally move to next-generation technologies. The supply chain is rapidly shrinking, not growing.
In contrast, certain packet-based technologies, such as IP/MPLS and Carrier Ethernet are well suited to the requirements of utilities in utility-owned networks. In a private network designed and implemented as ‘Utility-Grade’ these technologies can be deployed to meet the needs of the utility well into the future. In fact, these are fundamentally the same technologies that the carriers themselves are using to provide alternative service options – just without the strict traffic engineering controls on the quality of service, deterministic (consistent) path and delay that is required for many SCADA applications and any teleprotection service that the utility would put in place for itself to meet its own requirements. The carriers design systems that meet the requirements for the bulk of their customers, but not necessarily all of them, and utilities are outliers when it comes to network communications requirements.
As a result, now is the time when utilities need to consider whether it is most cost effective to continue to rely on carriers to address their critical operational needs and at what percentage, or whether it might be more appropriate to consider extending and transitioning their own privately-owned telecommunications farther out into the field, thereby taking control of both their costs and evolution plans.
So, how can utilities best ensure that they can continue to meet the expectations of their customers? Naturally it depends on the particular situation of the utility, but generally speaking, the business case for building and maintaining a private network becomes more compelling as the cost of leased services increases. Considering the growth in operations and maintenance costs in coming years, building a single, utility-owned network, or even extending that portion of the utility’s existing network further out into the field may well become an increasingly attractive option in terms of reducing overall costs and ensuring continuity of operations.
As importantly, it’s not entirely clear that carriers will be able or willing to address the utilities’ stringent latency and deterministic requirements, particularly since utility applications are a small minority in comparison to average business applications, which is the bulk of their business. Ultimately, however you slice it, unless they have a viable, reliable, long-term alternative to TDM and analog circuits, utilities will not be able to reliably and efficiently supply power to all of their customers.
There are a variety of ways to approach this end goal, but all involve utilities moving sooner rather than later, and coming up with a proactive plan before their hand is forced. The future is coming and it is coming quickly. Utilities need to come up with a plan to transition from their reliance on carrier-provided TDM services, equally quickly. They have the opportunity to control their own destiny, but time is of the essence, and they need to move now.
About the Author
Mark Madden is the Regional Vice President, North American Utilities, Alcatel-Lucent. He has over 25 years of experience with leading companies in the Information Technology and Telecom industries, and has diverse business, and technical expertise gained through a variety of positions in the industry. Mark is currently responsible for Market Strategy, Strategic Partnerships. and Business Development in Alcatel-Lucent’s Americas Region Energy Markets.
Mark is an accomplished Network Architect with a strong background in Network Capacity and Application Performance Management, as well as, Network Management Systems and Program/Project Management. He has been responsible for the formation and establishment of Business Consulting, Network Application Performance services, and Application Integration Services at Alcatel-Lucent. Mark has also consulted to a wide variety of companies on Network Performance Management, Network Architectures, Network Management Systems and Processes, and Communications Application Development.
