The Challenges

TEP needed to expand its network capacity at its Tucson data center backbone due to a large increase in business application data used to maintain high levels of customer service. It needed to easily manage all the internal business application traffic on the network including e-mail, billing, customer-records, metering data, financial reporting and even large aerial photos of transmission lines. The company also wanted to link four facilities across a large metropolitan area, and add a separate storage network between two major facilities.

TEP was an early adopter of virtualization, with over 90% of their IT infrastructure virtualized. This has a significant effect on storage and backup requirements, increasing by terabytes of data per month and this data had to be replicated across the network. The company clearly understood the need to backup and store data after a major flooding incident in its service center. After considering its needs, TEP quickly realized they exceeded its 1Gb passive optical network, and must upgrade to an active 10Gb network.

Figure 1: The upgrade yielded dramatic performance increases and operational improvements.
 

TEP owned a dedicated dark fiber network, which ran between its four facilities across the same infrastructure it owned and managed for power distribution. Having the fiber was a considerable benefit, The company required a robust carrier-grade telecommunications solution that it could manage without incurring large ongoing operational costs. From their research, they believed a Wavelength Division Multiplexing (WDM) solution would meet their needs.

Wavelength Division Multiplexing (WDM)

In an optical network, information is converted to series of light pulses, which are transported along optical fibers and retrieved at a remote location. Theoretically, any light source could act as the information transmitter, but to achieve the distinct shapes of pulses needed for high speed data transfer and to restrict the light to a particular wavelength, only lasers are used in telecommunications systems.

Figure 2: A simple optical link using a single fiber and having regenerators at intermediate sites.
 

The need for condensing, or multiplexing information on a single link arises because for most applications it costs less to transmit data at a high bitrate (e.g. Gbit/s) over a single fiber than it is to transmit it at lower rates (e.g., Mbit/s) over multiple fibers. There are two fundamentally different ways of multiplexing the lower bitrates onto a single fiber – time division multiplexing (TDM) and wavelength division multiplexing (WDM).

With TDM, the lower speed input channels are each allocated a defined timeslot on the outgoing higher speed channel – physically they are “taking turns” on the outgoing fiber. Time division multiplexing has been used in telecoms since the 1970s and was also the first technique employed in optical networks. Standards like the Synchronous Digital Hierarchy (SDH), with data rates denoted STM-1, STM-16 etc. and SONET with optical data rates denoted OC-1, OC-64, etc. are typical examples of TDM systems used in optical networks.

When using WDM, each input channel is assigned a unique wavelength (i.e. color of light), thus the channels can traverse the fiber “in parallel”. This technique enables multiplication of the capacity, but also bidirectional communication over one single fiber – a fact of significant importance when fiber is scarce or expensive to lease.

Figure 3: Wavelength Division Multiplexing (WDM)
 

Multiplexing of the wavelengths in a WDM system can be done in two different ways, depending on how much bandwidth each wavelength is allocated on the optical fiber: Coarse (CWDM) and Dense (DWDM).

Typically, CWDM systems provide up to 8 channels (i.e. wavelengths) in the 1450 to 1600 nm range. Some CWDM systems can allow for 8 additional CWDM channels in the 1270 to 1450 nm range, fully in accordance with the ITU-T Recommendation G 694.2 for a total maximum of 16 CWDM channels.

DWDM systems use a smaller transmission window than CWDM focusing on the 1530 to 1565 nm range and use much denser channel spacing. DWDM channel plans vary, but a typical system would use 40 channels at 100 GHz spacing or 80 channels with 50 GHz spacing. New amplification options, such as EDFA amplification, enable the extension of the usable wavelengths, greatly increasing the distance WDM systems can span.

DWDM systems have to maintain more stable wavelengths/frequencies than those needed for CWDM because of the closer spacing of the wavelengths. Precision temperature control of the laser transmitter is required in DWDM systems to prevent “drift” off a very narrow frequency window of the order of a few GHz. This higher performance results in DWDM systems typically being more expensive than CWDM, although metro focused DWDM solutions can provide DWDM functionality for very close to CWDM pricing.

Some DWDM systems support transport using pluggable and software-tunable transceiver modules capable of operating on 40 or 80 channels. This dramatically reduces the need for discrete spare pluggable modules, when a handful of pluggable devices can handle the full range of wavelengths.

The actual choice of multiplexing technology – CWDM or DWDM – depends on multiple factors such as:

• Distance to bridge
• The number of channels needed (Channel count) now and in the future – DWDM gives 40 or 80 channels, while CWDM is restricted to 8 or 16
• The data rate used per channel
• The number of fibers available

Generally, CWDM solutions give the lowest entry costs for metro/access networks while DWDM is more cost-effective in metro/regional networks due to the higher number of channels and longer distances. In fiber-scarce areas DWDM solutions can also be a suitable option if more than 4 channels are required. Additionally a higher utilization of each wavelength could be a way to reduce the number of wavelengths required. This can be achieved with muxponders that perform electrical multiplexing of multiple client signals onto one wavelength. Finally, some vendors support a mix of CWDM and DWDM solutions on the same fiber to maximize fiber usage and scale a CWDM network to higher capacity.

The Tucson Electric Power Network

Initially, Tucson Electric Power believed that operating a DWDM solution over the network would be too expensive and complicated to deploy. TEP’s engineering team looked at a variety of solutions that would provide increased capacity, as well as the scalability to meet TEP’s five year plan. The solution TEP chose to deploy had to meet stringent cost criteria, and provide increased flexibility and scalability without adding on-going management and operational costs.

“We wanted an optimal way to introduce some DWDM equipment. We had small distances, high speeds and very large bandwidth requirements. This solution seemed to do exactly what we were looking for while anything else we could have chosen would have cost substantially more.”
– Jim Taylor, Director T&D Engineering

The primary decision was to invest in a networking solution that maximized the capacity of the company’s existing fiber infrastructure and not force them to buy bandwidth from a telecom operator. This decision was quickly reached on the basis of cost saving. After looking at four system vendors to understand the possible options and cost implications of specific optical networking solutions, TEP chose Transmode’s TM-Series optical networking platform because it best fit its needs for expanding performance and capacity.

Economics was an important factor in the vendor selection decision. The increase from a passive 1 Gb network to an active network with 4x 10Gb pipes and 4x 1Gb pipes was a major change for TEP. TEP also wanted a flexible network that increased functionality and left room for expansion. The network TEP installed consists of 4 nodes carrying Ethernet traffic.

The topology of the network means traffic can be picked up and dropped off at any of the nodes. Even with the nodes as little as 15km apart and carrying standard traffic this network architecture requires auto-power-balancing.

Figure 4: TEP has 4 sites, with multiple 10G and 1G connections
 

Auto-power Balancing

In DWDM networks with closely spaced channels, it is imperative to maintain equal power levels among the different channels. This is especially important in dynamically provisioned optical networks in which it is possible to add and drop multiple wavelengths. The TEP network running across four nodes with the two primary nodes at each end of the network and two intermediate nodes has been designed to carry traffic from the two end points and pick up and drop traffic at the intermediate sites.

In such cases, there can be a significant mismatch in the optical power levels and therefore in the optical signal-to-noise ratio level among the different channels. An irregular optical power level can also introduce non-linear effects and cross talk from high powered to low powered channels. Some vendor solutions eliminate these issues because of its built-in auto-power balancing feature, which is more common on carrier type platforms.

Network Management and Green Power

TEP was concerned that it would not have the optical experience or skills to effectively manage a DWDM network. However, since the deployment, TEP has not needed to add any extra staff or take existing staff away from other tasks. “We check on it to make sure there are no alarms, but in general it just works” said Jim Taylor. Finally, as a power utility company, TEP is proud of its green credentials and although power usage was not a major cost concern for the power company, the solutions low power hardware design did meet TEPs criteria for energy savings.

Results

The DWDM network solution has enabled TEP to transform its operation, greatly increasing capacity and capabilities with a resilient network. Furthermore, the network can be maintained by the same staff, with little trouble. For the IT department, the main benefit is that they can now work more efficiently.

“It’s like a quantum leap. Once you have this system you start finding all the things you can do you would not have considered before. We now have the ability to engineer our systems as you would in a single facility but now we can do it across facilities.” – Tyler Kilian, IT Supervisor

By deploying a networking solution that meets its current needs and is flexible enough to meet its future expansion plans, TEP has minimized its investment in capital expenditure and on-going operational costs. It has a network that is reliable, cost efficient and easy to use. TEP can now add and delete channels easily with great flexibility. The existing team can manage the network without adding expensive headcount.

About the Authors

Jim Taylor is Director of T&D Engineering and also also acts as TEP’s technical lead for Smart Grid initiatives. During his ten years at TEP, Jim has worked as a Senior Substation Engineer, Metering Services Supervisor, Protection, and Communications, Automation and Metering Engineering Supervisor.

Tyler Kilian is IT Supervisor at Tuscon Electric Power, where he is responsible for the multi-disciplinary team that manages and engineers company Local, Wide-Area, and Metro-Optical Ethernet networks as well as geographically dispersed Voice systems, including a separate ACD/IVR platform. Tyler also manages multiple Data Center environments for corporate and operational equipment. and the company’s regulated Energy Management System (EMS) network.

Sten Nordell is Chief Technology Officer, Transmode

Sten Nordell is Transmode’s CTO and drives Transmode’s strategic technology roadmap to deliver the next generation of optical IP/Ethernet systems. As one of the few people in the industry to have senior-level experience in building networks from the operator’s perspective and in designing systems as a vendor to meet operator needs, Sten is highly regarded by his peers for his regular presentations at numerous conferences and shows worldwide.