John McCain and Russ Straayer

About Newnan Utilities
Since 1904, Newnan Utilities has been serving the residents of Coweta County, Georgia, about 40 miles southwest of Atlanta, Georgia. Newnan Utilities has been a technological leader in providing community utility services, greatly influencing cable and utility companies across the state of Georgia and beyond. Newnan Utilities provides electric power distribution, water, waste water services, cable television and Internet services to the County. Entering its second century of operation, Newnan Utilities continues to be a leader as a local utility provider.

SCADA, Internet and Security are Combined
A 2007 innovation by Newnan Utilities is the use of the cable television Internet and fiber optic Internet system for SCADA traffic. SCADA over the local Internet delivery system uses in-house assets, uses the Internet, and uses encryption. It is encryption that makes this mode of SCADA transport possible. Since SCADA data traffic controls electrical sub stations, relays and reclosers, Newnan Utilities needs to take every precaution to insure that the SCADA traffic is secure.

New System Drives Communications Needs
Newnan Utilities recently installed an upgraded Advanced Control Systems (ACS) Linux master SCADA system. The communications interface runs both serial DNP 3.0, new to the utility, and ACS-7000 protocol. Newnan Utilities needs to communicate with several DNP 3.0 protocol serial devices back to the SCADA master controller. The possibility of using DNP 3.0 TCP/IP was investigated, but determined not to be cost-effective for this project, due to high equipment costs for new hardware and software. The existing SCADA system and RTUs would have required major upgrades or complete replacement to become TCP/IP capable. Some of the RTUs in service are models that cannot be upgraded to TCP/IP. As long as the older RTUs are in service, they must communicate through serial ports. With the cost to convert the entire SCADA system to TCP/IP being prohibitive it was natural to look for other ways to best transport the serial SCADA data.

Reviewing the Alternatives
A number of options for transporting serial data were considered. Newnan Utilities owns a facilities based fiber infrastructure that provides SONET transport. However, the last-mile connectivity to all monitoring stations is not economically available via SONET. Newnan Utilities sought an economical and dependable communications solution to provide last-mile transport to their fiber network. With the existing cable network passing needed monitoring points, an encrypted solution for utilizing the cable network for last-mile connectivity was desired.

Some of the fiber in place at Newnan Utilities is used for SCADA on a point-to-point basis using fiber modems. During the initial fiber network build, all of the electrical substations were connected to the fiber network. As the system has grown, automatic circuit reclosers have been added to speed up restoration of electrical service and provide efficient switching operations. In most cases, the cable modem network passes near the reclosers, while the fiber network does not. These reclosers are connected to the SCADA network over the cable modem Internet service. With proper encryption, utilizing the existing cable network minimizes construction costs and expedites installation time.

Being a cable company and an Internet Service Provider (ISP) as well as an electric utility, provided an economic choice... take advantage of the installed communications backbone and use cable modems or the fiber dual-ring SONET network for the IP communications. SCADA for the electric utility includes not just reporting data, but is also used for control. It is imperative to design an electrical utility control system to be as secure as reasonably possible. To address security concerns, VPN tunneling and other methods of encryption were considered. The fiber network is TCP/IP, while the SCADA host ports and the RTUs are asynchronous serial. This requires a device to transport the serial SCADA polling information over the TCP/IP network.

Fostering Inter-Department Cooperation
SCADA is critical twenty-four hours per day, seven days per week. At Newnan Utilities the cable TV system and the fiber network support residential and business services that must also be operational 24 hours a day, seven days a week. Newnan Utilities has an always-on approach to maintaining the TCP\IP network simply because the cable modems and fiber are provided as a service for 12,000 customers. The provision of Internet service must be a 24 by 7 service. The priority to maintain the Internet network matches the urgency of maintaining an operational SCADA system. It is a perfect match, with everyone involved understanding the need and being committed to 24 by 7 operation.

It is also noteworthy that the cable and fiber Internet service and the electric utility both depend upon each other for reliable operation. The cable and fiber, used to deliver cable television and Internet service needs constant reliable power. The electric service provided by Newnan Utilities is expected to be available all the time. To do this, Newnan Utilities needs the cable and fiber service to be working 24 by 7. This is truly a symbiotic relationship, each dependent on the other. The two different departments have the same goals and find that this provides for a great working relationship.

Encryption is Recommended
Mary Hester of Intelligent System Solutions (ISS) worked with Newnan Utilities as a consultant on the SCADA system and the communications solution. Based on her industry experience and research, Mary recommended a serial data over IP solution that included encryption. The Federal Energy Regulatory Commission (FERC) is developing Critical Infrastructure Protection (CIP) standards. While encryption is not specifically required at this time, the encryption recommended by Mary Hester and selected by Newnan Utilities is based on the federal AES encryption standard. Should FERC mandate encryption, the mandate will likely have standards based on the Federal Information Processing Standards (FIPS). The AES encryption selected by Newnan Utilities has been a FIPS standard since November of 2001.

The solution, recommended by Mary Hester and selected by Newnan Utilities, is a Data Comm for Business (DCB) Encrypted EtherPoll (EEP) network. The EEP uses IP and ethernet to transport serial SCADA protocol data. In addition, it is specifically designed to broadcast polling data to multiple end points, either on a single ethernet segment or through a routed network. The EEP encrypts the user’s serial data using the AES encryption standard algorithms. By using this method, the data is encrypted while being transported. The network that transports the SCADA information also carries other Internet traffic, making encryption an important factor.

Easy Installation
As a first stage, the communications links were run over the cable modem Internet network. The installation was quickly up and running. George Lee of Newnan Utilities says he was shocked at just how easy it was. There were absolutely no problems he says. Early on in the installation period, George Lee, Carol Parks and Mary Hester of ISS concluded that to make managing and troubleshooting communications easier, some changes to the Encrypted EtherPolls’ indicators would be helpful. They convinced the manufacturer of the value of these changes, and soon they had new firmware for the units. George and Carol downloaded the new firmware to provide more useful indicator lights. It was a quick and easy firmware upgrade. George says he finds the equipment extremely easy to use and configure.

Eventually the electrical utility communications system was moved from the cable modems to the fiber dual-ring SONET network. The fiber network features redundancy and higher bandwidth, features that result in a more robust, reliable communications infrastructure. In fact when one compares cable modems on a cable TV network to a dual-ring SONET network, the the phrase “more robust” is an understatement.

Cable modems are located along the television coaxial cable. Coax bandwidth is very limited compared to fiber. If a cable is cut between the cable modem and the cable system head end, the communications link is severed. A dual-ring SONET system is fault tolerant and self-healing. If a node fails or a cable is cut, the equipment senses the failure and routes the data in the opposite direction around the ring. This failure sensing and re-routing provides the self-healing function. At Newnan Utilities, they refer to the business fiber network as the “premier service”. The SCADA network is only running at 9600 bps, but the high speed of the fiber insures that the Internet traffic does not impact the SCADA communications.

Offsite Backup will be easy
Newnan Utilities does not currently use a second host location for their backup SCADA system. The backup system is currently co-located with the primary SCADA host system. In the event of a primary system failure, the communications lines are switched to the backup system and it becomes primary. The serial to ethernet communications equipment selected by Newnan Utilities will make it very easy to locate the backup system at a different physical location. Using serial communications over the IP network makes it very easy to establish redundant communications to a backup system. In fact, there is no configuration or other change needed to activate the backup equipment. The serial-to-IP EtherPoll devices at the RTUs are configured with 2 IP addresses for responding to the polls. One reply always goes to the primary site, a duplicate reply goes to the backup site. There is no start up protocol needed when switching to the backup host. Both primary and backup always get the RTU data.

Extending SCADA to Waste Water
“Later this year the SCADA network will be extended to monitor waste water.” In this case, it is likely that cable modems will be used. Cable modems are not as reliable as the fiber network, but the cable infrastructure reaches all the lift stations. The lift stations are not as time critical as the RTUs only report status and telemetry, with no control information being sent back to the lift stations. Consequently, for this network expansion, DCB’s EtherPoll devices that are not encrypted will be used for a slightly more cost effective system.

Conclusion
Newnan Utilities is more than satisfied with their communications choices. The equipment is easy to set up, easy to maintain. The encryption protects the infrastructure from cyber attacks. Best of all, the communications system they selected is the most cost effective of all the choices available.

About the Authors
John McCain is the LAN/WAN product manager at Data Comm for Business, Inc. He is a graduate of Louisiana Tech University and a Registered Professional Engineer whose past titles include Manager of IS Technical Services for a FORTUNE 500 company, Vice President of Development at NSE Inc., Instructor of Data Processing at a community college, EDP Systems Engineer at a Fortune 500 manufacturer, Systems Engineer and Project Engineer at an electric utility, and Consulting Engineer. His articles have been published in several technical magazines and he has authored numerous tutorials and white papers for distribution over the Internet..

Russell Straayer is President of Data Comm for Business, Inc. (DCB), a position he’s held since 1981. He has a BS degree in communications from the University of Illinois. His experience includes telecom management with the State of Illinois, Vice President of Compre Comm, Inc., and he has taught technical classes in datacom, telecom, and local area networking. His papers have been presented at numerous technical conferences in the datacom, energy, SCADA, and traffic control fields and he has been published in numerous trade journals. He has consulted for GTE, AT&T, Mobil and many FORTUNE 100 companies and government agencies.