One MAS radio system, multiple channels required
The Rural Electric Convenience Co-Operative (RECC), with headquarters in Auburn, Illinois, is migrating its SCADA system from ILEX proprietary protocol RTUs to open protocol DNP3 RTUs. Migrating a SCADA system from one protocol to another can be very time consuming and expensive.
To keep the SCADA system operating on a non-stop basis, a migration normally requires all the new SCADA units with their new protocol to be deployed into the field at all the locations, installed, tested, and as the final step, all turned up at the same time. ”This is bad economics” says Tom Jones, SCADA Systems Manager at RECC.
For RECC, replacing and turning up all the new RTUs at the same time would mean sending personnel into the field to disconnect the old protocol equipment and then connect to the new equipment within minutes in order to keep the SCADA reporting real time, not to mention the cost of replacing all of this hardware at the same time. RECC is a relatively small Electrical Co-Operative, but they have 9 remote SCADA sites. They do not have the personnel to perform the migration quickly.
The Multiplexing solution
RECC chose instead to convert their single channel radio system into a multiple channel radio system. RECC uses Microwave Data Systems model 9710, 900 MHz licensed digital radio operating at 4800 bps. Adding multiple channels is accomplished by using the 4-port SCADA Multi-Drop Multiplexer (SMD Mux) from Data Comm for Business, Inc.
Going to multiple channels over the radio system enables RECC to convert one location at a time. A four port SMD Mux is deployed at the host site, another SMD is deployed at each remote site. With four channels, RECC operates the original ILEX protocol on the first channel. The new DNP3 protocol RTUs are deployed on the second channel. The deployment time for adding the SMD to the communications system is a fraction of the time necessary to replace all of the RTUs, and will not have to be repeated in the future when there is another change to the system.
The SMD allows simultaneous, independent operation on all four ports. The channels do not interfere with each other. The SMD divides the MAS system channel into 4 ports by digitally time-dividing the radio bandwidth. One might assume that this process will greatly slow down the collection of SCADA information, but it does not. The SMD divides up the data channel without noticeably slowing down the SCADA system polling.
A typical SCADA system polls at a rate much below the capability of the MAS radios. In the case of RECC, the ILEX RTUs were polled at a rate of 1 or 2 polls per second. The SCADA system does not need to poll as fast as possible to keep informed about the status at the substations. There was a lot of excess bandwidth available. Now with the SMD installed at the RECC host site, the polling rate is 25 times per second, far faster than the ILEX polling requires. This faster polling by the SMD results in a four to five time increase in the data transmission capacity of the MAS radios.
The SMD at RECC also performs data rate conversion. Data rate conversion is used to further speed up the polling process. The RECC MAS radios are 4800 bps, but the RTUs and the host computer operate at 9600 bps. The speed conversion increases the polling rate of each channel and is transparent to both the radios and the RTUs. The SMD momentarily buffer up the excess data. The RTU interface rate and the host computer interface rate were changed from 4800 to 9600 bps when the SMDs were installed. Comparing 4800 bps to 9600 bps, a small 10 character poll or response takes about 20 milliseconds to be transmitted or received by a serial port at 4800 bps, while the same 10 characters take only 10 milliseconds at 9600 bps.
Adding an AMR channel for free
The RECC SMDs are four port units, with the ILEX and DNP3 operating on 2 of the four ports. RECC uses the third port for Automatic Meter Reading. The AMR system at RECC is the DCSI TWACS AMR system. The AMR system is a polling system operating at 9600 bps. Adding the AMR onto the existing MAS radio system results in huge cost savings for the utility. To implement AMR without the SMD, RECC would have to install a duplicate, parallel communications system.
Another radio system would cost in the range of $30,000, installed. This is 3 times the cost of the SMD. If RECC decided that a licensed radio system was necessary, they would incur the additional cost of licensing. At 900 MHz, it is often difficult to find an available frequency. In addition, radio system maintenance often involves climbing a radio tower to check cables and align antennas. The SMD is serviced at ground level and does not require periodic preventative maintenance.
Access Switch function
With only 3 of the 4 ports used for polling communications, one more port is available on the SMD. Any SMD port can be a type called an Access Port. When an SMD port is set to the Access Port function, a point-to-point link is established from the host to any one of the drops. RECC has 9 remote sites. At each site is an RTU with an RS232 setup port, and each radio has a management port. When the system was initially installed, the MAS radio management port was attached to the fourth port of the remote SMD. The installer, from the host site, was able to remotely control any one of the 9 remote radios, one at a time, to check settings, signal strength, etc. Now the setup ports of the RTUs are being cabled into the extra port of the SMD. RECC will be able changes RTU options from the host site, rather than travelling out to a sub-station.
Other devices may be plugged into the ports of the SMD. Non-polled meters for example. The non-polled devices can be used on any of the 4 ports of the SMD. RECC can attach a terminal or PC to the host end SMD, then select a specific remote port to communicate with. By selecting a specific remote port, non-addressed devices can be managed while other ports are carrying polling traffic.
Ethernet/LAN/IP in the future?
RECC will move all the ILEX SCADA RTUs over to DNP3 over the next year, which will free up a port on the SMD. Once the ILEX to DNP3 conversion is complete, RECC will be able to add Ethernet bridging to the system if it is required. There are many new devices coming onto the market for electric utilities that utilize Ethernet ports, in lieu of, or in addition to serial ports. These devices, whether SCADA, AMR or other data collection and monitoring equipment, are typically very low data volume. These low data volume devices are an ideal fit with the SMD multiplexers.
Adding Ethernet is accomplished using an external box, a “SCADA Bridge”. The SCADA Bridge connects to the SMD through a serial port. The SCADA Bridge Ethernet port connects to the LAN equipment at the host and remote ends. Bridging (versus routing) involves just plugging in the units after setting the serial port speeds, a process that avoids the more complex setup used by routers.
How the multiplexed system performed
RECC did some minor tweaking of the system. Every few weeks RECC experienced the loss of a drop or two. This turned out to be a very subtle timing problem. Data was buffered at a drop for just enough time, a couple of seconds at most, to cause the SCADA host to timeout a poll, even though the RTU response eventually reached the host. With a number of older protocols that do not have numbered blocks or source addresses on the responses from the RTUs, delayed RTU responses coming to the host after the timeout period confuse the host polling software. RECC’s temporary fix for this problem was to reset an SMD, an event that occurred an average of once a week.
The problem was solved with updated SMD firmware. Port buffer sizes were changed and buffer timer was added. The timer is used to insure that data will not be buffered in the remote SMD so long that the host computer has already timed out the drop response. Now, with the updated firmware, if the RTU response sits in the SMD buffer too long, the data is discarded by the SMD, allowing the host computer to do its normal timeout and re-poll without getting an extra, delayed response.
The delays in RTU responses may come from heavy traffic, but the most typical cause of delays are errors on the communications links. The SMD must recover from errors on the radio link, just as a SCADA host must recover when an RTU fails to respond. While taking the time to recover, the SMD is not sending user data across the link. This is the time when the data was buffered at the drop and came into the host so late that the SCADA host assumed the data was lost and polled for the data again.
How the SMD improves the radio system throughput
The SCADA and ARM systems each poll at a rate of two times a second. They poll fast enough to keep the system control information up to date. The SMD does its polling at a rate of about 30 polls every second. Since the SMD polls so much faster than the SCADA and AMR systems, the SMD fills in the idle time gaps when the SCADA and AMR systems are not sending or receiving data. The SCADA system and the AMR system each use the MAS radio system at about 15 to 20% of capacity. The SMD fills in the time gaps, greatly increasing the efficiency of the communications channels.
Polling and response times will vary as the polling sequence of the SMD and the host computers go in and out of sync with each other. The SMD shares the network bandwidth by delivering data from all ports of the host SMD to the remote SMDs. At the same time that the host SMD is delivering data to the remote SMDs, it is also retrieving data from the remote SMD ports. The SCADA system and AMR systems are also polling systems, each operating on its own polling schedule. The SMD, SCADA and AMR polling rates are not in sync with each other. Consequently, the response times for the SCADA and AMR systems will vary. In the case of RECC, the average response time is about 250 milliseconds, with response times varying from about 100 to 400 milliseconds. This is typical for an MAS radio system application.
Conclusion
The bottom line for RECC has been the substantial cost savings. RECC has added 3 channels to their single channel radio system. Considering the alternatives of a parallel radio system or phone lines, the SMD multiplexer approach has saved months in time and at least $20,000 in new hardware and installation costs. A few small glitches had to be worked through, but all of the technical issues were solved with little or no impact on the SCADA and AMR systems. The impact was less than adding a new communications system for AMR and changing out all the RTUs from ILEX to DNP3.
About the Author
Mr. Straayer is President of Data Comm for Business, Inc., a position he has held since founding the company in 1981. Prior to that, Mr. Straayer was Vice President of Compre Comm, Inc, from 1977 until 1981.
Mr. Straayer is a graduate of the University of Illinois Springfield with a degree in Communications. Mr. Straayer has consulted for AT&T, Harris Bank and Trust, General Telephone and other major companies. He has been an instructor in data communications courses for the Federal Reserve, EDS/GM and for many public data communications courses.
Mr. Straayer was a telecommunications manager with the State of Illinois where he was responsible, in 1977, for a $35,000,000 budget. He was responsible for the 1977 implementation of a credit card calling system that included voice recognition equipment to automate the placing of credit card calls.
The Rural Electric Convenience Co-Operative (RECC), with headquarters in Auburn, Illinois, is migrating its SCADA system from ILEX proprietary protocol RTUs to open protocol DNP3 RTUs. Migrating a SCADA system from one protocol to another can be very time consuming and expensive.
To keep the SCADA system operating on a non-stop basis, a migration normally requires all the new SCADA units with their new protocol to be deployed into the field at all the locations, installed, tested, and as the final step, all turned up at the same time. ”This is bad economics” says Tom Jones, SCADA Systems Manager at RECC.
For RECC, replacing and turning up all the new RTUs at the same time would mean sending personnel into the field to disconnect the old protocol equipment and then connect to the new equipment within minutes in order to keep the SCADA reporting real time, not to mention the cost of replacing all of this hardware at the same time. RECC is a relatively small Electrical Co-Operative, but they have 9 remote SCADA sites. They do not have the personnel to perform the migration quickly.
The Multiplexing solution
RECC chose instead to convert their single channel radio system into a multiple channel radio system. RECC uses Microwave Data Systems model 9710, 900 MHz licensed digital radio operating at 4800 bps. Adding multiple channels is accomplished by using the 4-port SCADA Multi-Drop Multiplexer (SMD Mux) from Data Comm for Business, Inc.
Going to multiple channels over the radio system enables RECC to convert one location at a time. A four port SMD Mux is deployed at the host site, another SMD is deployed at each remote site. With four channels, RECC operates the original ILEX protocol on the first channel. The new DNP3 protocol RTUs are deployed on the second channel. The deployment time for adding the SMD to the communications system is a fraction of the time necessary to replace all of the RTUs, and will not have to be repeated in the future when there is another change to the system.
The SMD allows simultaneous, independent operation on all four ports. The channels do not interfere with each other. The SMD divides the MAS system channel into 4 ports by digitally time-dividing the radio bandwidth. One might assume that this process will greatly slow down the collection of SCADA information, but it does not. The SMD divides up the data channel without noticeably slowing down the SCADA system polling.
A typical SCADA system polls at a rate much below the capability of the MAS radios. In the case of RECC, the ILEX RTUs were polled at a rate of 1 or 2 polls per second. The SCADA system does not need to poll as fast as possible to keep informed about the status at the substations. There was a lot of excess bandwidth available. Now with the SMD installed at the RECC host site, the polling rate is 25 times per second, far faster than the ILEX polling requires. This faster polling by the SMD results in a four to five time increase in the data transmission capacity of the MAS radios.
The SMD at RECC also performs data rate conversion. Data rate conversion is used to further speed up the polling process. The RECC MAS radios are 4800 bps, but the RTUs and the host computer operate at 9600 bps. The speed conversion increases the polling rate of each channel and is transparent to both the radios and the RTUs. The SMD momentarily buffer up the excess data. The RTU interface rate and the host computer interface rate were changed from 4800 to 9600 bps when the SMDs were installed. Comparing 4800 bps to 9600 bps, a small 10 character poll or response takes about 20 milliseconds to be transmitted or received by a serial port at 4800 bps, while the same 10 characters take only 10 milliseconds at 9600 bps.
Adding an AMR channel for free
The RECC SMDs are four port units, with the ILEX and DNP3 operating on 2 of the four ports. RECC uses the third port for Automatic Meter Reading. The AMR system at RECC is the DCSI TWACS AMR system. The AMR system is a polling system operating at 9600 bps. Adding the AMR onto the existing MAS radio system results in huge cost savings for the utility. To implement AMR without the SMD, RECC would have to install a duplicate, parallel communications system.
Another radio system would cost in the range of $30,000, installed. This is 3 times the cost of the SMD. If RECC decided that a licensed radio system was necessary, they would incur the additional cost of licensing. At 900 MHz, it is often difficult to find an available frequency. In addition, radio system maintenance often involves climbing a radio tower to check cables and align antennas. The SMD is serviced at ground level and does not require periodic preventative maintenance.
Access Switch function
With only 3 of the 4 ports used for polling communications, one more port is available on the SMD. Any SMD port can be a type called an Access Port. When an SMD port is set to the Access Port function, a point-to-point link is established from the host to any one of the drops. RECC has 9 remote sites. At each site is an RTU with an RS232 setup port, and each radio has a management port. When the system was initially installed, the MAS radio management port was attached to the fourth port of the remote SMD. The installer, from the host site, was able to remotely control any one of the 9 remote radios, one at a time, to check settings, signal strength, etc. Now the setup ports of the RTUs are being cabled into the extra port of the SMD. RECC will be able changes RTU options from the host site, rather than travelling out to a sub-station.
Other devices may be plugged into the ports of the SMD. Non-polled meters for example. The non-polled devices can be used on any of the 4 ports of the SMD. RECC can attach a terminal or PC to the host end SMD, then select a specific remote port to communicate with. By selecting a specific remote port, non-addressed devices can be managed while other ports are carrying polling traffic.
Ethernet/LAN/IP in the future?
RECC will move all the ILEX SCADA RTUs over to DNP3 over the next year, which will free up a port on the SMD. Once the ILEX to DNP3 conversion is complete, RECC will be able to add Ethernet bridging to the system if it is required. There are many new devices coming onto the market for electric utilities that utilize Ethernet ports, in lieu of, or in addition to serial ports. These devices, whether SCADA, AMR or other data collection and monitoring equipment, are typically very low data volume. These low data volume devices are an ideal fit with the SMD multiplexers.
Adding Ethernet is accomplished using an external box, a “SCADA Bridge”. The SCADA Bridge connects to the SMD through a serial port. The SCADA Bridge Ethernet port connects to the LAN equipment at the host and remote ends. Bridging (versus routing) involves just plugging in the units after setting the serial port speeds, a process that avoids the more complex setup used by routers.
How the multiplexed system performed
RECC did some minor tweaking of the system. Every few weeks RECC experienced the loss of a drop or two. This turned out to be a very subtle timing problem. Data was buffered at a drop for just enough time, a couple of seconds at most, to cause the SCADA host to timeout a poll, even though the RTU response eventually reached the host. With a number of older protocols that do not have numbered blocks or source addresses on the responses from the RTUs, delayed RTU responses coming to the host after the timeout period confuse the host polling software. RECC’s temporary fix for this problem was to reset an SMD, an event that occurred an average of once a week.
The problem was solved with updated SMD firmware. Port buffer sizes were changed and buffer timer was added. The timer is used to insure that data will not be buffered in the remote SMD so long that the host computer has already timed out the drop response. Now, with the updated firmware, if the RTU response sits in the SMD buffer too long, the data is discarded by the SMD, allowing the host computer to do its normal timeout and re-poll without getting an extra, delayed response.
The delays in RTU responses may come from heavy traffic, but the most typical cause of delays are errors on the communications links. The SMD must recover from errors on the radio link, just as a SCADA host must recover when an RTU fails to respond. While taking the time to recover, the SMD is not sending user data across the link. This is the time when the data was buffered at the drop and came into the host so late that the SCADA host assumed the data was lost and polled for the data again.
How the SMD improves the radio system throughput
The SCADA and ARM systems each poll at a rate of two times a second. They poll fast enough to keep the system control information up to date. The SMD does its polling at a rate of about 30 polls every second. Since the SMD polls so much faster than the SCADA and AMR systems, the SMD fills in the idle time gaps when the SCADA and AMR systems are not sending or receiving data. The SCADA system and the AMR system each use the MAS radio system at about 15 to 20% of capacity. The SMD fills in the time gaps, greatly increasing the efficiency of the communications channels.
Polling and response times will vary as the polling sequence of the SMD and the host computers go in and out of sync with each other. The SMD shares the network bandwidth by delivering data from all ports of the host SMD to the remote SMDs. At the same time that the host SMD is delivering data to the remote SMDs, it is also retrieving data from the remote SMD ports. The SCADA system and AMR systems are also polling systems, each operating on its own polling schedule. The SMD, SCADA and AMR polling rates are not in sync with each other. Consequently, the response times for the SCADA and AMR systems will vary. In the case of RECC, the average response time is about 250 milliseconds, with response times varying from about 100 to 400 milliseconds. This is typical for an MAS radio system application.
Conclusion
The bottom line for RECC has been the substantial cost savings. RECC has added 3 channels to their single channel radio system. Considering the alternatives of a parallel radio system or phone lines, the SMD multiplexer approach has saved months in time and at least $20,000 in new hardware and installation costs. A few small glitches had to be worked through, but all of the technical issues were solved with little or no impact on the SCADA and AMR systems. The impact was less than adding a new communications system for AMR and changing out all the RTUs from ILEX to DNP3.
About the Author
Mr. Straayer is President of Data Comm for Business, Inc., a position he has held since founding the company in 1981. Prior to that, Mr. Straayer was Vice President of Compre Comm, Inc, from 1977 until 1981.
Mr. Straayer is a graduate of the University of Illinois Springfield with a degree in Communications. Mr. Straayer has consulted for AT&T, Harris Bank and Trust, General Telephone and other major companies. He has been an instructor in data communications courses for the Federal Reserve, EDS/GM and for many public data communications courses.
Mr. Straayer was a telecommunications manager with the State of Illinois where he was responsible, in 1977, for a $35,000,000 budget. He was responsible for the 1977 implementation of a credit card calling system that included voice recognition equipment to automate the placing of credit card calls.
