In an earlier edition of EET&D (November-December 2014), we looked at San Bernard Electric Cooperative’s (SBEC’s) installation of MultiSpeak specification including a metering system, an outage management system (OMS), a customer information system (CIS), and a geographic information system (GIS). Topics covered a company overview, the business challenge, and solution description.
Analytics and alarms using OMS and field viewer
When SBEC first implemented the MultiSpeak specification between the AMI system and the OMS, its system operators were receiving alarms posting as outages on the OMS. Obviously, this sort of reporting pattern is not desirable. Data has to be delivered and presented to the dispatcher in the right manner every time or the dispatcher will experience overload, lose trust in the systems, and begin to second-guess the meaning of the data.
The irregular reporting between the AMI and OMS system was caused by untested, purported interoperability between the vendors’ systems. The AMI vendor was quick to act and correct the problem by setting the meter alarms/meter events as ‘no response’ (NR), in the MultiSpeak message payload. However, the OMS system was unable to digest NR data. That is, OMS did not have anywhere to put such data since it was not setup to receive this data. This left utility operations personnel unable to see the alarms, which was unacceptable, particularly in light of the fact that the alarms feature factored into the decision to choose this particular AMI technology. The solution required customized code.
The smart meters of SBEC’s AMI system can send many types of different alarms. The following list provides examples of the alarms that the AMI meters can provide to operators:
- Power Fail – MultiSpeak Outage
- Tamper – MultiSpeak Outage
- Brownout – temporary low voltage
- ReadFail – Unable to read meter
- Hot Socket – Fire is imminent
- Low Volts – sustained, under-threshold voltage for a period of time
- High Current – temporary current overload
- Meter Power Fail – MultiSpeak Outage
- Reverse Power – Generator present?
- High Voltage – temporary above the voltage threshold
- Disconnect Fail – Unknown disconnect state, which could be serious, e.g., a meter fire
Some of the alarm data was transmitted correctly from the AMI system to the OMS for the utility’s MultiSpeak Version 3.0. In instances where the data was not transferred correctly, the utility’s IT staff developed customized integration code.
The AMI vendor was using an SQL database. First, the utility’s database expert had to identify which tables were storing the alarms. Then, the behavior of the AMI system and the significance of each alarm had to be determined.
For the OMS, the database tables and interrelationships had to be understood well enough to update the correct tables with SQL commands in order to display the desired results in the OMS in a usable manner. This took the cooperation of each vendor, expertise in using SQL, access to the right tables in the right sequence, and a patient group of employees. IT personnel had to explain the different alarms to the dispatchers so that they could take appropriate action in response.
Sometimes, more information does not improve situational awareness. For instance, the operators were seeing brownout alarms (low-voltage events) with almost every outage. During a lightning storm, the dispatchers’ OMS screen would be covered with brownout alarms that were simply dismissed as being related to an outage. In response, the dispatchers turned off the brown-out alarms. That also meant, however, that the utility would not be archiving those alarms when they weren’t related to an outage – an undesirable outcome. Figure 3 shows an alarms screen presented by SBEC’s Sensus software. It does not alert anyone when a new alarm occurs. This is true for all alarms using the software package SBEC chose. Sensus has since developed software for specifically dealing with alarms
Sifting through all of the alarms in the AMI system manually was too complicated and time-consuming. All of this interpretation would be left to the IT experts to analyze. (What if the customized code missed something that was important?)
Instead, the programmer wrote SQL code to look at the alarm view in the Sensus AMI database and present a brown-out alarm as a note on the screen. Each note simultaneously received an expiration date and time.
As part of the custom coding, the dispatcher receives an audible alarm as well as a symbol on the meter in the connectivity model within the OMS.
Another example of a customized code fix was related to providing an alarm when a voltage regulator malfunctions. In the screenshot example below (Figure 1), a view of alarms on an outage map (with custom SQL code integration), the dispatcher received all of the alarms shown in green at the same time. They were all high voltage alarms, all on the same phase (A phase) and all down-line from a regulator on the circuit connectivity model.
Figure 1: View of alarms on an outage map (with custom SQL code integration)
A crew was dispatched to the regulator where they discovered that the regulator controller had indeed malfunctioned and was outputting high voltage. The regulator was taken off-line and the problem was corrected. Less than an hour passed from the moment that the alarms were received until the problem was corrected. No customer phone calls were received.
Just a few months prior to implementing this system integration, SBEC experienced a similar incident. The utility was not aware of a high voltage issue until it received phone calls from its members. Once the source of the problem was identified as faulty equipment on the utility’s power line, SBEC received multiple insurance claims which it had to pay. If the MultiSpeak specification had been in use, its alarm system would have prevented such a development.
This example underscores that an investment in the smart grid and in interoperability of typically disparate systems provides a tangible return on investment
Figure 2: After custom SQL code integration: the dispatcher’s view of alarms on an outage map
Figure 2 presents the customized screen shot of the high voltage alarms after the customized SQL code integration. Once the alarm is dealt with, it is closed and becomes a permanent historical record on that meter location. Data collection showing the source of alarms is very helpful in determining the number of times a particular location has produced an alarm – either by a human phone call, IVR call or by a meter signal. The utility’s basic AMI system is set up to keep only 60 days of history. Therefore, another system is needed for storing a record of all the alarms to facilitate ensuing analytics. In SBEC’s case, the utility stored these alarms in the OMS (see Figure 3).
Figure 3: A permanent historical record is made of all alarms, outages, and phone calls at a particular location
Figure 4 illustrates the view of a dispatcher receiving a ‘Hot Socket’ alarm. In this case, the service crew was on the scene within 20 minutes of receiving the alarm. The crew reported the meter to be too hot to touch. There was a short in the underground service entrance below the meter. The meter was red-tagged as a hazard. The member was notified and a potential structure fire was prevented.
Figure 4: Dispatcher receives a ‘Hot Socket’ alarm
Not all alarms need to be dealt with by dispatch. For certain types of alarms (ReadFail, e.g.), SBEC wrote additional code that displays the alarms in the field viewer – these are designated as ‘Unplugged’ (see Figure 5).
Figure 5: An alarm indicates that a meter has failed to provide a read at a designated time
The software also sends an e-mail to the responsible parties to alert them to the ‘ReadFail’ alarm so that it can be dealt with accordingly (see Figure 6). Before using the MultiSpeak specification, these kinds of problems were not known until a problem appeared with the readings in the utility’s billing department, which might occur as long as a month after the problem transpired. Since the MultiSpeak implementation, the utility knows about a ‘ReadFail’ alarm within minutes and it can be resolved before it becomes a billing-related issue.
Figure 6: Automated notification of a ‘ReadFail’ alarm
In summary, SBEC wrote additional customized code to classify AMI alarms as part of AMI alarm integration with the OMS. Examples of those alarms included malfunctioning voltage regulator controls that could result in high voltage-damage to SBEC members’ equipment and result in insurance claims; malfunctioning transformers that fail to provide American National Standards Institute (ANSI)-compliant voltage to member’s residences and hot sockets that could cause a fire.
Interoperability-related lessons learned from SBEC’s Implementation
The lessons learned from SBEC’s MultiSpeak implementation (Specification Version 3.0) were many.
Careful attention must be paid to software issues relating to iterative versions. Just because a vendor is MultiSpeak compliant doesn’t mean that the vendor’s system can integrate seamlessly with MultiSpeak-compliant systems the utility already has in place. Different vendors may be running different versions, so it is important to understand the capabilities and limitations of the version of the specification that your vendor is supporting – especially as versions are not backwards compatible.
Further, methods needed to accomplish a business process may not be supported by the vendor. Be aware that software packages may include fixes written by ‘rogue’ software writers. Although those fixes may have been written to integrate with utility-specific methods, they may misinterpret the purpose of a method and/or fail to follow the standard. When MultiSpeak Version 3.0 was used to integrate SBEC’s OMS and AMI systems, the utility discovered that it did not specify how to handle transmitting meter alarms to the OMS. Since SBEC’s implementation, a newer version of the MultiSpeak specification, version 4.1, now has the capability to address meter alarms and AMI system and OMS vendors are now writing to the new version. Additionally, a version transition strategy is being developed for SBEC by its vendors which will avoid loss of data, downtime, and functionality.
Generally speaking, custom integrations are to be avoided, for several basic reasons. First, for example, the programmer who develops the customized interface may leave the utility and no one else will understand the implementation. Second, it is much better to have the utility specify its interoperability requirements to the vendors as part of the procurement process. Third, it is not good risk management practice to have someone other than the vendor access the databases of complex software because such access could negatively impact a utility’s mission-critical business processes by undermining certain core functionalities of the software.
General recommendations to industry
Before implementing major integration processes, the implementation team needs to identify and understand the utility’s methods and the correlated business processes and functions that these methods are expected to support. Mapping data transport from source to value creation sounds obvious, but there’s no substitute for mastery of the details in this context.
A utility procuring the MultiSpeak specification should require that prospective vendors be active participants in development activities. This requirement should be included in request for proposals (RFPs) as well as in awarded contracts.
Interoperability should be defined in broad terms. A particular version of software should not be specified. Instead, procurement requirements should focus on the business processes that need to be supported and integrated.
The burden for staying compliant, up to date, and functioning in an interoperable manner should be placed on the vendors.
This requirement should also be included in service level and maintenance agreements with vendors. In fact, testing regimes need to be improved. Testing is invaluable to vendors and utilities to ensure interoperability and provide confidence to the end user that systems and devices will integrate and interoperate. Similarly, interoperability testing between vendors should be required before signing contracts and interoperability affirmation/assertion documentation should also be required to prove that the testing has been successfully completed.
But documentation is not sufficient. The procuring utility needs to study the methods within the interoperability affirmation/ assertion documentation to ensure that the testing employed addresses their required business functions. If pertinent methods are being omitted in the testing, the utility needs to question why and insist on a commitment from the vendors to include those methods in follow-up testing.
Finally, interoperability testing between utility systems should be required as part of system-acceptance testing. This requirement should always be explicit in the contract. If there are methods that one or multiple vendors are not supporting in the future, but the methods are available in the current supported versions and the utility needs the methods for its business processes, it needs to get a hard commitment from all vendors that they will add those methods to the versions they are offering.
About the Authors
Doug Lambert is the Senior Principle of Distribution Engineering and System Automation with the National Rural Cooperative Association, an organization of more than 900 not-for-profit rural electric cooperatives and public power districts providing retail electric service to more than 42 million consumers. Previously, Doug worked with the San Bernard Electric Cooperative in Texas for 15 years as an IT manager and SCADA and engineering data supervisor, among other roles.
Dominic Geraghty is a senior consultant with the Smart Grid Interoperability Panel, a member-funded organization that engages Smart Grid stakeholders to create national standards and advance interoperability, Dominic has spent over 20 years working to advance the grid. He has been a general partner at a venture capital fund and worked in a large private equity fund, both focused solely on energy deals. He currently serves as managing editor of SmartGridiX and as executive chairman of both Smart Energy Instruments and N-Dimension Solutions.
