Utilities are facing an ever-expanding universe of devices, systems, and data sources that must be managed, controlled, and integrated into a variety of critical operational systems. Anyone who has been tasked with managing the software architecture of a utility’s smart grid knows just how difficult it is to integrate new smart grid systems and devices safely, cost effectively, and efficiently.

Some of the challenges you may face when integrating your smart grid systems and devices are:

• An alphabet soup of systems which you have to integrate to, such as OMS, EMS, SCADA, MDMS, ADMS, and DERMS systems. New systems are coming online every day at an increasing rate on both sides of the meter.
• Each OT system you employ has many man years of embedded operational and business logic that will need to be re-coded as each new system is brought on line.
• The appetite for consumption and analytics of real-time data is growing, but each new project brings new requirements for data cleansing known as ETL (Extraction, Transformation and Loading).
• Your smart grid data storage requirements are expanding exponentially, and you don’t have infrastructure in place to intelligently reduce the flow and storage of data.
• Not only are new systems expensive and slow to integrate, but the ongoing labor required to enhance, adapt, and maintain your point-to-point integrations is destroying your operating budget.
• Bridging data across your secure and CIP compliant systems and your IT systems is difficult to get right.

If you have read this far into the article, you likely already know that most of the systems comprising the smart grid are known as Operational Technology or OT systems. According to Gartner, Operational Technology is defined as ‘hardware and software that detects or causes a change through the direct monitoring and/or control of physical devices, processes and events in the enterprise.’ Throughout this article, we show that OT systems architects can learn a lot from IT enterprise architects, but before we get there, it is important to point out critical distinctions between OT and IT architectural design constraints. OT System architectures must:

1. First and foremost: protect life, equipment and environment
2. Maintain service reliability
3. Operationalize real-time bidirectional control
4. Support lossy and messy radio networks
5. Support legacy equipment and protocols
6. Respond to regulatory security and quality requirements
7. Deliver deterministic bi-directional real-time performance

IT systems connect the dots within business intelligence systems and databases. A failure in IT may result in lost data or a crashed application, but an OT failure carries catastrophic potential. Operational Technology carries the burden of operationalizing real-time, bi-directional control of field devices and the data they create, whereas IT simply must maintain the ability to consume and report data from within the grid and outside data sources as needed to maintain business processes. IT is maintained from the main office of utilities, while OT extends from the home office out into the field, to secondary energy markets, and back to Operations groups. IT looks at systems in terms of planned obsolescence, but OT must maintain legacy equipment and protocols as a matter of service reliability. The number of regulatory requirements in terms of grid security and mandated reporting are growing – and fall squarely in the wheelhouse of Operational Technology. OT systems must ensure that those requisites are met, with no room for error. Even though the OT and IT spaces are distinctly different, lessons from IT can be leveraged to create a rich OT environment.

We have demonstrated that OT systems are intrinsically different from IT systems, but there is an important software category that OT architects should be embracing from the IT world, and that is the concept of middleware. According to TechTarget:

Middleware often sits between the operating system and applications on different servers and simplifies the development of applications that leverage services from other applications. This allows programmers to create business applications without having to custom craft integrations for each new application.

Beginning in the 1980s, IT began implementing three-tier architecture that allowed a separation of data, logic, and presentation layers. With three tiers, changes to each can be made without affecting the overall stability or health of the IT environment. A critical aspect of successfully implementing a three-tier architecture is leveraging middleware to create fusion between each layer. Middleware acts as the glue that ties the three tiers together – delivering a homogenized environment in which systems and applications can utilize data from each other. While middleware is conceptually rooted in IT, middleware created expressly for use in OT scenarios is an emerging category. OT middleware is purpose-built to coalesce and manage data streams, data flows, and data networks.

By providing a rich real-time integration environment, OT middleware provides architecture that allows utilities to save time implementing and deploying grid connections, while reducing the number of ‘gotchas’ that come up in the course of these complex interconnection activities. OT architects use OT middleware to create a software pattern where new smart grid applications are created without having to custom craft integrations for each new application.

Let us use a real-world example of embedding operational logic into a middleware platform to help illustrate the value proposition of OT-centric middleware. Our customer has a SCADA system where the COV (change of value) counter will count up continuously until the SCADA system is reset. The OMS system in this architecture is looking for a counter that only shows the COV since the last data set was read from the SCADA system. The operational logic to make this conversion can be easily implemented in hours with an OT-centric middleware solution, but could take weeks or months if a SCADA or OMS vendor was required to produce a custom solution. Another benefit of the OT-centric middleware solution is that the operational logic remains when the inevitable new Outage Management System comes in to replace the current OMS.

There have been many approaches to integrating OT systems over the years. Here are some of the more pervasive approaches employed by many utilities:

Bring in the consultants
Bringing in a consulting company to launch a new OT system such as a SCADA or Outage Management System is standard practice. Man-years are dedicated to a custom coded point-to-point integration. These solutions are not only expensive in both schedule time and dollars, but also difficult to maintain and typically require ongoing consulting contracts or dedicated engineers to enhance as system requirement evolve.

SCADA System as the man-in-the-middle
SCADA systems are designed to enable Supervisory Control (and Data Acquisition). These systems are both over featured and under featured to act as an effective OT middleware solution. SCADA systems are great at consuming data and displaying data, but not very good at managing complex dataflows and transforming data and acting as a data source. Using a SCADA system as an OT middleware solution you end up with an expensive architecture that is hard to engineer into a generic reusable solution across multiple platform integrations. Using a SCADA system in this manner is like using custom coded software to deliver real-time management dashboards. You can make it work, but each new dashboard business requirement is painful, slow to implement, and expensive.

Standard Interface Connectivity
Smaller utilities have coalesced around MultiSpeak as a standard interface among their OT systems. Implementing MultiSpeak is a good solution if the only goal of the interface is to manage communications between applications. What this approach lacks is an engine to manage data flows and implement operational and business logic. There is no active filtering of data, and no native protocol support. When using MultiSpeak or similar technologies, the business and operational rules must be embedded in one of the OT systems. Embedded solutions mean custom code, higher maintenance costs, and more custom code when systems are replaced or upgraded in the future.

Using a Historian or a Database
Utilities are no strangers to traditional and historian databases. Many historians offer built-in connectors for various protocols. From a plumbing perspective, a database will surely move data from point A to point B; because of this ease of data movement, it is not uncommon to use a database as a form of OT middleware. Unfortunately, these solutions lack the ability to manage data flows, deliver deterministic real-time performance, or embed common operational and business logic. So while they do work, these approaches could be compared to using snail mail in place of email.

IT Middleware Solutions such as Oracle ESB, IBM WebSphere or TIBCO
IT Middleware or Enterprise Service Buses (ESBs) increase organizational agility by reducing time to market for new initiatives. ESB architecture serves as a way to implement communication between applications and systems; however, the data source and data consumer must share commonalities in data formatting, protocol language, and query-response pathways. Despite the fact that ESBs may be used to create basic ‘plug and play’ implementations, ESBs do not solve metadata incongruities, which are frequently an issue for OT systems. While ESBs are capable of meeting many organizational goals, they fall short of being truly performant based on the fact that they are an IT middleware solution, designed to work with other IT systems. The reality is that ESBs are not architected to handle the special loading instructions and customized data transformation functions with the fine grained, determinant timing required by OT systems. Rather than relying on simple database query and response patterns, OT systems rely on data that has undergone some level of transformation based on operational and business logic. ESBs lack native protocol support; given the diversity of OT systems, devices, and applications, such support is a must for systems employed by utilities.

Operational Technology Message Bus (OTMB) – OT ready real-time middleware is the solution
OTMB is a new category of middleware that was developed to meet the evolving needs of operations and business units alike. An OTMB is also sometimes called a Data Bus or Operational Data Bus. While a Data Bus is a good conceptual framework for an OTMB, it undersells an OTMB’s inherent ability to embed operational logic and manage complex data flows. OTMB architecture ensures that OT specific organizational needs are met, while helping to bridge the gap between Operational Technology and Information Technology systems. Fundamentally, an OTMB is a real-time data flow engine that is architected to meet the needs of integrating and maintaining an exponentially growing fleet of smart grid devices and systems. OTMB software patterns reduce the time and cost of integrating new OT systems and devices, and lower the long-term operating expense of maintaining and enhancing your OT systems.

OTMBs must have the following attributes:

• SCADA class in-memory processing to meet sub-second latency requirements.
• GUI-based Configurable data flows – mapping, statistical functions, filtering, and data reduction are required functions to meet the real-time data manipulations requirements of smart grid systems.
• Templating of data flows – Integrating hundreds of thousands of data points cannot be done through a GUI alone. An OTMB must support abstracting data types into a manageable and maintainable classification system.
• Populated at run time – critical OT systems must load new file configurations live. OTMBs must run 24/7 and allow for dynamic loading of new parameters.
• Network Scalable High Availability – single server is a single point of failure. HA support is required.
• Seamless integration with ESB and other IT systems – OT and IT systems must securely share information to maximize the value of your OT systems.
• Native Protocol Support – OT systems transact with protocols such as ICCP, Modbus, DNP3, OPC, and web services. OTMBs support a ripe environment to create reusable custom connections and legacy interfaces.

OTMB architecture allows for management of data flows and acts as a data engine – performing the critical transformation and loading functions required by outage management, distribution intelligence, and ISO/RTO markets. Rather than record and report data in the second range, OT systems are able to digest information in the millisecond range, and OTMB architecture enables high volume throughput to maximize the value and actionability of important system data. Most importantly, OTMB architecture facilitates control of devices, which IT systems are incapable of doing as a result of the real-time requirements associated with operating critical components of the power grid. Smart grid architecture encompasses field devices, distributed generation assets, and advanced data usage; OTMB architecture enables the advanced integration and manipulation of these attributes.

(click to enlarge)
 

OTMB architecture is established through the use of middleware like LiveData Utilities’ RTI Server Platform. RTI Server Platform and similar purpose-built middleware platforms are off-the-shelf products that have been designed from the ground up to enable efficient, cost-effective Operational Technology integrations. Utilities employing OTMB architecture can expect significantly lower integration and operating costs when creating data networks that utilize an OT centric middleware solution. Middleware eliminates the need for much custom coding, in-house development, or modifying IT systems to perform as a data engine. OTMB architecture anticipates the dynamic changes of applications, data sources, and data consumers within the grid and provides lower cost of ownership through the system’s lifecycle. Engineers utilizing OT middleware can integrate new systems and devices in a systematic, well-organized manner with the help of advanced graphic user interfaces.

OTMB uses operational protocols that existing operations applications already speak, and it interacts with each application with the correct latency, throughput, integrity, and dependability. Instead of relying exclusively on web technologies, OTMB uses high-performance communication protocols already in place across the grid and actively manages the flow and form of data across OT systems. It utilizes an organizing architecture to overcome the challenge of point-to-point integrations, so a utility does not have to solve the same problems to get systems to communicate, and it delivers complex data integrations in weeks instead of months or years.

Each day new technologies are changing the utility business. We are finding new ways to leverage data and are faced with an explosion of data sources that must be managed. Harnessing the power of data through OT organization is becoming a necessity for many utilities. At the same time, we must maximize the capabilities of field devices – ensuring consistent operation of distribution automation devices, maintaining service reliability, and improving transmission efficiency. OT systems keep our grids safe and allow them to function in a ‘smart’ manner.

Creating an Operating Technology Message Bus (OTMB) is one way to ensure that your organization will be able to cost-effectively manage and benefit from the explosion of data, systems and devices coming to the smart grid.
 

About the Author

Adam Reiss is a Senior Marketing Associate at LiveData Utilities™, managing the marketing activities for utility products and services. With a background in institutional research that covers everything from investor statements to coral reefs, Adam has a passion for data and enjoys delivering actionable insights from unlikely sources.