Today’s utilities are facing a lot of uncertainty about the future direction of their business and industry.

Will energy be more democratized, with advanced distributed energy resources such as solar panels changing how electricity is sourced? Are there other distributed energy technologies in development that will have a significant impact on the industry? What will be the impact of the decarbonization of energy? What about changing demand patterns caused by electric cars and fluctuating gas prices?

The ramifications of these market forces could mean new business models or regulatory environments for electric utilities.

Staying Competitive Through Business Agility
With all of these challenges, utilities need to find a way to remain competitive. Enter the growing concept of business agility. Analyst firm Forrester Research defines business agility as ‘the quality that allows an enterprise to embrace market and operational changes as a matter of routine.’ In other words, a utility can only be as successful as its ability to quickly adapt to changing market and customer needs.

A key component of business agility in this environment is partner relations. Utilities will need new technology and energy acquisition partners and will need to manage those relationships well to improve their business agility and keep pace with industry changes.

Data is a Bottleneck
But here’s a situation that happens all too frequently. A company wants to do business with another company, executives do the negotiations, promises are made, a deal is struck. Then it comes time to integrate data and the process slows to a halt.

That’s because oftentimes data formats are different. With record sizes and data sets being very large, bringing this third-party data into a business system gets very complex. If the business analyst can’t do it using a spreadsheet and macros, then the IT team is called in to manipulate the data. All the while, the clock is ticking and the partnership is not moving forward. This can upset the partner and they may choose to go elsewhere.

Data integration is a growing solution to this problem that allows utilities to more smoothly and quickly – sometimes within hours – to integrate external data files into a business system.

What is Data Integration?
Data integration takes data from multiple sources and in multiple formats and transforms that data to be viewed within an existing business process. The result is that users have a unified view of multiple data sets. This process can be difficult because it often requires different coding for different data sets depending on the data type and source.

Data integration is powerful, but it’s no easy task and therefore has historically been in the realm of IT. But as utilities seek to expand their agility, there has been a movement to simplify this technology in order to provide software that can be used by the data analysts or customer integration teams that are close to the customers.

Data integration has been seen by utilities as a way to transform data for regulatory purposes. In these situations, the mission critical nature of the data meant that IT needed to ensure data accuracy after the transformation.

What is ETL?
Extract, transform, and load (ETL) is an industry term used to describe data movement and transformation across or within systems involving high data volumes and complex business rules. ETL processes are widely used in data integration, data migration, and master data management initiatives, as well as for loading data from the source systems into data warehouses, operational data stores, and data marts.

ETL processes are critical components for feeding a data warehouse, a business intelligence system, or a big data platform by retrieving data from operational systems and pre-processing it for further analysis by reporting and analytics tools. The accuracy and timeliness of the entire business intelligence platform rely on ETL processes.

Energy Acquisition Partnership Example
Acquiring electric energy assets is a great example of the data integration challenge facing utilities.

This transaction is collaboration between an energy company and other energy vendors with a heavy need to integrate each vendor’s uniquely formatted data into the main system in order to finalize an acquisition.

Too often, energy companies have home-grown or manual systems for this data integration that dramatically increase the time involved in onboarding a new acquisition. However, they also carry the risk of inaccurate information.

This was the case for one major regional energy corporation with nearly 400,000 ratepayers. The company received information formatted according to each of its energy vendors’ own systems, which meant manually copying that data into the company spreadsheet. Macros then had to be run in order to produce an output file sufficient for the company’s current acquisition system. The inefficient process was consuming valuable time and resources.

The company’s IT staff wanted to build a uniform process that would import vendor data directly into the acquisition system. The solution had to be able to handle each vendor’s file format, as well as translate some of the vendor-specific terms. The new system would have to be flexible and scalable, because new vendors and their proprietary formats are added all the time.

After evaluating available technologies on the market, the energy company chose an ETL-based data integration software solution because it provided an easy-to-use interface that enabled users to quickly map the vendor fields to the company’s own fields.

The software the utility selected allowed its partner team to convert the data taking a project off of the plate of IT without sacrificing the flexibility, scalability, and functionality required for the energy company’s complex application. At the same time, the team was able to quickly and easily map new vendor data to the main system. The company was also able to take advantage of the software’s static lookup functionality to translate vendor-specific codes.

Now, any time data needs to be integrated from an existing vendor (or a vendor who uses a familiar system), the energy company can open up the transfer setting corresponding to that specific vendor file format. For new vendors, a new transfer setting can be created in a matter of minutes and saved for reuse.

Business agility is a macro solution for many of the challenges that face utilities, but data and data integration are the foundations for making the whole effort successful. Data integration technology is a major step forward and should play a key role in the data processing infrastructure of all utilities.

About the Author

Jay Mishra is the Vice President of Product Development at Astera Software. Mishra leads the product vision and development for Astera’s data integration products and has over 12 years of experience in software engineering with over 8 years of experience in managing mid-sized, fast-paced and dynamic software engineering teams. Mishra holds a Master of Science degree in Computer Science from the Virginia Tech and a Bachelor of Science in Mathematics and Computing from the Indian Institute of Technology.