Undifferentiated data fueled the mortgage crisis. Now it’s exacerbating Texas’s energy woes.

The subprime mortgage crisis of 2007-2010 was, in large part, a failure of data analysis. By bundling subprime mortgage customers with different credit profiles together and monetizing their debts in bulk through mortgage-backed securities, banks and other financial players obscured homeowners’ wildly differing risk profiles. The result: buyers of these securities did not fully understand the risk they were taking on until non-payment started and everything bloomed into a full-blown global economic crisis.

Earlier this year in Texas, we saw a similar crisis unfold — not in the real estate markets, but in the energy sector. The blackouts that rocked Texas are the result of unexpectedly cold weather, of course. While a lot of scrutiny was placed on the generation outage, on a deeper level, just like the mortgage crisis, this is the result of using undifferentiated data to guide decision-making. And these problems aren’t unique to cold weather. As we saw this past summer, warnings on AC usage to curb electricity consumption in the warmer months too, also showcase the issue as much greater than just a snap of cold weather. Without efficient demand analysis, we can’t price energy effectively. We also can’t send the right signals to energy producers and ERCOT to articulate current and anticipated future energy needs. And we can’t send the right signals to customers about how they should manage their energy usage during periods of high demand and systemic stress.

Let me explain. As things stand, many energy providers today are “bundling” customers with different demand profiles to come up with load forecasts. In other words, energy consumption forecasts are based on average energy usage by average consumers. Often this works, more or less. But when you encounter unexpected events — such as a sudden plunge into Arctic temperatures — those averages aren’t as much use. Not only this, but compounding issues, like increased electrification, climate change and population growth, show that the industry has turned a blind eye to these issues for years.
 

Figure 1: The above costs are derived from trailing 3-year spot market prices
 

To cope with tumultuous times, energy providers need to use granular data, based not on broad historical averages but rather on bottom-up insights from smart meters and other localized data sources, such as weather and socio-economic trends. Only by understanding how different kinds of customers use energy, and how that varies according to factors such as weather or time of day, can we hope to develop rigorous, effective systems for energy management and eventually energy transition.

To put this concept in context, Texas energy providers typically bucket residential customers into two broad groups: electric heat and non-electric heat (highlighted in blue in the graph – Figure 1). Look closer, though, and it becomes apparent that there are actually at least 36›groups of different residential demand profiles present in this market.

If you’re looking at only the electric heat and non-electric heat profiles, then the price separation between the two doesn’t seem that wide — just a 19% difference between the two groups. When you look at the full spectrum of 36›groups, though, we can see that in reality, the price spread is 107%. So, what’s the impact of these two approaches? Well, if you’re only grouping your customers into two buckets, you’ll wind up with a significant number of customers being under-priced by an average of 36%. In these cases, energy providers are not including enough risk premium, and are relying too much on ERCOT spot markets. This can lead to huge collateral calls, which ultimately result in sudden bill increases for customers and default for energy providers.

At the other end of the spectrum, some customer groups are overpriced by an average of 46%. When that happens, energy providers will lose these customers to smarter energy providers that have differentiated and customized pricing. That, in turn, skews the energy provider’s portfolio, since they’ll be left with a higher proportion of high cost-to-serve customers.

This is a simple example, but the reality is that a lack of granular data affects all aspects of energy generation and distribution. Without efficient demand analysis, we can’t price energy effectively. We also can’t send the right signals to energy producers and ERCOT to articulate current and anticipated future energy needs. And we can’t send the right signals to customers about how they should manage their energy usage during periods of high demand and systemic stress.

If we don’t understand demand drivers at the level of individual customers, we’ll never be able to cope with disruptions like the extreme weather that impacts Texas. A granular understanding of customer needs can help you understand how demand patterns will change in unusual circumstances — but if you simply lump everyone together, you’ll be subject to unexpected shocks and spikes, with disastrous results. Of course, the blackouts that have rocked Texas aren’t solely due to poor data practices. It took a succession of errors — including subpar weatherization, incorrect pricing plans, lack of customer communication, poor maintenance and bad planning — to create the perfect storm. But better data and more granular insights into energy demand patterns will make it far easier to make the right infrastructure and maintenance investments, smooth out energy-demand peaks and make better decisions.

This isn’t the first time the U.S. has faced blackouts and failures of energy infrastructure. With the world warming and extreme weather growing more common, it certainly won’t be the last. Winter is again fast approaching, and we need to get to work now to build more resilient and sustainable energy systems — and that means not just tougher pipes and turbines, but also better data intelligence and analytics to help us make smarter decisions in a turbulent and rapidly changing world.

The 2008 financial crisis taught us a hard lesson about the need for granular, well-differentiated data. It’s time for us to apply those lessons to the energy sector, and ensure we have the tools and data we need to effectively serve consumers.