Perhaps the most talked about issue currently plaguing companies in the energy sector is the surge in weather-related power outages. While escalating weather-induced power outages necessitate a response from these organizations (which, thankfully, many have via investment in alternative energy sources and smart grid technologies), such outages do not represent the only type of outage afflicting those in the industry. In fact, network outages, also known as unplanned downtime or the failure of internet services, continue to rise alarmingly and precipitously. If energy companies do not proactively mitigate these worsening network outages, the consequences could be severe — particularly for global organizations with limited engineers covering vast business regions.

Measuring the costs of mounting network downtime

Network outages are not a novel phenomenon — however, their increasing frequency does represent a more recent development. An independent study conducted in 2022 discovered that over half of the 500 network engineers surveyed reported a notable rise in outages since 2020. Likewise, the time it takes network engineers to find and resolve an outage (the MTTR or Mean Time to Repair) continues to lengthen. By comparing the findings of a similar study in 2020 to the one from 2022, one will see that the MTTR grew by almost two hours, with the average today being 11.2 hours.

The growth in MTTR is especially concerning for energy companies, and the culprit is the increasing complexity of modern networks. Trends like digital transformation, virtualization from cloud migration and the rise in remote and hybrid work continue to complicate a network engineer’s job, making them more challenging to manage and remediate. Simultaneously, the network is becoming more inseparable from everyday business operations. As a result, the consequences of unplanned downtime have also gotten costlier.

The total cost of downtime is difficult to calculate, involving many factors like lost revenue and productivity and recovery fees. Nevertheless, a report from the Ponemon Institute estimated that the cost is around $9,000 per minute. Another survey conducted by Enterprise Management Associates and facilitated by BigPanda, found a correlation between the increase in cost per minute and the size of the business. For example, an organization with 1,000 to 2,500 employees would lose $1,850 per minute, whereas a company with over 20,000 employees would lose $25,402 or $1.52 million per hour. While these metrics help one conceptualize the monetary consequences, there are also intangible costs, like brand reputation and employee well-being, which can be just as, if not more, damaging.

In light of these findings, those in the energy sector must build more resilient networks through out-of-band management solutions and remote management software to minimize and recover quickly from downtime, thereby ensuring business continuity.

Case in point: Remote network access

The network is vital to modern businesses, especially for a global industrial automation and energy management leader, touting 128,000 employees worldwide and $32 billion in annual revenue. This company provides products, including connected room solutions, switchgear and smart panels, automation software and digital services to help data centers, businesses and homes become more energy-efficient and sustainable. Over the years, it developed a thriving open ecosystem network of various parties, from customers and suppliers to universities and technology companies, driven by the need for interoperability, strong IT network and technology partnerships.

The energy management company’s network is pivotal to powering its smart factories and distribution centers. And its IT Network Service Delivery team was responsible for ensuring the timely and secure delivery of data and communications. However, the company’s global footprint required that its IT teams get distributed among its four regional hubs: North America, Europe, China and India. While this distribution allowed them to respond to local circumstances and challenges, like recruiting, its IT Network Service Delivery team was relatively small. Ten engineers were assigned to a region, supporting hundreds of sites across different countries — in the US alone, engineers had to manage 350 sites.

Moreover, like the rest of the world, this leader in energy management experienced considerable challenges and supply chain disruptions from pandemic-related lockdowns and travel restrictions. Nevertheless, despite these limitations, users and devices must still access the network; other essential tasks like provisioning, configuration and troubleshooting (in the event of an outage) could not get put on hold either.

Needing a secure and intelligent means to access, monitor and control critical devices, including their remote management interface in their data centers, the IT Network Service Delivery team looked to their out-of-band provider, a partner of over a decade with whom they already had many solutions across the enterprise. By implementing the provider’s out-of-band management solution, the energy management company’s IT team could streamline tedious administrative processes, conduct proactive maintenance and facilitate remote recovery.

The company also used its out-of-band provider to console into routers, switches, firewalls and over 112 access points, helping it spin up new offices and stage new sites or upgrade equipment in others. Lastly, the leader in energy management leveraged its provider’s remote management software, a unified and easy-to-access dashboard, to access and orchestrate its routers, switches and firewalls from anywhere.

The out-of-band management solution brought peace of mind to the IT Network Delivery team, helping them reliably and remotely access, manage and remediate critical devices even during a disruption, promoting high network service quality. Access and provision to remote sites to perform essential tasks was invaluable — especially during the pandemic, where physical travel was impossible due to the lockdowns.

On average, it can take a network engineer 3.6 days, including travel time, to go to a remote site and remediate an outage. These trips are also expensive due to travel and hotel bills, and expenses typically compound with global organizations. Additionally, they’re using valuable employee time that could have been spent on high-priority or business-enabling projects However, with its out-of-band management solution the IT Network Delivery team reduced travel costs from $80-100k to $8k for 90% savings. Likewise, remote capabilities allowed the team’s limited staff to cover the company’s massive area of business while decreasing the duration of network downtime significantly. As an additional bonus, regularly accessing connected devices from leading vendors helped the IT Network Delivery team members upskill and earn certifications, boosting their career journeys.

Important considerations

When comparing vendors and assessing the effectiveness of different solutions, companies in the energy sector need to ensure that they ultimately create a resilient network. Regrettably, many businesses will settle with a redundant network. However, as demonstrated by the global leader in industrial automation and energy management, a truly resilient network will save much more money in the long run. This network could provide and maintain an acceptable level of service amid challenges to normal operations while allowing engineers to access critical connected devices remotely.

Additionally, those in the energy industry need to find a network resiliency solution provider that can prepare and support them for the first day, worst day and every day in between during the network lifecycle. Ideally, this provider should offer remote management software and an out-of-band management solution with the presence and proximity at every location to support emerging network requirements. Such solutions will permit IT teams to establish a strong network every day and seamlessly access devices in remote data centers to save money and stay prepared.