The Good: A FERC Order providing an extension of time to defer the implementation of the critical infrastructure protection (CIP) version 5/6 Reliability Standards from April 1, 2016 to July 1, 2016.

The Bad: Establishing and maintaining cybersecurity in an industrial setting, such as the substations, is still extremely complex and difficult. The concern is always, “Am I taking the appropriate security measures, is my organization capable of sustaining this level of operations, and ultimately is the money being spent in appropriate places to enhance security?” as opposed to just making sure a utility complies with the literal meaning of the current standards.

The Ugly: Version 5/6 is steeped in the fear that a major cybersecurity breach, impacting the North American Grid, is just one hack away. Today, cyber terrorism is expected and it’s a utility’s responsibility to have effective protection and a tested response and recovery plan in place. The primary goal of Version 5/6 is to prevent, or at least minimize the impacts, of any cyber attack on the Bulk Electric System (BES) components of the North American Grid.

The Opportunity: To use this extension to not only continue to drive towards compliance and meet the letter of the law but to now take this opportunity to plan and deploy more optimum technology and process solutions that also meet the intent of the law – to improve the security posture of the North American Grid.

There Is No Substitute for Connectivity and Visibility into the Substations
With regard to CIP Version 5/6 compliance, there needs to be a full understanding of what works and what does not. The majority of utilities may not be confident that they have the right approach yet – even at this late stage. Many organizations will now target compliance by July 1st, but will compliance adherence be improved? Or still just enough? Will their security posture have been improved? Is it more sustainable? As of July 1, was this time used effectively?

The choice of appropriate ‘security (compliance) management’ tools, technologies and processes to be deployed at the substation and in the associated control and switching centers has been largely left to each utility. As illustrated here, the thoroughness of the solution deployed will be a function of that utilities current capabilities, risk appetite and accuracy of the protected asset inventory.

For instance, as part of its security operations each utility needs to determine to what extent the substations and the level of connectivity can or should support remotely handled password management and account management. Often this may require significant reconfiguration and perhaps device upgrades. Utility companies are struggling to succinctly address this question because the number of devices to manage at every substation is significant, and the variation in types of devices is overwhelming. Following the extensive inventory efforts at the substations that were demanded by CIP-002, there is a realization that a significant number of IEDs (intelligent electronic devices) will still require manual intervention for password management as opposed to being automatically managed by a tool. For each type of these legacy devices a manual protocol must be designed and implemented requiring extensive vendor coordination and compliance and awareness training for the substation support staff.

Additional considerations for security monitoring of these devices include:

• Are utilities looking at the appropriate type of alerts and events?
• Who is tasked to monitor all of this information?
• How is this data being analyzed?
• Is the assessment of risk based on local or regional knowledge?

Another example from the model relates to Asset and Configuration Management. Of course, Asset Management is a huge enterprise concern at most utilities, but now it takes on a very serious edge, as its effectiveness will impact the security and compliance of the utility. Utility organizations must now address the management of the asset-related information across the information and operational technology boundary. Even now as we approach the original target date, many utilities are still having an internal battle deciding which, and what level of granularity, information goes into corporate asset management systems versus into similar types of systems at a utility operating or business unit. The issue is not with regard to the federated data model, but simply how to coordinate the update of the information effectively in the industrially oriented, critical infrastructure environment that the substations represent. Simply put: as we understand the situation today, utilities are not comfortable that all of their current solutions are the right ones!

Because of this conundrum, the result may be a sub-optimal approach, where different versions of the same data are maintained on separate systems. Taking this approach, utilities will be still prone to preventable problems – if the wrong database is selected, an entire substation can be affected. The best solution resides with a properly targeted asset management solution – away from antiquated spreadsheets and multiple localized technology solutions which if not coordinated with the main asset management system, inevitably will cause many problems.

Exacerbating the asset management problem is the fact that at this point in time many utilities have chosen not to ‘IP enable’ their substations, even though it may make operations more effective. While understanding that this minimizes the number of CIP Version 5/6 compliance actions a utility needs to take today, this results in a blind spot or a lack of visibility into the ‘hackable devices’ at the substations, or the presence of unapproved devices and local malware agents. Not to mention that the intent of the regulation was to improve security at our critical infrastructure not to provide an open invitation and opportunity for cyber terrorists. The bottom line is that if this opportunity is taken the next set of regulations will be even more extensive mandates as to what the utilities need to do to protect the nation’s infrastructure.

It seems that the security problems targeted for solution by CIP Version 5/6 may continue at every corner, and additionally the cost of compliance and an improved security posture is a major concern.

Consider the financial cost associated with achieving the cybersecurity capabilities that were the intent of full CIP Version 5/6 compliance. It is a significant monetary commitment for a utility to implement the needed IP connections and monitoring capabilities at every substation in order to ensure adequate protection. The fact is that most utilities cannot afford this, so they conduct a balancing act where the security breach risk is weighed against the operational cost of running and sustaining IP connectivity to every substation. On one side of the scale, utilities have a reduction in operational costs and complexity by limiting the number of compliance requirements to deal with, on the other side of this they are reducing the visibility into the substations for monitoring any unwanted activities, and in many cases also limiting the operational efficiencies a fully ‘IP enabled’ substation will bring.

But it’s not all financial gloom and doom: utilities can leverage some of the existing communication infrastructure and also use this moment in time to improve and update outdated and unsupported technologies that may be deployed at some of their substations. While understanding that this cost is not insignificant, the cost of implementing the necessary IP to adhere to the security intent of the CIP V5 Standards at the substations is reduced if the facility is already connected to the main facility with fiber optics, and the long term operational, security and compliance benefits are definable and appropriate for action today.

Who Is Inside The Substations?
In addition, careful consideration must continue to take place to ensure the appropriate team is upgrading and replacing the devices at the substations. And while compliance demands that utilities need to know the background of every individual walking around their substations, it should be operational and security concerns that determines the mix of IT and OT capabilities and skills of these people. Changing the firmware in an operationally deployed relay or RTU requires coordination with system operations as well as an appropriate understanding of how to ensure that the unit is being put back into service appropriately. You cannot just pull the relay and shove it back into the rack.

Utilities are continuing to think through these issues and as before indications are they are not sure they have yet arrived at the right answer. Concerns remain:

• Should the general IT staff have access to a substation to install switches?
• Should the general IT staff have administrative access to our critical control systems and technologies?
• How is the company managing the operations and maintenance of the substations that are shared between the distribution and the transmission systems?
• How do you manage the actions of those who have access to the substations?
• What is the best governance and operating model for the ownership and management of the substation devices and other control systems?

Adding Some Perspective on Asset Management
From a security conscious asset management perspective, utilities need to take a good look at their device inventories and the required configuration data to effectively formulate a proper governance plan for substation and control system O&M. Oftentimes this is more difficult than initially perceived because asset management at utilities has multiple completely different sets of internal stakeholders with their own operational drivers, in addition to the vast array of information forms and genre of data to contend with. There is a significant investment of time, money and operational egos required to effectively address this issue and effectively ‘normalize’ the data to appropriately match all of their needs.

Unfortunately, many, if not most, utilities do not have the time to spend to properly understand all the data to the needed comfort level of each stakeholder to achieve consensus; it helps that most of this data is already electronically maintained in some form, so utilities have an identified starting point. However, simple compliance with Version 5/6 remains the current ‘heartache’ around the extended July 1st deadline. Given where we are today, for most utilities it has to be sufficient to make sure that the devices are configured correctly, governed by the correct firmware, or secured with appropriate measures and the mitigation plan is defined.

The “Low Impact” Substation Loophole
To fulfil the objectives of v5/6, the compliance (and security) of the low impact substations will still need to be addressed by April 1, 2017. The glaring issue is that most utilities in the U.S. have not even begun to consider this issue. The scale of the Version 5/6 High/Medium Impact work has been so significant that most utilities do not yet have the available resources to even begin to plan or consider the operational issues ahead for this effort.

In fact, when you look at the compliance requirements for the low impact category of systems and facilities, there are actually a very small number of requirements that are mandated to be addressed. For instance, as many of us recall from our frantic efforts throughout the past year, and the discussions above, the maintenance of an accurate and useful inventory of high and medium cyber systems and assets alone is often an Herculean effort. But in this case for the low impact cyber systems and assets this is not required. This may sound good, but this lack of a mandate may drive some additional sub-optimal practices.

Most likely we will now have additional sets of inventories, devices and configurational complexities to manage and upkeep for the lowly ‘rated’ groups of our protected assets. Purely from an efficiency perspective, let alone security considerations, shouldn’t all of the information be handled and managed in the same manner regardless of currently defined impact levels? Threats change, and quite simply put you are only as secure as your weakest link. And if that link happens to sit on the same network as your high impact systems and assets – advanced malware and persistent attack methodologies may represent a severe security risk.

Additionally, because the ‘operational’ application of the advances in security and compliance are coming at a significant investment for the high and medium ‘rated’ facilities, it may just be business reality that limits their deployment to the low sites. Consider a utility that has 50 medium impact ‘rated’ substations. The same utility may have 300 low impact ‘rated’ substations. Evaluating the effort and costs of securing these low impact ‘rated’ substations with the same tools and processes starts out with a 6X multiplication factor and goes upwards from there as many of these substations may not have connectivity, may largely consist of electromagnetic devices not IEDs and may be in distant locations with minimal physical security.

The good news (from a cost perspective) is that utilities have this ‘loophole’ to make a judgment call as to how much security to deploy in addition to the minimal required compliance requirements.

The bad news is that most utility companies may have not even begun to consider this because they are still rushing to get the more highly ‘rated’ facilities and systems ready for the July 1st deadline.

The big question is: will they address the threats at the lower rated facilities adequately? Just being compliant will not necessarily improve the security posture.

In Conclusion
Finally, even if all the planets properly align on July 1st, there is still the ‘stress test moment’ to contend with. All the planning is completed and utilities have put the best systems in place, but the rush to meet the deadline may be so extreme that most of these utilities may still not have had the chance to test the security solutions and validate that they’re operating properly with no impact to the reliability of the grid. In many cases, the effectiveness and efficiency of these new security systems and processes will have to be assessed, and more appropriate solutions defined and deployed to fill any gaps.

The CIP version 5/6 compliance efforts is more like a triathlon, just when utilities have completed one section of the race, there is another leg of the triathlon to accomplish, and version 7 is already in the works.

Utilities should view the compliance mandates, and in particular the extension to July 1, as an opportunity to do the right thing from a security perspective – that is FERC’s intent. It is understood that it’s next to impossible to be 100 percent secure, particularly with the utilities’ scale and variety of assets and facilities; it all comes down to matters of degrees in reducing the risk of a breach. Ultimately, it’s about finding and establishing the best-fit solutions, with the best combination of stakeholders that address your threat and risk profile across your operational assets and facilities regardless of compliance driven ratings. Finally, if utilities don’t have the in-house expertise or available staff, there are consultants that understand the big picture and can help manage the associated costs and meet the needed level of resources. Maintaining the reliability and ongoing security of the North American Grid must continue to be the paramount objective of the utility.

About the Author

Richard Jones, VP, Grid Security, for BRIDGE Energy Group, is a recognized thought leader in Cybersecurity, NERC CIP and general utility regulatory compliance and reporting with over 25 years of energy and utility industry experience providing business, technology, and management consulting based services. Prior to joining BRIDGE, Richard held a number of security leadership positions with the big 5 and industry focused consulting firms.