Bahman Hoveida
President & CEO Open Systems International Inc.
I’m very fortunate to have had the pleasure of knowing Bahman Hoveida as both a colleague and a friend for many years. During that period I’ve watched with interest as the co-founder and president of Open Systems International (Minneapolis, MN), led his company from a relatively small niche player into a widely known and highly respected force in supervisory control and data acquisition (SCADA), energy management systems (EMS) and distribution management systems (DMS) on a global scale.
Today, as the entire automation/IT landscape is undergoing what is arguably a makeover of potentially epic proportions, EMS is once again being redefined, this time from an operational standpoint in response to new regulatory, security and reliability factors. Bahman, clearly among the foremost experts in this crucial automation/IT market component, was kind enough to talk with me about where he feels the EMS market is today and where it is headed. His views and insights will be invaluable to anyone involved in the market settlement process, whether from a hardware, software, systems or operations perspective.
- Mike Marullo, Automation/IT Leadership Series Editor
EET&D: Let’s begin by discussing the recent regulatory changes, such as the newly enacted FERC/NERC reliability rules and how they will affect the SCADA/EMS. Now that NERC has been given reliability enforcement authority, what role does automation play in that picture?
Hoveida: The new electric reliability standards contained in FERC’s Order 693 are now mandatory for all U.S. owners and operators and are intended to eventually become applicable to the balance of the North American bulk power system by future agreements with Canada and Mexico. After all, as NERC noted in its incident report, the failure of a SCADA/EMS system was partially responsible for the Northeast Blackout on August 14, 2003, precipitating the enactment of these mandatory reliability standards by FERC.
In my opinion, these new reliability standards will have an immense impact on the way utilities operate and will undoubtedly affect the real-time operation of the power grid. Reliable and robust automation systems are needed to effectively monitor and control the power system, optimize its reliability, and avert potentially catastrophic failures. NERC mandatory guidelines for reliable control systems; disaster recovery and emergency control systems; operator training simulators; and systems designed to be impervious to cyber attacks are a few examples of the impact these reliability standards will have on the automation systems deployed by utilities.
EET&D: As you know, NERC also recently issued new security guidelines as well. What impact do you expect these will have on the market and again, what is the role of automation in securing the grid against cyber attacks?
Hoveida: As a result of the NERC CIP/Cyber Security requirements, utilities are scrambling to audit and certify their automation systems. This is an effort comparable to the Y2K initiative of the 1990s, which has created a plethora of activities on the part of consultants, vendors, and utilities. Ultimately systems should be subject to independent audits to certify they meet the NERC standards.
The DOE, through a voluntary initiative between suppliers and the Idaho National Labs, has been providing independent testing and certification of SCADA/EMS platforms from various suppliers, including ours.In my opinion, such voluntary measures will become mandatory in the near future due to the immense impact a critical system failure could have on grid reliability and the economy.
EET&D: Are more or different types of EMS systems needed because of the new rules and/or other factors?
Hoveida: The majority of the EMS system architectures in operation were designed in the early 1990s or late 1980s. Although they may employ new hardware architecture they are somewhat archaic, highly complex, and hard to use and maintain. In my dealings with the industry I often come across utilities that are unable to use their EMS systems and applications effectively due to complexity.
For example, the Security Analysis applications mandated by NERC for bulk power operation are too complicated, slow, and in many cases are hard to maintain and keep up to date, rendering them virtually useless. How can utilities cope with these imminent reliability standards if their main EMS systems - which are intended to be their “Security Radar” - are not fully functional or cannot deliver the required guidance? Utilities need to press their suppliers for much more reliable systems, systems that are easy to maintain and upgrade; systems that can handle hundreds of thousands of events and alarms without failure; applications that work properly without a large technical maintenance burden.
Additionally, the next generation EMS systems will have to incorporate advanced features such as 3D data visualizations, use of phasor measurement units, dynamic security analysis, on-line voltage stability analysis, etc. Pushing these new frontiers is absolutely necessary if we are to achieve the security and reliability goals being set by regulators and promised to ratepayers.
EET&D: Does (or will) this mean that existing EMS systems are outdated and need to be upgraded or replaced?
Hoveida: Well, this depends on who you are and what system you are using. You would be surprised if you knew about the state of the systems used by some of the largest utilities in North America! Most of these systems are plagued with the same issues I mentioned in the previous question; namely reliability, maintainability, usability, and very high cost of maintenance.
Because of this cost burden and the cumbersome nature of software upgrades, these companies typically run on software releases that are at least five years old, or sometimes even older. In our customer community, we have been promoting annual upgrades and making sure the entire user base is on the latest releases of our software. We have been successful in cutting the time to perform an upgrade to 2-3 weeks as opposed to months, which seems to be the industry norm.
In addition to the very fundamental issue of technology obsolescence, I believe it is time for a new generation of more intelligent systems, operating platforms and application software that proactively manage grid reliability and cyber security issues. For example, one initiative we are currently working on is an intelligent RTU communications front-end that uses pattern recognition technology to validate a request for switching operation, which would not permit a potential cyber attacker to switch critical components, or drop substantial load or generation. This is just one example of new intelligent technology that should eventually be moved down to the RTU and IED level.
EET&D: A lot of people are wondering whether new ISO/RTO EMS systems will be replacing those at the individual utility level, or are systems needed at both levels?
Hoveida: I believe a hierarchical and multi-level system and control authority infrastructure is needed for enhanced reliability. The ISO/RTO needs to be responsible for the coordination of regional grid reliability via adoption of what I’ll call “macro strategies," while the individual utility/control area should remain responsible for the operation and security of their own internal area employing “micro strategies." In addition there is this notion of keeping the ISO and RTO honest! Most utilities want to be able to monitor their own transmission and ascertain the state of their systems independent of the ISO/RTO. This would be even more amplified now with FERC/NERC imposing stiff penalties for reliability infractions.
EMS systems are definitely needed at both levels, but given the complexity and size of the grid monitored by the ISO/RTO, in my opinion the power system solutions at the control area level would be much more reliable and robust. Time skew, outdated data models, and of course, the sheer size of the network models themselves always add uncertainty in any solution reached by an ISO/RTO. A case in point was the Midwest ISO operation during the August 2003 Blackout, which according to NERC’s report, could not ascertain an impending grid problem until it was too late.
EET&D: Where do you think we currently stand with regard to SMD (standard market design) now that the whole restructuring issue seems to have wound down?
Hoveida: Simplicity and transparency are essential to successful markets. If structured and implemented properly, FERC's standard market design proposal would lead to a robust open marketplace for affordable and reliable wholesale energy, as well as jump starting much needed investments by ensuring regulatory certainty for investors. Of course, it remains to be seen in the near term how effectively the SMD is implemented in various markets around the country.
I really think that the PJM experience is probably the closest to the SMD model. It is also one of the most successful market experiments in balancing equal open access, reliability, and predictable stable energy pricing. The counter to this, of course, is the California experiment, but it is understandable that initiatives of this complexity often must go through difficult and sometimes painful trials.
EET&D: Are there any new dimensions of market settlement either as a result of new reliability guidelines or SMD that you’d like to discuss?
Hoveida: Yes, I think that demand response must be an essential component of a well-functioning market. After all, we see very effective demand response programs implemented by smaller utilities, public power and cooperatives for mitigating resource availability, economics, or reliability issues. Historically however, electricity markets have encouraged or implemented relatively very little demand response by end users.
As a result, markets have been prone to wholesale price volatility, price spikes, system reliability issues and potential market abuses. Introducing demand response into wholesale markets would definitely enhance electricity market performance and grid reliability. Using voluntary price-based controls on consumers, congestions could be managed and ultimately demand side management programs could be employed to deal with and even avert potential grid imbalance contingencies.
EET&D: Are we likely to see the formation of more ISO/RTO organizations? What about ITCs?
Hoveida: This is a very good question! The new FERC SMD and mandatory reliability standards would have a dramatic affect on the landscape of the utilities, RTOs, ISOs and ITCs. I personally believe that there is potential for 2-3 new entities in the USA. The Southeastern, Northwestern, and Southwestern regions of the country have their unique energy and reliability demographics and may opt to develop their own focused Independent System Operators.
EET&D: Is it really feasible to strive for a grid that doesn’t experience major outages?
Hoveida: Yes, but only if we as consumers would be willing to pay $5.00 per kilowatt-hour for electric energy! Like anything else, there is a price for reliability and the real question is who will pay it. Foremost, it is a matter of adequacy of generation and transmission assets. Most major outages occur when a system is pushed to its limits. Lack of economic incentives and/or excessive regulatory challenges that discourage or prevent investing in generation and transmission expansion projects will always keep the complete achievement of reliability goals out of reach.
Nonetheless, a more pragmatic approach - let’s call it reaching for the low hanging fruit – is striving for “optimal reliability.” This means that taking into account the present condition of the power system along with its various constraints, still being able to achieve the highest level of reliability at any given time, or as a minimum, avert or mitigate any contingencies that could move the system away from this optimal reliability point.
EET&D: Are we spending too much time trying to predict/prevent major outages rather than focusing on rapid restoration as a primary means of mitigating the impact?
Hoveida: Our first and foremost efforts need to be focused on preventive measures. Strengthening the grid, optimally operating the grid, predicting failure modes, and devising remedial plans to deal with contingencies are the most important factors in achieving reliability. Naturally, we also have to be prepared to deal with the aftermath of an outage, and that is having robust and tested rapid restoration strategies in place.
EET&D: Whereas SCADA, EMS, DMS and many other parts of the utility automation/IT infrastructure were almost completely stand-alone not so very long ago, now almost everything is at least interconnected, if not interoperable and/or fully integrated. What are your thoughts about how this integration of systems, platforms and technologies should be planned and managed?
Hoveida: Full integration is a lofty goal for most utilities. In my opinion, this is not a technology problem anymore, as there are many standards and mechanisms to interface systems and exchange data appropriately and efficiently. The real issue, I believe, is a lack of a true vision and finding a strategic champion who stewards the implementation and integration of these systems into the business, and determines which data or systems to integrate. Automation systems are among the most critically important IT assets a utility has, but historically, many CEOs and CIOs have been uninformed or mal- informed about the existence, importance and criticality of these systems.
As a result, important decisions have been relegated down the organizational hierarchy to people guided mainly by a narrow view of - and focus on - technology rather than on the business itself. I am hopeful that the newly mandated reliability standards and the NERC cyber-security standards will begin to shine the light on these systems and that CEOs and CIOs will start paying more attention to automation/IT solutions and embrace the growing importance these systems have for the safe, reliable and efficient conduct of their enterprise. I think I can state fairly unequivocally that everyone involved in the automation business is hopeful that utilities will start allocating a larger portion of the resources that are routinely spent on ERP and corporate IT initiatives, on these increasingly critical automation systems. Between the declining infrastructure, aging workforce and rising costs, automation may not be a panacea, but it certainly represents a sound and proven investment strategy that works well and has countless financial and operational benefits when properly conceived, planned and deployed.
EET&D: I think we can probably agree that the EMS market area has proven to be a particularly challenging and sometimes rocky road for many. Indeed, only a handful of EMS suppliers survived the project downsizing trends in the 1980s – first with the introduction of workstations and shortly thereafter, PCs. Do you think there is another round of changes on the horizon, or will we see some stability for awhile?
Hoveida: Yes, I think we can expect more consolidation in the future. I also believe we will see some new players in the market, mainly by the acquisition of present suppliers. There seems to be a race by larger IT software companies to carve up a chunk of the utility software and services market, and the automation software and systems sector is definitely part of their strategy.
At the low end of the market there will be some consolidation as a result of technology obsolescence, lack of market success and stranded investments in research and development. I believe the more focused companies with innovative modern products, an agile management style and both a knack for and commitment to customer service will be the ones to thrive and succeed in this traditionally difficult part of the market.
EET&D: The fundamental shifts that have taken place over the past two decades in system topology and architecture totally redefined previously well-established technical and financial parameters for these projects and so-called ‘forklift replacements’ eventually gave way to incrementally scalable architectures, which OSI was among the first to embrace. How do you see SCADA/EMS changing over say, the next 3-5 years?
Hoveida: As the technologies have matured, it is my view that this has steadily become more of a customer service business. Any company that can remain very responsive to the needs of its utility customers in the face of an ever-changing electric utility marketplace will flourish. Of course, an integral part of good customer service methodology is having technically innovative and modern solutions, a reliable and proven delivery track record, etc.
A large part of our success at OSI has been the promotion of this customer–focused philosophy as our number one priority and main mission. I am sad to say this focus has been substantially lost in most American businesses today as compared to say, 20 or 30 years ago. I’m sure most consumers would agree that quality of service and a total positive customer experience have become things of the past for most businesses today.
EET&D: What do you think the utility market itself will look like five years from now, in terms of structure, composition and reliability?
Hoveida: It is hard to imagine, but given the trends in the past few years there will likely be more consolidation in the regional markets forming mega-utilities. There will be at least one or two new ISOs and several new ITCs.
A case in point might be the state of the commercial airline industry following de- regulation. We have far fewer carriers, but whether the result is better, safer, more affordable or more reliable is certainly a cause for debate.
Indeed, the huge appetite among larger companies for more consolidation is frequently driven solely by their quest for market dominance. And, while there is certainly something to be said for size in certain situations, I have my doubts that sheer size, bureaucracy and the traditional conservatism of industry behemoths will allow them to embrace and take advantage of newer technologies or initiatives.
On the contrary, I believe that many will continue with business as usual and keep pushing the reliability and regulatory bounds to their limits. I am very optimistic, however, about the role of Public Power in the future. There are countless examples of innovative and forward-thinking initiatives that have and will continue to originate from this part of the industry.
EET&D: So, as you gaze into your crystal ball, what predictions can you offer regarding what we can expect in the future?
Hoveida: Well, I am not a betting man, but I’ll be happy to offer a couple of general predictions for your readers to ponder. First off, I think the Smart Grid is quite achievable – in a prototype form and on a limited scale - over the next 5-7 years. However, my view is that this will most likely be achieved by a progressive power cooperative somewhere in the rural heartland rather than by a very large investor-owned utility, contrary to what many might otherwise expect.
Secondly, each operating entity has a critical mass in terms of grid or market size, which once achieved, will be very difficult to expand without jeopardizing grid security and reliability. Therefore, it’s probably not unrealistic to predict the breakup of larger RTO/ISOs into smaller entities in the next 5-10 years. The fact is, smaller entities may eventually be better equipped to deal more nimbly with the evolving complexity of the grid and related markets.
Yet another scenario could have the ISO acting as not only the market/system operator, but also having jurisdiction over a number of smaller sub-ITCs, each being responsible for regional transmission operations. Of course, as a system supplier, we are very much encouraged by the opportunities that lie ahead; especially with regard to the impending technology needs of the utilities we serve.
EET&D: Yes, it will indeed be interesting to see how these things evolve over the next 12-18 months, and I hope that we’ll have a chance to revisit some of these issues with you again soon. Meanwhile, one thing we can both be sure of is that there will be changes, although what those changes are and how they will manifest themselves remains to be seen.
