Timely availability, reliability, and longevity of appropriate quality materials and supplies are critical to the well being of any AMI (Advanced Metering Infrastructure) or Smart Grid project. In the past, utilities have generally taken for granted that trusted suppliers would provide needed equipment, materials, and services in a timely and high-quality fashion.
Today, however, these suppliers are facing unprecedented demand, not only from the United States but also from utilities within Europe, China, and India. Accordingly, globalization of suppliers is occurring at an accelerating rate and many manufacturers and service companies will be facing the prospects of capacity constraints. While, in general, metrology vendors tend to be more established and mature businesses, many more AMI communications suppliers tend to be newer, smaller, and generally unproven companies with technologies that are in early stages of evolution. This has the potential to lead to further concerns regarding product availability and quality levels.
It is not unusual for items purchased from a local supplier to be manufactured in one foreign country with raw materials sourced from yet another. Given the increased level of capital requirements for AMI and Smart Grid mass-deployments today, the need to ensure high levels of reliability and to limit potential equipment failure has become increasingly important. To ensure financial success of an AMI or Smart Grid project, it is imperative that utilities planning or embarking on project deployment have thoroughly evaluated their suppliers throughout the supply chain processes to assess product design, operating performance, quality management systems, and delivery capabilities. Further, utilities considering AMI deployment are well advised to develop sufficient documentation outlining the standards, processes, and resource requirements to implement an end-to-end quality assurance (QA) program, from project conception to full deployment.
A thorough QA evaluation of potential vendors or supply chain partners should, at a minimum, include the following components:
• Equipment specification and design review
• Risk-based assessment of a supplier’s manufacturing and delivery capability
• Assessment of a service provider’s quality management systems and performance
• Ongoing performance reviews to assure continuing quality and on-time delivery
• Monitoring of performance metrics to identify early warning signs, should a supplier be faced with limited delivery capability or declining product performance quality
Beyond Product Testing
Some utilities may focus exclusively or too heavily on product testing when considering a possible vendor, often relying exclusively on random sampling methods. Certainly, meter testing and certification is a foundational step in determining whether a vendor can meet the specifications of a project, but there other equally important components that must be considered when making a QA determination. In fact, experience accumulated in the growing AMI/Smart Grid sector suggests that vendor evaluation and QA review should be approached from two distinct vantage points: AMI system technical requirements measurements (standard controls) and vendor risk profiling.
Even with standard controls, some utilities may not be applying sufficient processes to evaluate potential vendors and their solutions. Many new and next generation technologies are often associated with new and untested companies. In many circumstances, these attractive and promising new technologies may fit the technical requirements of a project and purchaser, but the company providing the technology may have an associated increased risk as a smaller firm with limited funding and minimal proven track record for scaling up design, manufacturing, service and delivery.
While minimum standard controls require meeting applicable ANSI, IEC and/or ISO standards for health and safety, data protection, data management, and other key factors, often utilities with limited resources are unable to conduct a more extensive due diligence. The multitude of subcontractors or original equipment manufacturers that a
vendor may use can also create increased levels of risk in which a utility enters into lengthy and expensive contracts based on a cursory QA review. A utility entering into a contract with a particular meter manufacturer should be well-informed and have relevant supplier information at its disposal (e.g., patent ownership, on-going legal proceedings, supply chain contract details) and, if it is unable to conduct an evaluation beyond the minimum standards, it should seek out external assistance.
The risk-profiling component of the QA assessment should include on-site process audits performed for each vendor prior to initial production runs to ensure that vendors can comply with the utility-specific design, manufacturing, and delivery requirements. Physical inspections performed both at the vendor manufacturing site and the utility central receiving station should also be included, along with diagnostic tests performed on-site with remote equipment. Lab tests – including meter type tests and reliability (or life expectancy) testing – should take place prior to shipment or upon receipt of the equipment at the utility receiving unit, based on varying sample levels commensurate with prior results. Meter type testing should include functionality, environmental, and safety testing, as well as targeted testing of embedded components, such as integrated latching relay (remote reconnect/disconnect) switch testing.
Specific components that are candidates for testing would also include, but not be limited to:
• Mechanical requirements
• Insulation properties
• Accuracy requirements
• Electrical requirements
• Electromagnetic compatibility
• Effects of climatic environments
Development of a statistical system in which the meters are divided into groups (meters with the same metrological characteristics) will also provide longer-term analysis of key trends and potential predicative failure analyses, using random checking of each group on a periodic basis.
Vendor Risk Profiling
A utility may be able to perform adequate due diligence on the technical aspects of the product and yet find that the biggest risk is not in the technology itself but in the perceived – and often unknown – capability and capacity of the supplier to successfully scale and deliver the product. When considering acquisition of new technologies for large scale deployment in mission critical systems, the most significant purchase and project risk is likely the failure of the supplier(s) to be able to scale up processes and deliver the required product volume on the schedule and at the level of quality required. Traditional due diligence processes may not be able to adequately assess the capability and the subsequent risk for companies in these early stages. Put another way, the technical product due diligence process will provide some insight into basic capabilities of a potential vendor, but a utility would be shortsighted to make a selection decision based solely on technical capability.
Assuming that a company has demonstrated that their technology is solid and meets the business requirements, the challenge is developing a due diligence effort to appropriately assess the company’s projected capabilities to deliver – sometimes in a situation where no track record or minimal demonstrated capability exists. This process therefore is fundamentally an assessment and “educated best guess” measure of “what will be” as opposed to the more traditional due diligence approach which focuses more on “what is.” Utilities are finding that it is useful to employ external expertise to provide direct and independent validation of the capabilities espoused by vendors in their procurement responses.
A suggested first step is to classify all potential vendors into standard categories of “start up or emerging,” “growing,” and “mature.” Characterization would generally follow a previously established set of criteria roughly defining the various business growth stages. The characterization helps to assure that the appropriate assessment criteria are applied.
For instance, start-up or emerging stage companies are not likely to be able to demonstrate the same financial strength or proven manufacturing capability that a mature stage company could provide.
Placing a large order with an emerging or early growth stage company is essentially an “investment” in that company. In most cases, a large order will provide the cash infusion needed for a small company to begin the process of scaling up to deliver design, manufacturing, quality assurance and delivery systems, some of which may not even exist at the time the order is placed. Given this – albeit simplified – similarity, the appropriate due diligence process used to screen an emerging or early growth stage company might be similar to the due diligence processes used by venture capitalists (VC) investors when considering investments in emerging or early growth stage companies.
There are a set of fundamental screening questions and criteria that VCs may typically use to evaluate a company’s potential to succeed. These criteria go beyond the screening and assessment of the merits of the technology, product or service offering. A conceptual framework is typically built around investigating potential vendors within seven major focus areas, including some of the following:
• Company business plan
• Company basic financial performance and structure
• Management team, intellectual property, inventory and protections
• Current product status, sales and deployment
• Product development roadmap
• Scalability (including design, manufacturing, QA, sales, delivery and support)
• Investment plan
In many cases, a vendor’s initial or subsequent business plan can provide the basis for a number of these requested areas. More mature companies could be asked to provide current and three-year financial reports (e.g., income statement, balance sheet, capitalization/debt structure); one-, three- and five-year budget projections, including capital requirements to meet sales projections; and other applicable financial records.
Other relevant financial areas for exploration could include the following:
• Insurance and bonding instruments are sufficient to protect and cover both physical
and financial assets
• Credit ratings are at investment grade or better (S&P ratings are at BBB- or better
(up to AAA) or Moody’s ratings are at Baa3 or better (up to Aaa))
• Current funding (capital) sources are sufficient to cover product development
roadmap efforts for the next three years
• Analyst coverage (if applicable) deems company a positive performer or better,
or identifies no significant impediments to meeting financial targets and growth objectives
• Documentation exists to demonstrate compliance to Sarbanes-Oxley financial controls,
if applicable
• Business qualifications, standards and ownership requirements (e.g., MWBE,
US Corporations/foreign exclusion)
• Minimum levels of financial capability or damage provisions
• Bonding, collateralization, and security requirements
• Contract exception policies
In closing, there are various risk assessment and characterization criteria that should be considered as suitable for increased levels of due diligence that would support the needs of utility QA processes. Of course, the unique aspects of each AMI/Smart Grid deployment warrant a customized QA process that use these general touch points as a foundation.
Representative Questions to Ask Of Each Vendor During an On-Site Risk Profile:
• How well defined are processes to determine and review AMI requirements as they are established, and to track and ensure that each requirement is met, each requirement is traceable from source to component? Does this system effectively address configuration management concerns as architecture evolves and when requirements change?
• How well does the organization have rigorous and forward-looking program management capabilities that will ensure that AMI timeline requirements are met, and that risks to schedule are identified, communicated, and mitigated in a timely manner?
• How well does the organization have the resources and capabilities to effectively address the issues raised during this assessment?
• How well can the organization effectively collaborate with the utility and with other AMI vendors and with supply chain sub-systems on the AMI development?
• How well does the organization have experience in development of open architecture systems and have they demonstrated this through effective application of technical and regulatory standards, and through involvement in relevant technical consortiums?
• How well does the organization have formal and effective processes and systems for cost analysis and control, and do these systems encompass all cost sources throughout all phases of the AMI program?
• How effectively will the vendor be able to meet product reliability requirements established by the utility AMI team?
• How effectively will the vendor be able to meet product performance requirements established by the utility AMI team?
• How effectively will the vendor be able to meet system security requirements established by the utility AMI team?
About the Author
Will McNamara is a Principal Consultant at KEMA. He is a regulatory and legislative affairs expert with 15 years of energy industry policy-making, rate design, expert testimony, and lobbying experience. His expertise includes developing AMI policy and managing business plans and regulatory filings within the areas of energy efficiency, demand response, and smart grids.
