Module 9

Mini-grid Performance Monitoring

April 2020
Module Overview

Introduction to the Module

• Learning objectives
• Module Requirements
• Agenda

Topic 1 Topic 2 Topic 3 Topic 4 Topic 5

Introduction Data Collection Analysis Performance Reporting for
Indicators Mini-grids

End of Module
• Module recap
• Further reading

2
Objectives & Requirements
Learning Objectives Module Requirements
§ Learn about the process of mini-grid § This module is targeted at mini-grid
performance monitoring. developers and operators at all stages of
§ Understand the type of data to collect development.
and the different collection methods. § They are expected to have a basic
§ Learn about the analysis tools available understanding of rural, off-grid energy
to developers, including the Mini-grid markets and community dynamics in
Quality Assurance Framework. developing countries.
§ Learn about the three most important
§ No prior knowledge of performance
areas for mini-grid reporting and the
key performance indicators within monitoring is required.
them.

3
 Agenda
1. Introduction 4. Performance indicators
§ Definition § Commercial & financial
§ Benefits of Performance Monitoring § Customer & utility accountability
§ Process overview § Technical
2. Data Collection 5. Reporting for Mini-grids
§ Data collection overview
§ Data classes
§ Automated vs. manual data collection
§ Data collection tools
3. Analysis
§ Data analysis tools
§ Quality Assurance Framework (QAF)
§ QAF Service Levels
§ Analysis of socio-economic data

4
 Introduction

Definition

Performance monitoring is the process of tracking technical,

business, and social metrics for a mini-grid, and providing this

information to all stakeholders to be measured against relevant

performance agreements, standards, or specifications.

5
 Introduction

Benefits of Performance Monitoring

Developers Regulators & Policy Makers
• Improve demand forecasting for existing and • Assess better the short and long-term energy needs
future micro-grids. of a community or region.
• Better understand energy needs and growth • Standardise system performance and services across
opportunities. installations and ensure regulatory compliance.
• Learn about impacts of regulation and if any
• Build trust with customers and evaluating modifications to the regulatory structure are
customer satisfaction. necessary.
• Maintain customer service levels.
• Optimise operations by reducing O&M costs, Investors & Donors
improving revenue collection, and reducing
system losses. • Report and document business models, financial
sustainability, and returns.
• Improve troubleshooting of technical system
• Measure socio-economic impacts of mini-grids on
issues and failures. the local community.
• Monitor and mitigate against safety concerns. • Better understand risks and risk mitigation
opportunities.

6
 Introduction

Process Overview
• Define goals & benchmarks: Define goals and target benchmarks that the project is
Define goals &
seeking to achieve. benchmarks

• Define requirements: Identify the data required to evaluate these benchmarks, and
the method used to collect the data. Different goals will require different
Define
measurements or methodologies. Implementation
requirements

• Data collection: Use automated or manual methods as defined in the previous stage.
Timely and accurate collection of data is critical to performance monitoring. Quality
and relevant data is vital for data analysis.
• Data analysis: Verify and analyse the raw data to determine potential improvements
to the mini-grid. Developing data analysis tools at an early stage in a project enables
efficient and immediate analysis throughout the project. Analysis tools should be
Information
automated as much as possible and include an initial quality control to identify flaws distribution
Data collection
or errors in the data.
• Evaluation & distribution: Evaluate potential improvements to determine if they are
viable for the mini-grid’s environment and resources. Communicate results of the
evaluation to stakeholders for their approval.
• Implementation & verification: Implement corrective actions or system Evaluation Data analysis
improvements that are necessary and cost effective. The actual impact of these
improvements must be evaluated against the original benchmarks by a continued
process of data collection and analysis.

7
 Data Collection

Data Collection Overview

Methodology
• Type of Data
• Collection Method
• Automated data recording instruments such as energy management systems, inverters, or smart meters.
• Manual / hand-held devices.
• Frequency of collection (hourly, daily, monthly, quarterly, semi-annually, annually)

Resources
• Some developers require a lot of time and resources to manage data collection, but with well-designed collection infrastructure
(e.g. smart meters) the burden can be reduced.
• Decide which measurement system to use during the development phase of the project so that the appropriate technology can
be costed, procured, and installed.

Impact
• Collect data to measure two types of impact: economic and social.
• Measuring the social impact of a project is more difficult, but developers, investors and regulators will often all want to
understand this.
• There are no industry standard frameworks or indicators for social impact for developers to use.

8
 Data Collection

Data Classes
There are 3 “classes” of data:
Remote Monitoring Data Socio-Economic Data Commercial & Financial Data

Smart meter and plant-side data collected Customer feedback: Commercial & financial data monitoring performance of
monthly containing: • Customer complaints and the business:
• Energy consumption data: if possible on satisfaction. • Financial information on past and projected CAPEX
hourly basis • Surveys on demographic, and OPEX costs (cost of connection, cost of power,
• Energy payment data: number of household, and income payroll, etc.).
customer payments and amounts information, which is • Information on projected revenues and consumption
• Power quality (AC / DC): voltage stability, conducted by (number of customers, of new connections, of
ripples & switching noise, transients, faults enumerators on the unconnected customers). Information on tariff
/ day ground via mobile structure.
• Power reliability: SAIFI, SAIDI (see notes) surveying applications. • Information on mini-grid sizing (total losses).

It is important to capture all these classes because they give a different perspective on mini-grid performance.
Sources: Quality Assurance Framework for Mini-Grids, NREL, U.S. Department of Energy, Nov 2016
Mini-Grids Performance Monitoring work done by E4I & NREL as part of the Power Africa Beyond the Grid programme

9
 Data Collection

Automated vs. Manual Data Collection

Automated Manual
• Continuous recording of data with high flexibility of • Low up-front and maintenance costs
recording rate • Allows for flexibility in the data collection process and
• Less labour intensive and time consuming parameters recorded over time
Pros

• Generally more accurate • Qualitative data can be collected

• Can collect data at any level depending on equipment used
(e.g. at the customer, feeder, or plant level)
• High up-front cost • Time consuming
• Expensive to maintain and replace • Labour intensive
• Requires a high level of technical skills to install and • Requires data entry
Cons

maintain • Can be inaccurate due to human error in data collection

• Focus typically on quantitative data and entry
• Can require a robust mobile network to transmit data

10
 Data Collection

Data Collection Tools

Automated
• Smart meter
• Data about electricity supplied to each connection.
• Inverter data
• Generation data: voltage, energy, power, battery charging / discharging and the amount of energy evacuated.
• Energy management systems
• Same capabilities as an inverter, with some more advanced features: remote control of components of the mini-grid
and demand prediction algorithms.
• Automated SMS feedback surveys
• Surveys to gauge customer satisfaction in areas of the mini-grid service.

Manual
• Qualitative / subjective data collection gathered from:
• Face-to-face surveys
• Phone Interviews
• Other customer feedback and staff

11
 Analysis

Data Analysis Tools

• There are currently no standard data analysis tools for the mini-grid industry,
but several companies are working on components of this.
• These companies include Spark Meter, Odyssey Energy, and a spin-off from Rafiki
Power based on their Asset Management and Monitoring Platform (AMMP)
technology.
• There are generic customer relationship management (CRM) tools which can
be adapted to a developer’s needs. These can incorporate payment systems
(such as the Angaza Technologies platform) and SMS aggregators for bulk-
sending of SMS messages.
• Analysis is often done on an adhoc basis. Developers need support, training and
tools to create standards aligned with reporting requirements.

12
 Analysis

Quality Assurance Framework (QAF)

A mini-grid’s technical performance can be measured using NREL's Mini-grid QAF.

The QAF evaluates performance against the service level of power supplied. Level of service measured by

• Quality of power: Voltage imbalances, transients, frequency variations, DC ripple, etc.

• Availability of power: Maximum power draw, amount of energy available and duration of daily service.
• Reliability of power: how consistently the power system provides power (SAIFI & SAIDI).
QAF service levels are based on international standards and typical power requirements of different
electrical devices. These levels range between:

• High (highest number): parity with the grid.

• Basic (level 1): safety-based minimum.

Source: Baring-Gould, I. et al 2016 & Booth, S. et al 2019

13
 Analysis

QAF Service Levels

Here is an example showing the service level definitions for peak available power, which
is a measure of power availability.

For more information on electricity service levels for quality, availability and
reliability, please see Chapter 3 of Baring-Gould, I. et. al. (2016).

14
 Analysis

Analysis of Socio-economic Data

To assess the impact of testing and changes to a mini-grid’s business model, standard statistical study designs and
analyses should be used. This will enable the developer to understand the effect of new strategies (“treatments”) on
the mini-grid customers, without hidden biases skewing the results.

Study designs: Useful analysis techniques:

1. Randomised control trial 1. Propensity Score Matching (PSM)
• Control and treatment groups are assigned randomly. While • Used to match members of a treatment group with a
this is the gold standard for study design, it can sometimes suitable control group member. PSM matches treatment
be impractical / unethical to randomly assign treatments to samples to the control samples with the closest propensity
a population. score I.e estimated probability that the sample has received
a treatment.
2. Encouragement design
2. Difference in Difference
• Randomises likeliness of taking up treatment if
randomisation of treatment access is not possible. • Studies the differential effect of a treatment between a
treatment and a control group.
3. Natural experiment
• Individuals are exposed to treatment based on natural
conditions and not randomisation.

15
 Performance Indicators

Performance Indicators
Most performance monitoring happens during the operation and
maintenance phase of a project.

O&M performance monitoring can be split into three main functional

areas:

• Commercial and financial monitoring

• Customer and utility accountability and demand monitoring
• Power quality, reliability, and availability monitoring (QAF - technical)

16
 Performance Indicators

Commercial & Financial Indicators (1)

Key indicators: Customer numbers

Indicator Units
Number of customers # (number)
New connections # / month
Potential unconnected customers #
Expired connections #
Metered customers #
Monthly payment collections %
(number of customers who payed / total number of customers)
Total electrification (number of customers / total population in %
reach)

17
 Performance Indicators

Commercial & Financial Indicators (2)

Key indicators: Revenues & Costs

Indicator Units
Revenue USD1 / month
Other revenues (e.g. from other services offered by the project, USD / month
connection fees, wiring fees, etc.)2
Administrative and/or distribution cost per connection USD / connection
Cost of power generation, distribution and administration USD / kWh
(disaggregated)
Total cost of power USD / kWh
Total fuel costs (differentiating fuel costs and fuel USD / month;
transportation costs) USD / litre

1 When using currencies, any currency can be used. It is advisable, however, to

use one that is stable so that trends are not hidden by currency fluctuations.
2 Revenue data should be disaggregated by service level and sector.
18
 Performance Indicators

Commercial & Financial Indicators (3)

Key indicators: Power efficiency & contractual

Indicator Units
Fuel efficiency (to assess fuel loss and generator operational gallons / kWh
efficiency)
Total losses (kWh generated / kWh sold) %
Number and type of suppliers or contractors #
Performance of suppliers or contractors description
Payroll (disaggregated by team) USD
Team responsibilities description

19
 Performance Indicators

Customer & Utility Accountability Indicators (1)

Key indicators: Performance, complaints and safety
Indicator Units
Performance of the project against the customer service level description
agreement
Annual electricity production kWh
Renewable energy contribution (renewable kWh / total kWh) %
Duration of daily service hours / day
Number of complaints #
Number of consumer safety incidents # / month
Number of health and safety incidents (including near misses) # / month
H&S report (identifying the incident or near miss, the staff / team report
involved, the root cause of the incident, and the corrective action taken)

20
 Performance Indicators

Customer & Utility Accountability Indicators (2)

Key indicators: Connections & community

Indicator Units
Power consumption (disaggregated by customer, service level, kWh / month
and type)
Number of new connections # / month
Number of disconnections # / month
Reasons for connection or disconnection description
Number and type of outreach activities #
Percentage of the community connected %
Increases in the community’s participation in local industry %, by value chain
value chains

21
 Performance Indicators

Customer & Utility Accountability Indicators (3)

Key indicators: Services, appliances and other

Indicator Units
Increases in the number of products and services available in # & types
the local community
Sample appliance ownership by customer # & appliance
Customer satisfaction with various aspects of mini-grid service Satisfaction score (1-
5), with reasons
Training methodologies description
Customer training on energy usage # trained

22
 Performance Indicators

Technical Indicators (1)

Key indicators: Power quality and voltage Key indicators: Reliability

Indicator Units Indicator Units

Average power kW Planned power outages #
Maximum power kW Unplanned power outages #
Average power factor kW / kVA Length of power outage hours & minutes
Energy generated kWh Average hours in a day hours / day
Amount of fuel used Litre that power is available

Voltage imbalance % SAIFI* (planned & Total number of customer

unplanned) interruptions / total
Transients # number of customers
Voltage variations # / day SAIDI** (planned & Total minutes of customer
Frequency variation Avg. deviation (Hz) / time unplanned) interruptions / total
number of customers
DC ripple %
*SAIFI - System Average Interruption Frequency Index
**SAIDI - System Average Interruption Duration Index
23
 Performance Indicators

Technical Indicators (2)

Key indicators: Battery performance Key indicators: Event recording

Indicator Units For each key event (e.g. power outage /
voltage variation / etc.) record:
Battery voltage Volt
Indicator Units
Current in & out Ampere
Type of event predefined category
Battery (charge) %. Ratio of charge that can be
efficiency extracted from a battery, compared Cause of event predefined category
to the amount that has gone in Outcome of event predefined category
during charging
Date & time date & time
Battery temperature Degrees
Duration of event time
Cycle count Unit

24
 Reporting

Reporting for Mini-Grids

It is important for mini-grids to periodically report on their performance – often this is
required by donors or investors.

Business Reporting: (usually quarterly) Technical Reporting: (usually monthly)

Operational performance, financial Power quality and reliability, energy
performance, and growth potential: production and consumption, generation
• Total revenues sources, and system efficiencies, KPIs:
• Number of customers, customer by level • Annual electricity production and sales
of service, by sector • System losses
• Payment collection • Renewable energy contribution
• Revenues from other services • Number of outages
• Electrification rate • O&M events
• Operating costs • Power quality events

25
Module recap
• Performance monitoring is important for all stakeholders: the mini-grid operator, their
customers, regulators & policy makers, and investors & donors.
• Performance monitoring is a continuous process. Strategies are implemented, their impact is
analysed, then strategies are updated, and the process starts again.
• Developers can collect data on the ground e.g socio-economic data or remotely e.g on
consumption and payments.
• There are no industry standard data analysis tools, but some companies are offering
platforms for such work.
• Developers report on 3 main areas of performance: customer demand/utility accountability;
commercial/financial; and technical performance.
• Technical reporting can be done using the Quality Assurance Framework (QAF) which
measures services levels based on quality, availability and reliability of power supply.

26
Further Reading
Booth, S.; Li, X.; Esterly, S.; Baring-Gould, I.; Clowes, J.; Weston, P.; Shukla, P.; Thacker, J. & Jacquiau-Chamski, A. (2019). Performance Monitoring of African Micro-grids: Good
Practices and Operational Data. National Renewable Energy Lab.(NREL), USA; Energy 4 Impact, Kenya; Spark Meter International, Kenya. Not published as of writing this
module.

Baring-Gould, I., Burman, K., Singh, M., Esterly, S., Mutiso, R., & McGregor, C. (2016). Quality assurance framework for mini-grids (No. NREL/TP-5000-67374). National Renewable
Energy Lab.(NREL), Golden, CO (United States). https://www.nrel.gov/docs/fy17osti/67374.pdf

Androsch, H., Foster, R., Orozco, R., Isiolaotan, O., Ramasubramanian, V., Kabir, H., & Dr. Staedter, H. (2017). Mini-Grid Design - Focus on Solar Photovoltaic and Micro Hydro.
Abuja: Deutsche Gesellschaft fuer Internationale Zusammenarbeit (GIZ) GmbH. https://energypedia.info/images/a/a8/Mini-Grid_Design-Training_Handbook-
_Nigeria_2017.pdf

ESMAP Technical Paper. (September 2000). Mini-Grid Design Manual. Washington DC: The World Bank. https://www.esmap.org/node/1009

Garside, B., & Wykes, S. (December 2017). Planning pro-poor energy services for maximum impact: The Energy Delivery Model Toolkit. iied, CAFOD.
https://pubs.iied.org/pdfs/16638IIED.pdf

Green Mini-grid Help Desk (2019). Help Desk For Developers And Operators. African Development Bank, Energy 4 Impact, Inensus. https://greenminigrid.afdb.org/help-desk-
developers-and-operators

Gollwitzer, L., & Cloke, J. (May 2018). Briefing Paper 1 - Lessons From Collective Action For The Local Governance of Mini-Grids For Pro-Poor Electricity Access. Low Carbon For
Energy Development Network. https://www.gov.uk/dfid-research-outputs/lessons-from-collective-action-for-the-local-governance-of-mini-grids-for-pro-poor-electricity-
access-briefing-paper-1

27