Indicators and Calculating

Coverage indicators
M&E Indicators

For Malaria Programs

M&E Indicators: Module Objectives

At the end of the session, participants will be able to:

1. Critique indicators
2. Identify criteria for selection of sound
indicators
3. Understand how indicators are linked to the
frameworks covered in the Frameworks
Module
4. Select indicators and complete an Indicator
Reference Sheet
 What is an Indicator?

An Indicator is…
– a variable
– that measures
– one aspect of a program/project or health
outcome

 An appropriate set of indicators includes at least one

indicator for each significant aspect of the program or
project (i.e. at least one per box in an M&E
framework)
Anatomy of an Indicator Metric

 Proportion of Households with at Least One ITN*

– Numerator: Number of households surveyed with at least

one ITN.
– Denominator: Total number of households surveyed.

*An ITN is 1) a factory treated net that does not require any treatment, 2) a
pretreated net obtained within the past 12 months, or 3) a net that has been
soaked with insecticide within the past 12 months.
 Common Indicator Metrics
 Counts
– Number of providers trained
– Number of ITNs distributed

 Calculations: percentages, rates, ratios

– % of facilities with trained provider
– Under 5 mortality rate, case fatality rate, annual blood examination rate (ABER)

 Index, composite measures

– Quality index comprising the sum of scores on
six quality outcome indicators
– DALY
– Wealth index

 Thresholds
– Presence, absence
– Pre-determined level or standard
– Cut-off point
Characteristics of Good Indicators

 Valid: accurate measure of a behavior, practice or task

 Reliable: consistently measurable in the same way by different
observers

 Precise: operationally defined in clear terms

 Measurable: quantifiable using available tools
and methods

 Timely: provides a measurement at time intervals relevant and

appropriate in terms of program goals and activities

 Programmatically important: linked to a public health impact

or to achieving the objectives that are needed for impact
Characteristics of Good Indicators: Valid

Accurate measure of a behavior, practice or task

• Indicator measures what it is supposed to

measure
– Direct measures
– Indirect/Proxy measures
– Straightforward interpretation: change in value
signals a change in focal concept or behavior
 Validity: Class Activity

1. Is parasitemia a valid measure of morbidity?

2. Is fever a valid measure for malaria?

3. Is parasite testing a valid measure for parasite

prevalence?

4. Is the number of people reached by BCC

campaigns a valid measure of malaria
knowledge?
 Characteristics of Good Indicators:
Reliable
Consistently measurable in the same way by different
observers

 Types of measurement error

– Sampling Error: over-representation of urban populations
because access is easier
– Non-Sampling Error: survey estimates of bed net use, due
to response bias
– Subjective Measurement: indicators that ask for personal
judgment such as “quality,” “environment” and “progress”
Characteristics of Good Indicators: Precise

Operationally defined in clear terms

Activity: Develop definitions for:

 Effective treatment
 Population at-risk
 Suspected cases of malaria
Characteristics of Good Indicators:
Measurable
Quantifiable using available tools and methods

 Are the following indicators measurable?

1. Number of ITNs distributed

2. Compliance to antimalarial treatment
3. Anemia
4. Parasitemia
Characteristics of Good Indicators: Timely

Provides a measurement over periods of time of interest

with data available for all appropriate intervals

 Timeliness Considerations
– Reporting schedules
– Recall periods
– Survey schedules
– Length of time over which change can be
detected
 Characteristics of Good Indicators:
Programmatically Important
Linked to a public health impact or to achieving the
objectives needed for impact
 Are the following indicators programmatically important?

 Example 1: ITN distribution program

– Indicator: # ITNs distributed in past quarter

 Example 2: Program to increase access to ACTs through

community-based health workers
– Indicator: Number of ACT sales points with
antimalarial drugs
 Factors to Consider When
Selecting Indicators
 Link to framework
 Programmatic needs/information for
decision making
 Resources (Time)
 External requirements (government, health
partner, headquarters)
 Data availability
 Standardized indicators
 Operationalizing Indicators
Establish exactly how a given concept / behavior will
be measured
– Precise definition and metric
– How the value will be reliably calculated
– Anyone using the same data will arrive at exactly the same
indicator value
– Challenges
• Subjective judgment
• Local conditions
• Unclear yardsticks
• Skills of the users
 Sources of Indicators:
Using Pre-Defined Indicators
What are some sources for pre-defined
indicators?
– Past years of the program
– Related or similar programs
– Lists of global or recommended indicators
• Roll Back Malaria. Guidelines for Core Population-based
Indicators, 2009
• Global Fund Indicator Guide
• Global Fund performance framework
 Indicator Pyramid

Decreases
Global
Compare countries
Overview world-wide situation Number of
Indicators
National/Sub-national
Assess effectiveness of response
Increases
Reflect goals/objectives of national/
sub-national response

District or Facility
Identify progress, problems, and challenges
 Indicator Matrix

Data Source Frequency Level Decision

points/comments
Output
Number of health Program Quarterly Facility Disaggregate by
personnel and Records district
community health
care agents
trained in case
management
Outcome
Proportion of Representative Periodic National To be used to
children under 5 household (Every 1-5 determine where to
years old who survey (ex. years) target ITN
slept under an ITN DHS, MICS, distribution and
the previous night MIS) BCC activities
 Indicator Reference Sheet

Compile detailed documentation for each indicator:

 Basic information
 Description
 Plans for data collection
 Plans for data analysis, reporting, and review
 Data quality issues
 Performance data table (baseline and targets)
Indicator Reference Sheet: Example
Name of Indicator
DESCRIPTION
Rationale

Definition of the Indicator:

 Numerator:
 Denominator:

Measurement:
Frequency:
Interpretation:
Data Source(s):
Strengths:
Limitations:
THIS SHEET LAST UPDATED ON: 07/18/2011
 Indicator Strengths & Limitations

All indicators have limitations, even those

commonly used:

• Household spraying, net impregnation: Recall bias this can

result in considerable ‘heaping’ of dates

• Net Use: self-reporting bias, seasonality of survey may affect

net use
Indicators:
How they link to frameworks
 Logic Model Indicators

INPUT PROCESS OUTPUT OUTCOME IMPACT

• Human and • Establish • ITNs sold •Use of ITNs • Prevalence
financial distribution and of malaria
resources points for ITNs distributed
•ITNs
Indicator:
Number of ITNs sold and Indicator:
distributed Prevalence of
malaria
Indicator: parasite
Proportion of household infection
members who slept
under an ITN the previous
night
 Results Framework Indicators

IR-1: Improved Number of malaria cases

malaria prevention

IR-1.1: Access to and Proportion of household members who slept

coverage by ITNs increased under an ITN the previous night

Proportion of women who received IPT during

IR-1.2: Improved antenatal care visits during their last
coverage of IPTs pregnancy

IR-1.3: IRS coverage Proportion of households that received spraying

increased through an IRS campaign within the last 12
months
Setting Indicator Targets: Useful
Information Sources
• Past trends
• Client expectations
• Donor expectations
• Expert opinion
• Research findings
• What has been accomplished elsewhere
• International conventions
 Common Pitfalls in Indicator
Selection
• Indicators not linked to program activities
• Poorly defined indicators
• Indicators that do not currently exist and cannot
realistically be collected
• Process indicators to measure outcomes &
impacts
• Indicators that are not very sensitive to change
• Too many indicators
Pitfalls with Selecting Indicators

Indicator not linked to program activities

 IR: Expanded access to malaria treatment services
 Activities: train providers in current clinical
protocols
 Inappropriate Indicator: % of facilities with adequate conditions to
provide care
 Better indicators: # of clinicians trained, % of facilities with a trained
provider

The program is not aiming to affect facility

conditions, only provider skills.
 Pitfalls with Selecting Indicators

Data needed for indicator not available

• Inappropriate Indicator: % of days per quarter that
service delivery points have stock-out of drugs
• Data issue: Information on stock-outs may not be
collected daily
• Better indicators: % of service delivery points that had
a stock out of drugs at some time during the last
quarter
 If relying on routine data, indicator definition must depend
on how data are collected
 Pitfalls with Selecting Indicators

Indicator does not accurately represent desired

outcome
 IR: Access to effective treatment among children <5
years old with malaria
• Inappropriate Indicators: % of children <5 years old who
received ACTs; % of people who received ACTs for malaria
infection who are children<5
• Better indicator: % of children <5 years old who were
diagnosed with malaria in the past 2 weeks who received
ACTs
What does it mean if inappropriate indicators
increase? Decrease? Do they reflect the desired
program effect?
Indicator systems -- How much is
enough?

Rule of thumb
 At least one or two indicators per key activity or result
(ideally, from different data sources)
 At least one indicator for every core activity (e.g., ITN
distribution, IRS, training, BCC)
 No more than 8-10 indicators per area of significant
program focus
 Use a mix of data collection strategies/source
Choosing the right number of
indicators
 Good indicators:
• Provide information useful for program decision-making

• Are consistent with international standards and other

reporting requirements, as appropriate

• Are defined in clear and unambiguous terms

• Are non-directional, “independent”

• Have values that are:

– Easy to interpret and explain
– Precise, valid and reliable measures
– Comparable across relevant population groups,
geography, other program factors, as needed
NOT EVERYTHING THAT CAN BE
COUNTED COUNTS, AND NOT
EVERYTHING THAT COUNTS CAN BE
COUNTED. Albert Einstein
 Summary: Guiding principles to
selecting indicators

 Ensure that the indicators are linked to the

program goals and are able to measure change
 Ensure that standard indicators are used to the
extent possible
 Consider the cost and feasibility of data
collection and analysis
 Keep the number of indicators to the minimum
and include only those needed for program and
management decisions or for reporting
Calculating and Interpreting
Coverage Indicators

For Malaria Programs

 Learning Objectives
By the end of the session, participants will be able to:
• Identify sources of data for calculating coverage
indicators
• Estimate denominators for routine coverage estimates
• Calculate and interpret coverage indicators from
routine data
• Use online resources for estimating coverage indicators
• Assess the quality of relevant data sources
• Reconcile coverage estimates from different
data sources
 ITN Coverage Indicators

• Proportion of households with at least one

ITN/LLIN
• Proportion of population with access to an ITN
within their household
• Proportion of households with at least one ITN
for every two people
• Proportion of the population/children under 5
years old/pregnant women who slept under an
ITN/LLIN the previous night
 IPTp Coverage Indicator

• Proportion of women who received two, three

and four or more doses of intermittent
preventive treatment for malaria during their
last pregnancy in the last two years
 Diagnostics and Treatment Coverage
Indicator

• Proportion of children under 5 years old with

fever in the last 2 weeks who had a finger or
heel stick
• Proportion receiving ACTs (or other first line
treatment), among children under five years
old with fever in the last two weeks who
received any antimalarial drugs
 IRS Coverage Indicators

• Proportion of Households which Received

Spraying through an IRS Campaign within the
Last 12 Months

• Coverage of vector control: Proportion of

Households with at least one ITN/LLIN and/or
sprayed by IRS in the last 12 months
 Why Coverage Indicators Are Important

• Understand how effective program is

• See if one target group is reached more
effectively than another
• Identify underserved areas/regions
Estimating Coverage
From Routine Data
 Indicators for Program: Numerators

• HMIS and routine reports give information on

numerators
• Numerators: number of houses sprayed with
IRS, number of LLIN distributed through
antenatal care, number of women receiving at
least two doses of SP during antenatal care
• Denominators: ?
 Example: Importance of denominator

Numerator Denominator
• IPTp Provided to Question: What will be the
denominator?
– Town A= 200 women Response: Number of women that
need IPTp who visited ANC clinics in
– Town B= 400 women each town
– Town C= 600 women • Number of pregnant
women:
Question: Can we say that Town C
has the highest coverage? Please
– Town A= 10,000
justify your response.
– Town B= 30,000
Answer: No. We need the
denominator for each town – Town C= 60,000
 Indicators for program: Denominators

• Population that are targeted by given

intervention
– District population
– Women of childbearing age
– Pregnant women visiting ANC
– Children under the age of five
– Children under 5 years old who had a fever
How Do We Get Denominators?

• Population registers
• Censuses
• Population projections
• Population growth rate (r)
• Rate of natural increase = crude birth rate (CBR)
minus the crude death rate (CDR)
• Net migration rate: inmigration - outmigrants per
1000 population
• Population growth = rate of natural increase + net
migration rate
 Estimating population size

P(t )  P( 0) * exp( r * t )
• Where:  Example:
– P(t) is the population size  300,000 people at census
after t years  Growth rate = 3% (0.03),
– P(0) is the population size at  What is the population after
the last census 10 years?
– r the annual population 404,958 people
growth rate

• Use the national statistics office project national and sub-national level
• Use UN population, World Bank estimates for national level
• Use the official figures and only make projections if they are not available
 Defining Population at Risk

A group of people who share a characteristic that

causes each member to be susceptible to a
particular event, such as people living in an
endemic area who are exposed to malaria
• Mid- term population (Mid-year)
• Expresses the population at the middle of the year
• Person-time
• Estimate of the actual time-at-risk in years, months, or days that all
persons contributed to the period/under a particular intervention.
• Only possible if individuals are follow-up
 Mid-Year Population vs. Person-Years

Person time

Individual 5 3/12=0.25

Individual 4 5/12=0.42

1
Individual 3 7/12=0.58

Individual 2
1 10/12=0.83

1
Individual 1 12/12=1

Jan 01 Dec 31
Mid year pop: Person year:
1+1+1=3 0.25+0.43+0.58+0.
83+1=3.08
 Estimating Target Population

• A District has 10,000 inhabitants in 2009, and 3%

are children under 1 year of age
– What is the annual target population for ITN
distribution for infants?
– What is the monthly population for ITN distribution for
infants?

 Answers:
 Annual target population = 10,000 x 0.03 = 300
 Monthly target population = 1,800/12 = 25
 Challenges in Estimating Coverage from
Routine Data

• Limited knowledge of target pop/denominators

• Low timeliness & completeness of reporting
• Poor data quality
– Lack of written standard reporting procedures
– No systematic supervision on data management
• Dual reporting systems (EPI, HMIS)
• Data from private sector not often included
 Assessing Reliability of Routine
Coverage Indicators
• Understand how denominators are derived
• Understand the process of collecting the information
• Look for inconsistencies and surprises
• Look for reliable data from other sources to use as a
basis for comparison
• Cross-check
Estimating Coverage
From Survey Data
 Tools for Coverage Estimation

• Large-scale population-based surveys

• Malaria Indicator Surveys - MIS
• Demographic and Health Surveys- DHS
• Multiple Indicator Cluster Survey- MICS
• Post-campaign coverage Surveys
• Other local surveys
• Lot quality sample coverage surveys
 Routine Data vs. Survey Data

Proportion of Households Owning at Least One ITN

Survey (2008) Routine MoH (2008)
100
ITN Coverage

80
60
40
20
0

Source: WHO, World Malaria Report 2009

 Reconciling Coverage Estimates from
Different Data Sources

• Age group & geographic scope

• Health cards versus recall
• Different sources for different purposes
• Not all coverage data can be compared in
constructive way
• Differences in inclusion of private sector
• Selectivity
 On-line Resource: STATcompiler

• Innovative online database tool

• Allows users to select numerous countries and
hundreds of indicators to create customized
tables that serve specific needs
• Accesses nearly all malaria and population
and health indicators published in MIS/DHS
final reports
http://www.statcompiler.com
 STATcompiler

www.statcompiler.com
Challenges with Routine-based Coverage

Advantages: Disadvantages:
• Provides  Denominator errors
information on  Poor quality reporting
more timely basis
• Makes use of data
routinely collected
• Can be used to
detect and correct
problems in service
delivery
Challenges with Survey-based Coverage

Advantages Disadvantages
• Avoids problems with  Larger standard errors at
denominators sub-national levels
• Includes community  Irregular and expensive
based information
 Survey timing may affect
coverage rates
 Group Project

Form country groups

For your project:
Identify 4-6 indicators based on your framework and
define metrics
Create an indicator matrix for your indicators
If frameworks are not finished, continue working on
frameworks
For two indicators, complete indicator reference sheet
 References
 Bertrand, Jane T., Magnani, Robert J, and Rutenberg, Naomi, 1996. Evaluating Family Planning Programs,
with Adaptations for Reproductive Health, Chapel Hill, N.C.: The EVALUATION Project.

 Bertrand, Jane T. and Escudero Gabriela, 2002. Compendium of Indicators for Evaluating Reproductive
Health Programs, vols. 1 and 2, Chapel Hill, N.C.: MEASURE Evaluation.

 Roll Back Malaria. 2009. Guidelines for core population-based indicators. January 2009. MEASURE
Evaluation: Calverton, MD.

 Tsui, Amy. 1998. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina,
Chapel Hill.

 Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina,
Chapel Hill.

 UNICEF. 1998. State of the World’s Children.

 USAID/Tanzania Country Strategic Plan, 2005-2014.

 WHO, 1999. The Evolution of Diarrhoeal and Acute Respiratory Disease Control at WHO: Achievement
1980-1995 Research, Development and Implementation (WHO/CHS/CAH/99.12).
MEASURE Evaluation is a MEASURE program project funded by
the U.S. Agency for International Development (USAID) Through
Cooperative Agreement GHA-A-00-08-00003-00 and is
implemented by the Carolina Population Center at the University
of North Carolina at Chapel Hill, in partnership with Futures Group
International, John Snow, INC., ICF Macro, Management Sciences
for Health, and Tulane University.

VISIT US ONLINE AT HTTP://WWW.CPC.UNC.EDU/MEASURE.