Monitoring and Evaluation:

Indicators

Adapted from the AAU & MEASURE Evaluation 2009

Regional Anglophone Africa Workshop on M&E of PHN
Programs; University of Nairobi 2007 Workshop on M&E of
HIV/AIDS Programs; INHL 712 Slides, Tulane University
 Learning Objectives

At the end of the session, participants will be able

to:
• Critique indicators
• Identify criteria for selection of sound indicators
• Know how indicators are linked to the frameworks
covered in the Frameworks Module
• Select indicators and complete an Indicator
Reference Sheet
An Indicator is…

• a variable
• that measures
• one aspect of a program/project or health
outcome

An appropriate set of indicators includes at least one indicator

for each significant aspect of the program or project
(i.e. at least one per box in an M&E framework)
Characteristics of Good Indicators
• Valid: accurate measure of a behavior, practice or task
• Reliable: consistently measurable in the same way by
different observers
• Precise: operationally defined in clear terms
• Measurable: quantifiable using available tools
and methods
• Timely: provides a measurement at time intervals
relevant and appropriate in terms of program goals and
activities
• Programmatically important: linked to a public health
impact or to achieving the objectives that are needed
for impact
Characteristics of
Good Indicators: Valid
Accurate measure of a behavior, practice or task

• Indicator measures what it is supposed to

measure
– Direct measures
– Proxy measures
– Straightforward interpretation: change in
value signals a change in focal concept or
behavior
Validity: Class Activity
• Is ideal family size a valid indicator of fertility
demand?
• Is the maternal mortality ratio a valid indicator of
the impact of a family planning program on
women’s health?
• Is “% Willing to buy vegetables from HIV+
shopkeeper” a valid measure of AIDS stigma?
• Is “More than one non-marital partner in past year”
a valid measure of “risky sex”?
Characteristics of
Good Indicators: Reliable
Consistently measurable in
the same way by different observers
• Types of measurement error
– Sampling Error: over-representation of urban
populations because access is easier
– Non-Sampling Error: survey estimates of abortion
incidence, due to response bias
– Subjective Measurement: indicators that ask for
personal judgment such as “quality,” “environment”
and “progress”
Characteristics of
Good Indicators: Precise
Operationally defined in clear terms

Activity: Develop definitions for

• New user
• Knowledge of AIDS
• Quality of care
• Trained provider
Characteristics of
Good Indicators: Measurable
Quantifiable using available tools and methods

• Examples of measures
– Days of drug stock outs
– Total Fertility Rate
– Indicators related to Poverty Reduction Strategy
Characteristics of
Good Indicators: Timely
Provides a measurement over periods of time of
interest with data available for all appropriate
intervals

• Timeliness Considerations
– Reporting schedules
– Recall periods
– Survey schedules
– Length of time over which change can be detected
Characteristics of Good Indicators:
Programmatically Important
Linked to a public health impact or
to achieving the objectives needed for impact

• Example: Condom distribution program

– Indicator: # of sex partners in past 4 weeks
• Example: Program to increase access to oral
rehydration salt for childhood diarrhea through
community based distributors
– Indicator: # of ORS packets distributed in past month
Factors to Consider When
Selecting Indicators
• Logic/link to framework
• Programmatic needs/information for
decision making
• Resources
• External requirements (government, donor,
headquarters)
• Data availability
• Standardized indicators
How to Operationalize Indicators

• To operationalize an indicator is to identify how a

given concept or behavior will be measured:

• Challenges
– Subjective judgment
– Local conditions
– Unclear yardsticks
 Activity: Operational Definition of
Indicators
• Get into groups of 4
• Write an operational definition for one of the following indicators:
– Prompt treatment of common childhood illness
– Trained community health worker
– Quality of counseling (provided by CHW)
• After 15 minutes, a member of each group will share the
operational definitions with all participants
Common Indicator Metrics
• Counts
– Number of providers trained
– Number of condoms distributed
• Calculations: percentages, rates, ratios
– % of facilities with trained provider
– Maternal mortality ratio, Total fertility rate
• Index, composite measures
– Quality index comprising the sum of scores on
six quality outcome indicators
– DALY (Disability Adjusted Life Years)
• Thresholds
– Presence, absence
– Pre-determined level or standard
Anatomy of an Indicator Metric
Indicator 1: # of ANC care providers trained
“providers” include any clinician providing direct clinical
services to clients seeking ANC at public health facilities
“trained” refers to attending every day of a five-day training
course offered by the program
Indicator 2: % of facilities with a provider trained in ANC
Numerator: # of public facilities with a provider who attended
all 5 days of ANC training offered by the program
Denominator: total # of public facilities providing ANC services
Always Specify the Details!
Threshold indicator for post-abortion family planning
• Facility provides post-abortion FP if
– 90% PAC clients received FP counseling
– 90% PAC clients who want no more children were informed
about sterilization services
– Facility has all of 6 relevant IEC materials
Composite Indicators
• How to measure, “Infection Prevention at facilities
providing obstetric care”?
• Could construct an index, have various
measurement options:
– “Facility Readiness,” from facility audit
• Has regulation sharps container
• Up to date guidelines on hazardous and medical waste
disposal
• Leak-proof lidded container for medical waste, etc.
– Knowledge, from Provider interview
– Practice, from Provider-client observation
Issues with Composite
Indicators
• How to “scale” the index
– Weighted v. un-weighted components
– Value of each component: yes/no,
present/absent, count of sub-components
(e.g., drugs in stock)
– Zero/one total indicator value? % of
maximum possible score? At least 8 out of
10 positive (=1) scores?
– How to interpret and report?
 Example of Composite Indicator:
Proper Disposal of Medical Waste
Percent of Facilities

100

80 74
69
64 64
60

39 41
40 37
27
23
20 15

0
Infection Regulation Leakproof, lidded Lidded plastic All Items
Prevention or Container for container for bucket for CL Available
Hazardous disposal of Medical Waste solution
Waste Protocol sharps

Government Private
Using Pre-Defined Indicators
• From past years of the program
• From related or similar programs
• From lists of global or recommended indicators
– Millennium Development Goals
– HIV/AIDS: UNAIDS/WHO
– Reproductive Health: Compendium of Indicators for
Evaluating Reproductive Health Programs
– Child Health: Guide for Monitoring and Evaluating Child
Health Programs
– Poverty Reduction Strategy Goals (World Bank)
Indicator Pyramid

Decrease
s
Global Number of
Compare countries
Overview world-wide situation Indicators

Increases
National/Sub-national
Assess effectiveness of response
Reflect goals/objectives of national/sub-national response

District or Facility
Identify progress, problems, and challenges
Indicator Matrix

Indicator Data Frequency Decision

Source Points/Comments

Outputs

% of facilities with a Program Quarterly Disaggregate by district.

provider trained in ANC records
Outcomes

% of pregnant women HMIS and Annual Does not provide

attended at least once during census information about quality
pregnancy by skilled health of care provided.
personnel for reasons
related to the pregnancy
Indicator Reference Sheet
Compile detailed documentation for each
indicator:
• Basic information
• Description
• Plans for data collection
• Plans for data analysis, reporting and review
• Data quality issues
• Performance data table (baseline and targets)
Indicator Strengths & Limitations
All indicators have limitations, even those
commonly used:
• Low birthweight: cannot use in rural areas where few
births are in facilities
• Sexual behavior (e.g. condom use, number of
partners): self reporting bias
• Maternal Mortality Ratio: maternal death is infrequent
event
• Couple Years Protection: information on program
activity but not on number of people benefiting
Setting Indicator Targets:
Useful Information Sources
• Past trends
• Client expectations
• Donor expectations
• Expert opinion
• Research findings
• What has been accomplished elsewhere
Common Pitfalls in Indicator
Selection
• Indicators not linked to program activities
• Poorly defined indicators
• Indicators that do not currently exist and cannot
realistically be collected
• Process indicators to measure outcomes &
impacts
• Indicators that are not very sensitive to change
• Too many indicators
Pitfalls with Selecting Indicators
Indicator not linked to program activities
IR: Expanded access to AIDS prevention & treatment services
Activities: train providers in current clinical protocols
Inappropriate Indicator: % of facilities with adequate
conditions to provide care
Better indicators: # of clinicians trained, # of facilities with a
trained provider
The program is not aiming to affect facility conditions, only
provider skills.
Linking Indicators with Activities

IR1: Availability of
quality services Activities:
Development of tools
IR1.1: Information and for monitoring quality
of care
services increased
Management training
for supervisors
IR1.2: Practitioners’
skills and knowledge Clinical training for
increased providers
CBD support/supplies
IEC programs
IR1.3: Sustainable
effective management
Linking Indicators with Activities Cont’d
1. % of facilities scoring 85-100 on MOH Quality of
IR1: Availability of Care checklist (rural, urban)
quality services 2. % of facilities with at least one trained provider in
each targeted service area (rural, urban)

IR1.1: Information and 1. # of IEC programs on radio in past year

services increased 2. % of facilities providing all targeted services
1. # of providers who have completed clinical
IR1.2: Practitioners’ training (pass post-training testing)
skills and knowledge 2. # of CBD personnel who have completed training
increased (pass post-training testing)

IR1.3: Sustainable
effective management 1. % of supervisors who pass post-training
management testing (national, district levels)
Interventions: 2. % of providers who report satisfaction with facility
Development of tools for supervision practices
monitoring quality of care
3. Number of districts with key stocks continuously
Management training for
supervisors
available for previous 90 days, according to
Clinical training for providers district CBD supply records
CBD support/supplies
IEC programs
Pitfalls with Selecting Indicators
Indicator poorly defined
Activity: radio campaign to dispel myths about AIDS transmission
and prevention
Inappropriate Indicator: % of population with AIDS knowledge
Better indicators: % of population with no incorrect beliefs about
AIDS (defined as: % correctly rejecting the two most common local
misconceptions about AIDS and who know that a healthy-looking
person can transmit AIDS)
AIDS knowledge does not indicate what is desirable:
knowledge that AIDS exists? How it is transmitted?
Pitfalls with Selecting Indicators

Data needed for indicator not available

Inappropriate Indicator: % of days per quarter that service
delivery points have stock-out of drugs
Data issue: Information on stock-outs may not be
collected daily
Better indicators: % of service delivery points that had a stock
out of drugs at some time during the last quarter
If relying on routine data, indicator definition must depend on
how data are collected
Pitfalls with Selecting Indicators
Indicator does not accurately represent desired
outcome
IR: Expanded access to antiretroviral treatment (ARVs) for
pregnant women to prevent mother to child transmission of HIV
Inappropriate Indicators: % of women on ARVs who are
pregnant; % of people on ARVs who are pregnant women
Better indicator: % of HIV positive pregnant women who
are on ARVs
What does it mean if inappropriate indicators increase?
Decrease? Do they reflect the desired program effect?
Indicator systems -- How much
is enough?
Rule of thumb
• At least one or two indicators per key activity or
result (ideally, from different data sources)
• At least one indicator for every core activity (e.g.,
training, BCC, CSM)
• No more than 8-10 indicators per area of significant
program focus
• Use a mix of data collection strategies/source
Not everything that can be
counted counts, and not
everything that counts can be
counted.
Albert Einstein
References
• Bertrand, Jane T., Magnani, Robert J, and Rutenberg, Naomi, 1996.
Evaluating Family Planning Programs, with Adaptations for
Reproductive Health, Chapel Hill, N.C.: The EVALUATION Project.
• Bertrand, Jane T. and Escudero Gabriela, 2002. Compendium of
Indicators for Evaluating Reproductive Health Programs, vols. 1 and
2, Chapel Hill, N.C.: MEASURE Evaluation.
• Tsui, Amy. 1998. Frameworks (ppt). Presented at the Summer
Institute, University of North Carolina, Chapel Hill.
• Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer
Institute, University of North Carolina, Chapel Hill.
• UNICEF. 1998. State of the World’s Children.
• USAID/Tanzania Country Strategic Plan, 2005-2014.
• WHO, 1999. The Evolution of Diarrhoeal and Acute Respiratory
Disease Control at WHO: Achievement 1980-1995 Research,
Development and Implementation (WHO/CHS/CAH/99.12).
Group Project
• Form groups
• For your project:
– Identify indicators and define metrics
– If frameworks are not finished, continue working on
frameworks
– For two indicators, complete indicator reference sheet