TDR Performance

Assessment Framework
REVISION—2012–2017
Measuring for improvement

For research on
diseases of poverty
UNICEF • UNDP • World Bank • WHO
 TDR Performance
Assessment Framework
REVISION—2012–2017
Measuring for improvement

For research on
diseases of poverty
UNICEF • UNDP • World Bank • WHO
For more information, please contact:
Dr Beatrice Halpaap
WHO/TDR
halpaapb@who.int

Layout: Lisa Schwarb, Lausanne

Printed by the WHO Document Production Services,

Geneva, Switzerland

TDR/STRA/13.1
TDR PERFORMANCE ASSESSMENT FRAMEWORK 3

Contents

Abbreviations....................................................................................................................................................................................................... 4

Foreword to the 2012–2017 revision. .......................................................................................................................................... 5

About the framework.................................................................................................................................................................................. 6
The framework: an overview. ............................................................................................................................................................. 7

PART I: WHY DO WE ASSESS PERFORMANCE AND WHICH APPROACH DO WE TAKE?........... 8

1. Performance assessment as an essential element of the TDR’s 2012-2017 Strategy.............................................. 8
2. Towards performance improvement................................................................................................................................................... 9
3. Guiding principles to enhance ownership and utilization................................................................................................... 10
4. A comprehensive scope of assessment............................................................................................................................................ 10
4.1. Assessing performance at activity, team and Programme levels........................................................................... 10
4.2. Assessing performance in: achieving expected results, applying TDR core values and
effective management...................................................................................................................................................................... 11

PART II: ASSESSING PERFORMANCE AGAINST EXPECTED RESULTS........................................................ 12

1. The TDR expected results guide the assessment of performance at the Programme level............................. 12
2. The team-specific expected results guide the assessment of performance at team
and activity levels........................................................................................................................................................................................... 13
3. Monitoring and evaluation....................................................................................................................................................................... 15
4. Defining performance indicators across the Programme.................................................................................................... 16
5. TDR key performance indicators........................................................................................................................................................... 16

PART III: HOW DO WE MONITOR AND EVALUATE TDR PERFORMANCE?..............................................19

1. Engagement of TDR and stakeholders...............................................................................................................................................19
1.1. Team and activity levels.....................................................................................................................................................................19
1.2. Programme level.....................................................................................................................................................................................19
1.3. Roles and responsibilities.................................................................................................................................................................21
2. Independent programme evaluation................................................................................................................................................. 22
2.1. External and independent review.............................................................................................................................................. 22
2.2. External audits........................................................................................................................................................................................ 22

PART IV: HOW APPLYING THE FRAMEWORK HELPS TDR..................................................................................... 23

1. Optimizing the framework as needed............................................................................................................................................... 23
2. Doing what is already done in a systematic and standardized way.............................................................................. 23
3. Utilizing monitoring and evaluation findings to learn, share and improve...............................................................23
4. Main challenges............................................................................................................................................................................................... 25

Adopting common terminologies................................................................................................................................................ 26

Related documents...................................................................................................................................................................................... 28

Annex 1 Engaging stakeholders in the development of this framework....................................... 30

Annex 2 Reporting....................................................................................................................................................................................... 32
Annex 3 TDR monitoring and evaluation matrix...................................................................................................... 33
4 TDR PERFORMANCE ASSESSMENT FRAMEWORK

Abbreviations
DEC Disease endemic country
GSPA-PHI Global Strategy and Plan of Action on Public Health, Innovation
and Intellectual Property
JCB Joint Coordinating Board
KPI Key performance indicator
LIC Low-income country
MDG Millennium Development Goal
OECD-DAC Development Assistance Committee of the Organization for Economic
Co-operation and Development
PAF Performance Assessment Framework
SAG Special Advisory Group
SC Standing Committee
SMG Senior Management Group
SPT Special Project Team
STAC Scientific and Technical Advisory Committee
TDR Special Programme for Research and Training in Tropical Diseases,
co-sponsored by UNICEF, UNDP, World Bank and WHO
UNEG United Nations Evaluation Group
WHA World Health Assembly
WHO World Health Organization
TDR PERFORMANCE ASSESSMENT FRAMEWORK 5

Foreword to the 2012–2017 revision

B etween 2009 and 2012, both TDR’s

Performance Assessment Framework (PAF)
and the related TDR Results Reports published
input received from the sources above and further
feedback from stakeholders. In-depth discussions
with UK DFID and the World Bank helped us to
annually have been increasingly used by TDR con- crystalize the results chain and the M&E matrix into
tributors and partners. In 2012, TDR’s Joint Coordi- their current form. In March 2013, the Scientific and
nating Board (JCB) encouraged TDR core contribu- Technical Advisory Committee STAC(35), reviewed
tors to “harmonize their reporting requirements the proposed changes to the PAF and made recom-
and accept TDR’s reporting through the JCB”, mendations on specific indicators and their suit-
highlighting the importance of a well-designed able targets.
monitoring and evaluation (M&E) framework
adapted to the Programme’s new strategy. Three key issues are specifically addressed in the
2012–2017 revision of TDR’s Performance Assess-
TDR’s Interim External Review in 2011 used TDR’s ment Framework:
PAF to evaluate progress and strategy implementa- • Improving the set of indicators and adapting
tion; the report recommended, among other things, them to the new strategy.
that TDR teams make more use of the Framework in • More clearly quantifying and expressing value for
their planning process. money.
• Aligning the monitoring and reporting aspects
At JCB(35) in June 2012, when approving the new TDR at activity, team and Programme levels with the
strategy, the JCB made specific recommendations: revamped management review system in TDR.
“The Performance Assessment Framework needs to
be updated, adjusting expected results and targets Overall, the 2012–2017 revision of TDR’s Perfor-
to the outcomes of the new strategy and workplan, mance Assessment Framework provides the tools
ensuring measures for outcomes of publications, to measure the Programme’s contribution towards
training courses and networks are included.” translating innovation to health impact in disease
endemic countries to the benefit of those burdened
By utilizing the performance assessment frame- by infectious diseases of poverty.
work and the M&E matrix over the past few years,
we noted some aspects that invited improvement.
These “lessons learnt” allowed us to better clarify
indicators’ definitions and wording, evaluate the
suitability of measurement methods and replace
those that were not feasible or relevant any more.
This current revision takes into consideration
6 TDR PERFORMANCE ASSESSMENT FRAMEWORK

About the framework

This Framework is a key element in the implementa- Assessing performance is an ongoing process and
tion of TDR’s 2012–2017 strategy1. It has the following this framework is continuously being reviewed
objectives: and refined in order to address the needs of the
• Promote continuous performance improvement Programme to achieve its objectives. It outlines the
through organizational review, learning and in- proposed framework in the context of the current
formed decision-making. systems in place to review TDR's performance and
contains four parts:
• Enhance accountability to stakeholders, including
beneficiaries, partners and resource contributors. • Part I describes the purpose, proposed approaches
and principles of performance assessment in TDR.
• Ensure strategic relevance and coherence of TDR's
It defines the different levels and specific areas of
activities to meet the aspirations expressed in the
assessment.
vision, mission and strategy.
• Part II presents TDR's expected results and the
• Ensure TDR’s performance assessment is harmo-
key performance indicators identified to measure
nized and consistent with international practices.
progress and reflect the Programme's perfor-
mance.
An initial framework was developed in 2009 in
• Part III describes the current process for monitoring
consultation with TDR staff, WHO research-related
and evaluating this performance.
programmes and regional offices and TDR's co-
sponsors, as well as external advisers from research • Part IV explains how monitoring and evaluation
and training funding institutions, development agen- findings are utilized for organizational learning
cies, research institutions and individual researchers and performance improvement.
from disease endemic countries (DECs), as shown
in Annex 1. Terms adopted by TDR are listed at the end of this
document2. Annex 2 provides a summary of the
The framework is a tool used by both TDR staff various reporting instruments. The TDR monitoring
and a broad range of stakeholders involved in the and evaluation matrix is presented in Annex 3.
governance and implementation of TDR's strategy. For each key performance indicator it lists: (i) the
It promotes and guides the systematic assessment specific achievement target; (ii) baseline data
of TDR’s strategic and technical relevance and representing the situation at the beginning of the
contribution towards its vision and mission, and it reference period; (iii) the source of verification; (iv)
clarifies how performance assessment at various who is responsible to conduct the measurement;
levels fit together into one integrated system. and (v) when the measurement needs to be made.

This current revision (2012–2017) of the framework

provides an even clearer approach to measuring
TDR's value-adding outcomes that lead to global
health impact. It builds upon a streamlined TDR
architecture to further emphasize TDR’s guiding
principles such as equity, quality, partnerships and
value for money. The changes made have taken into
consideration recommendations made by TDR’s Joint
Coordinating Board, the Interim External Review in 1. http://www.who.int/tdr/publications/strategic_plan/en/index.html
2011, TDR’s STAC, as well as feedback from various 2. Definitions of monitoring and evaluation terms were proposed and/or
adapted from terminologies used by TDR cosponsors and international
stakeholders who have been using the framework. organizations. See the 'Adopting common terminologies' section and
related references.
TDR PERFORMANCE ASSESSMENT FRAMEWORK 7

The framework: an overview

A framework to guide systematic assessment Performance is assessed against expected

of performance results described in the TDR results chain
The Performance Assessment Framework focuses To guide the performance assessment, the
its monitoring and evaluation efforts on the Programme’s expected results have been clearly
outcomes leading to global health impact that outlined. The results chain (Fig. 4, Section 1, Part II)
are most relevant to stakeholders. The framework shows the higher level results, while a complete
builds upon the existing review process (recently list of expected results (deliverables) of capacity
streamlined and strengthened) and guides TDR building and research activities is being used in
staff and stakeholders through a more systematic routine monitoring and reporting. The current
way of monitoring and evaluating the Programme's results chain, highlighting the new strategy’s focus
performance. and approach, reflects the Programme’s logic to
achieving its objectives by contributing to the
Towards continuous performance improvement broader impact of reducing the global burden
While enhancing accountability, measuring the of infectious diseases of poverty and improving
Programme's performance gives an understanding health in vulnerable populations, including women
of “what works and what doesn't” and also of the and children.
underlying or contributing factors. This leads to
enhanced organizational learning and informed Key performance indicators are used to reflect
decision-making, which in turn foster performance the main aspects of performance
improvement. At each level TDR assesses its performance in three
areas: (1) achievement of technical expected results;
Performance is assessed at activity, team and (2) application of core values; and (3) management
Programme levels performance. Key performance indicators have
To ensure consistency and coherence, the various been developed to reflect performance across the
measurements are aggregated as much as possible Programme (Annex 3).
across the Programme. The indicators have been
selected based on relevance; however, feasibility
and ease of measurement have also been taken
into consideration.
8 PART 1 - TDR PERFORMANCE ASSESSMENT FRAMEWORK

PART I: WHY DO WE ASSESS PERFORMANCE

AND WHICH APPROACH DO WE TAKE?

1. Performance assessment as an essential

element of TDR’s 2012-2017 Strategy

TDR's vision is for “the power of research and A suitable system to assess performance allows
innovation to improve the health and well-being of for cost-efficient and real-time measurement
those burdened by infectious diseases of poverty”. and monitoring of progress indicators to inform
decision-making. Aligned with the new TDR
For that purpose, TDR has set its mission to “foster strategy, the current revision of the framework
an effective global research effort on infectious further demonstrates TDR’s focus on health impact
diseases of poverty and promote the translation and value for money throughout the whole results
of innovation to health impact in disease endemic chain, from using resources carefully to building
countries”. efficient processes, to quality of outputs, and to the
sustainability of outcomes (Fig. 1).

Figure 1. TDR’s strategic impact goals

IMPACT GOALS

• Foster research on infectious diseases of poverty that leads to health improvement.

• Engage disease endemic regions and countries in setting the health research agenda
and harmonizing the global response.
• Strengthen the capacity of individuals and institutions in disease endemic countries
to perform research related to their own priority health issues.
• Develop innovative knowledge, solutions and implementation strategies that respond to
the health needs of disease endemic countries.
• Translate innovation, knowledge, solutions and implementation strategies to policy and
practice that improves health.
• Promote the involvement of individuals, communities and societies in the use of research
evidence to reduce the burden of endemic diseases in their countries.
TDR PERFORMANCE ASSESSMENT FRAMEWORK - PART 1 9

2. Towards performance improvement

The purpose of assessing performance is to analyse • Enhance accountability to stakeholders – benefi-

the Programme’s added value and to understand ciaries, partners and resource contributors.
the factors that affect the achievement of its objec- • Ensure strategic relevance and coherence of TDR
tives. activities to meet the aspirations expressed in the
vision, mission and strategy document
TDR's performance assessment has the following
• Ensure TDR’s performance assessment is harmo-
objectives:
nized and consistent with international practices.
• Promote continuous performance improvement
through organizational review, learning and
informed decision-making (Fig. 2).

Figure 2. Role of performance assessment in the

continuous performance improvement process

Adapted from Handbook

on planning, monitoring and
Informed Performance
decision-making assessment evaluating for development
results. New York, United
CONTINUOUS Nations Development
PERFORMANCE Programme, 2009.
IMPROVEMENT

Organizational
learning
10 PART 1 - TDR PERFORMANCE ASSESSMENT FRAMEWORK

3. Guiding principles to enhance ownership

and utilization

The performance assessment, including monitoring performance improvement, policy analysis, in-
and evaluation activities, is guided by TDR's past formed decision-making and enhanced strategic
experience, principles outlined in international relevance of the Programme.
guidelines3 and lessons learnt from other interna-
• Harmonization within TDR and with
tional organizations (Annex 1). Guiding principles
international practices
include:
Seeking to harmonize monitoring and evaluation
practices with those of its co-sponsors and other
• Inclusiveness and transparency
international stakeholders to enhance coherence,
Engaging TDR staff and stakeholders in the devel-
collaboration and synergy.
opment of the monitoring and evaluation matrix,
as well as in the assessment of results. Sharing • Credibility and practicability
monitoring and evaluation data to enhance orga- Applying the ‘keep it simple’ concept to the moni-
nizational learning and utilization of the evidence. toring and evaluation system to ensure feasibil-
ity and credibility, and to facilitate the system's
• Usefulness implementation by stakeholders.
Promoting user performance assessment owner-
• Incremental approach
ship at each Programme level and ensuring that
Optimizing the system progressively and continu-
the system is useful to staff and stakeholders
ously while building on existing systems and good
alike. Promoting organizational learning towards
practices.

4. A comprehensive scope of assessment

The assessment framework has a broad and To ensure consistency and coherence, the various
comprehensive scope when addressing the measurements need to be aggregated as much as
Programme's expected results, core values and possible throughout the Programme. Monitoring
management performance. These are monitored and evaluation findings at the activity level are
and evaluated at activity, team and Programme aggregated at the team level. Measurements
levels, as described below. at the team level are, in turn, aggregated at the
Programme level, as shown in Fig. 3.
4.1. Assessing performance at activity, team and
Programme levels
The framework provides a performance assessment
structure at the following levels:
3. Principles for evaluation of development assistance. Paris, OECD
• Activity level (project management and contract
Development Assistance Committee, 1991 (http://www.oecd.org/
management, including research grants) dataoecd/31/12/2755284.pdf, accessed on 4 January 2013); UNEG
ethical guidelines for evaluation. New York, The United Nations
• Team level (areas of work)
Evaluation Group, 2007 (http://www.unevaluation.org/ethicalguide-
• Programme level. lines, accessed on 4 January 2013).
TDR PERFORMANCE ASSESSMENT FRAMEWORK - PART 1 11

Figure 3. Aggregation of Programme performance

PROGRAMME
PERFORMANCE
PLANNING
Programme MEASUREMENT
performance

Team
performance

Activity
performance

4.2. Assessing performance in: achieving Effective partnerships

expected results, applying TDR core values and Measuring the extent to which TDR is working
effective management through useful and productive partnerships.
At each level, TDR assesses performance in three
specific areas: Sustainability
Measuring the extent to which benefits continue
• Achievement of technical expected results after TDR guidance and support have been discon-
Measuring the extent to which expected results: tinued.
(1) remain strategically relevant and coherent
within the global context; and (2) have been Quality
achieved. Achievement of expected results Measuring the extent to which TDR outputs (re-
represents a measure for progress towards the search and capacity strengthening) are recognized
global health impact. as being of good quality and in line with interna-
tional standards.
• Application of TDR's core values
Equity • Management performance
Measuring the extent to which TDR has: 1) 1) Measuring the extent to which objectives have
mainstreamed equity issues, such as gender been achieved efficiently through contribution
balance and other social determinants of health, from teams and individuals; and 2) the extent to
in its portfolio; and 2) the extent to which disease which significant risk factors have been taken into
endemic countries have an influential/critical/ consideration and successfully addressed.
leadership participation in TDR research-related
activities, from research priority setting and The performance assessment described in this
research partnerships to strengthening policy- section will be conducted through systematic
making. monitoring, surveys, interviews, analyses,
documented reporting and evaluation processes.
12 PART 2 - TDR PERFORMANCE ASSESSMENT FRAMEWORK

PART II: ASSESSING PERFORMANCE

AGAINST EXPECTED RESULTS

1. TDR expected results guide the assessment of

performance at the Programme level

To guide the performance assessment, the Pro- TDR’s outcomes contribute to WHO’s outcomes.
gramme’s expected results are clearly outlined. The They are reported to the World Health Assembly in
results chain (Fig. 4) presents these expected results conjunction with other WHO departments, offices
and reflects the Programme’s logic to achieving and regions that share the same objectives.
its objectives and in contributing to the broader
impact on global health.

Figure 4. TDR results chain

TDR results chain 2012–2017

UN co-sponsored Stimulating research

programme initiatives in DECs
implemented
through WHO and Providing technical
governed by a support, guidance
34 member board and leverage funding
with global High quality intervention
for intervention and
representation and implementation
implementation
research evidence
research
produced
Team of 30 staff
with Supporting
multi-disciplinary individual and
institution research New and Infectious Reduced global
expertise in
leadership improved disease burden of
research, capacity
development solutions knowledge, infectious
building and
Enhanced research and and imple- solutions and diseases of
knowledge
knowledge transfer mentation implementation poverty and
management in Expanding
capacity within disease strategies that strategies improved health
infectious partnerships for
endemic countries respond to the translated into in vulnerable
diseases of povery research & capacity health needs of policy and populations,
building with disease practice in including
stakeholders and endemic disease endemic women and
Forecasted peer organizations countries countries children
income developed
from multiple
sources: US$ Facilitating the
180M (2012-2017) harmonization of Key stakeholders in disease
research related endemic countries engaged
stakeholders priorities in setting the research agenda
and funders policies and and ensuring research reflects
Strong country-
practices their needs
based and global
networks of
scientists and Creating and
partnering sustaining
institutions productive networks

Inputs Processes Feeder outputs Main output Outcome Impact

TDR PERFORMANCE ASSESSMENT FRAMEWORK - PART 2 13

2. Expected results guide the assessment of

performance at team and activity levels

Each team’s specific expected results are consistent Financial implementation is done by comparing the
with the overall TDR results chain and are feeding amounts spent or contractually committed versus
into TDR’s outputs and outcomes. planned cost for each output and outcome (at team
level). This information is available in quasi real-time
Technical progress is measured in relation to to project managers and, together with information
financial implementation, both at activity and team on technical implementation, helps inform decision-
levels, and against initial or revised targets (agreed making, management review and reporting.
with donors where applicable) for deliverables.
Monitoring of milestones, addressing delays and
other issues that may appear during project imple-
mentation, are part of the monitoring and report-
ing at team level (Fig. 5).

Figure 5. Progress monitoring at team level

Planned timelines to reach milestones Progress Revised date

VES TECHNICAL IMPLEMENTATION

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
GANTT CHART

Outcome 1 Promotion and adoption of new and improved vector control methods and strategies

Improved tsetse control methods

Output 1.1
and strategies

Glossina genome generated and

Output 1.2
expoited

Best practice guidance for

Output 1.3
deployment of GM mosquitoes

Guidance framework for testing

Output 1.4
GMM for safety and efficacy

Improved methods for integrated

Output 1.5
malaria vector control

Improved methods for targeted

Output 1.6
dengue vector control

Methods for preventing

Output 1.7
reinfestation by triatomine bugs

Strategies for altenative Chagas

Output 1.8
vector control methods
14 PART 2 - TDR PERFORMANCE ASSESSMENT FRAMEWORK

VES TECHNICAL IMPLEMENTATION

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
GANTT CHART

Outcome 2 Policies and strategies influenced by new evidence from community-based vector control

Situation analysis of dengue and

Output 2.1
Chagas in Asia and Latin America

Evidence for community-based eco-

Output 2.2 system management interventions
for dengue and Chagas disease

Sustainable Communities-of-Prac-
Output 2.3
tice (CoP) of researchers

Promotion and strategies influenced by new evidence about climate and environmental
Outcome 3
change impact on vector-borne diseases

Evidence on the effects of climate

Output 3.1
and environmental change

Decision-support processes and

Output 3.2
tools for health impact assessment

Output 3.3 Capacity and networks building

Policies and strategies influenced by new evidence about community-based strategies

Outcome 4
for enhanced access to control interventions

Evidence for CDI* for strengthening

Output 4.1
Primary Health Care

Evidence on incentives for health

Output 4.2
workers

Evidence on integrated
Output 4.3 Community Case Management
(iCCM) of malaria and penumonia

Knowledge generation and

Output 4.4 management on community-based
interventions

* CDI: Community-directed interventions

TDR PERFORMANCE ASSESSMENT FRAMEWORK - PART 2 15

3. Monitoring and evaluation

Monitoring activities focus on tracking progress Managerial control of the process is greater during
towards results (Fig. 6). Evaluation activities focus the implementation phase. Delivery of outputs can
on assessing relevance, impact, effectiveness, therefore be clearly attributed to the Programme.
efficiency and sustainability. Evaluation helps to However, the Programme cannot achieve expected
understand the role of various underlying factors outcomes and impacts on its own – various
in the success or failure of activities and work stakeholders and external factors contribute
areas. Although both monitoring and evaluation to their attainment. While the TDR specific
are ongoing processes from input to impact, contribution to outcomes and impacts cannot
monitoring is more relevant during implementation always be measured, it is possible to demonstrate
(from input to output), while evaluation is more the link between outputs and the desired/achieved
relevant to results and expected changes (from outcomes and impact.
output to impact). Periodic external evaluation will
provide input so that the Programme maintains
strategic relevance to global issues.

Figure 6. Monitoring and evaluation approach

Results
Inputs Process
Outputs Outcomes Impact

• Financial, • Activities • Products and • The predicted or • Changes at global

human and services achieved effects health level
material delivered of outputs
resources used (deliverables)

Attribution Contribution
Inputs, processes and outputs are directly It is expected that TDR outputs will
attributed to TDR contribute to global benefit

Evaluation >>>
Monitoring >>>
Are we on the right track?
Are we on track?
16 PART 2 - TDR PERFORMANCE ASSESSMENT FRAMEWORK

4. Defining performance indicators across

the Programme

Out of a multitude of possible indicators, TDR has Additional performance indicators, at all three
selected a limited number of relevant quantitative levels, may be developed in order to measure
and qualitative key performance indicators to help performance in a comprehensive way or highlight
measure progress and assess performance at the specific aspects that require attention. Performance
Programme level (see key performance indicators, indicators are selected at activity and team levels
Part II, Section 5). and aggregated up to the Programme level.

5. TDR key performance indicators

A range of indicators has been carefully selected Table 1 presents a consolidated list of key
to measure performance across TDR, as described performance indicators used across the
in Part I, Section 4.2. It is understood, however, Programme to measure and report on the three
that the use of indicators has limitations when main performance areas and progress made in
the objective is to express different aspects of implementing the strategy.
programme performance (see quote below).
TDR’s monitoring and evaluation matrix is
"Everything that can be counted does not presented in Annex 3. For each indicator, it
necessarily count; everything that counts presents:
cannot necessarily be counted." (i) the specific achievement target;
(ii) baseline data representing the situation before
Albert Einstein, 1879–1955
the start of activities;
(iii) the source of verification;
With the proposed indicators TDR is aiming to
(iv) who is responsible to conduct the
reflect performance aspects that are traditionally
measurement; and
hard to quantify. All the proposed indicators satisfy
(iv) when the measurement will be made.
the SMART criteria (specific, measurable, attainable,
relevant, and time bound).
TDR PERFORMANCE ASSESSMENT FRAMEWORK - PART 2 17

Table 1. TDR key performance indicators

Expected results Key performance indicators

Technical expected results

Outcome: 1. Number and proportion of new/improved solutions, imple-

Infectious disease knowledge, mentation strategies or innovative knowledge successfully
solutions and implementation applied in developing countries.
strategies translated into policy
and practice in disease endemic 2. Number of tools and reports that have been used to inform
countries policy and/or practice of global/regional stakeholders or ma-
jor funding agencies.

Main output: 3. Number and proportion of new/improved solutions, imple-

New and improved solutions and mentation strategies or innovative knowledge developed in
implementation strategies that response to requests from WHO control programmes and/or
respond to health needs of disease disease endemic countries.
endemic countries developed
4. Number of peer-reviewed publications supported by TDR and
percentage published in open access journals.

Feeder outputs: 5. Number and evidence of new/improved tools, case-manage-

High quality intervention and ment, control or implementation strategies generated through
implementation research evidence TDR facilitation with systematic quality review by external
produced committees.

6. Percentage of peer-reviewed publications supported by TDR

with first author from DEC institutions.

Enhanced research and knowledge 7. Number of DEC institutions and/or networks demonstrating
transfer capacity within disease expanded scope of activities and/or increased funding from
endemic countries alternative sources thanks to TDR support.

8. Number of TDR grantees/trainees and proportion

demonstrating career progression and/or increased scientific
productivity within 3 to 5 years.

Key stakeholders in disease endemic 9. Number and evidence of research-related agendas, recom-
countries engaged in setting the re- mendations and practices agreed by stakeholders at global,
search agenda and ensuring research regional or country level.
reflects their needs
10. Proportion of TDR outputs produced with key DEC stakeholder
active involvement (within calendar year).
18 PART 2 - TDR PERFORMANCE ASSESSMENT FRAMEWORK

Table 1. (Continuing) TDR key performance indicators

Application of core values

Equity 11. Proportion of TDR grants/contracts awarded to institutions or

Socio-economic: individuals in DECs (total count and total dollar amount).

12. Proportion of experts from DECs on TDR advisory committees.

Gender: 13. Proportion of women among grantees/contract recipients

(total count and total amount).

14. Proportion of women on TDR advisory committees.

15. Proportion of women as first author of peer-reviewed publica-

tions supported by TDR (within a calendar year).

Effective partnerships 16. Resources leveraged as direct contributions (co-funding, ser-

vices or in-kind) to TDR projects (examples).

Sustainability of outcomes 17. Number of effective public health tools and strategies devel-
oped which have been in use for at least two years.

Quality of work 18. Proportion of project final reports found satisfactory

by peer-review committees.

Management performance

Effective resource mobilization 19. Percentage of approved biennial budget successfully funded.

20. Percentage of income received from multi-year agreements.

Effective management 21. Percentage of staff workplans and performance reviews

(including personal development plan) completed on time.

22. Proportion of expected results on track.

23. Proportion of significant risk management action plans that

are on track.
TDR PERFORMANCE ASSESSMENT FRAMEWORK - PART 3 19

PART III: HOW DO WE MONITOR AND

EVALUATE TDR PERFORMANCE?

Both the TDR secretariat and stakeholders (such as grant and contract managers, advisory
committees, partners and governing bodies) carry out regular performance assessments. The
frequency of these reviews varies from monthly to yearly. Independent external evaluations
of TDR as a Programme are done once every five to seven years.

1. Engagement of TDR and stakeholders

1.1. Team and activity levels conducted as required. These may be requested
Monitoring at team and activity levels by TDR managers or by advisory committees or, in
Team leaders and project managers have devel- special circumstances, by TDR's governing bodies.
oped indicators which contain a specific achieve-
ment target and timeline for measurement. These 1.2. Programme level
elements are reviewed internally at the quarterly Internal evaluation at quarterly management
management review meetings and externally by review meetings
expert advisory committees and TDR’s governing At the quarterly management review meetings,
bodies. Performance monitoring activities are con- team leaders present highlights of the progress
ducted according to the respective team’s schedule, made both on the technical side (project mile-
as shown in Fig. 5. stones) and on the financial side of projects
and activities (funds spent and obligated versus
Evaluation by Special Advisory Groups (SAGs) planned costs). Any issues encountered, as well as
Ad hoc, time-limited independent SAGs assist TDR risk mitigation measures, are discussed in the quar-
in the technical review of activities by focusing on terly management reviews. The quarterly reviews
specific areas or projects requiring additional or provide an opportunity for sharing experience and
specialized input. The need and the specific issues organizational learning.
taken on by the SAGs are proposed by TDR staff
and endorsed by STAC and may include: advice on Governing bodies oversight
strategic direction, priority setting, screening and Joint Coordinating Board – Due to its nature as a
selection of projects, recommendations for fund- United Nations co-sponsored research and training
ing, follow-up of progress and evaluation of results. programme, TDR benefits from a special gover-
nance structure. The Programme is governed by the
SAGs are proposed by the TDR Director to STAC, Joint Coordinating Board (JCB), consisting of coun-
which appoints a chair from amongst its members tries elected by the six WHO regional committees;
with the most relevant scientific and technical resource contributor countries or constituencies;
expertise. other cooperating parties and the four co-sponsor-
ing agencies. The JCB reviews the expected results,
Ad hoc contracted evaluation studies performance and relevance of the Programme
Evaluation studies to address specific issues or annually and approves the Programme's budget
questions related to work areas or activities are for each biennium. This Performance Assessment
20 PART 3 - TDR PERFORMANCE ASSESSMENT FRAMEWORK

Framework and the corresponding TDR Results UNICEF, UNDP, the World Bank and WHO, the chair
Report are used as tools to guide the JCB's review. and vice-chair of the JCB, chair of STAC, a represen-
tative of the JCB resources contributors group and a
Scientific and Technical Advisory Committee – The representative of the disease endemic countries. It
JCB and TDR Director are supported by a Scientific reviews the overall management of the Programme.
and Technical Advisory Committee (STAC) com-
prised of globally recognized experts. This commit- Processes – STAC reviews a draft version of the
tee undertakes an annual scientific and technical annual technical reports (by work area) and the
review of the Programme and advises on strategy TDR Results Report highlighting the Programme’s
directions. STAC reviews the Programme's expected performance and makes recommendations. The re-
results and performance as presented in the TDR vised documents and the draft TDR Annual Report
Results Report and in the respective annual techni- are then reviewed by the Standing Committee, with
cal reports. The present framework guides this the final reports submitted for approval to the JCB.
review. The oversight review model described in Fig. 7
provides TDR with convening power, credibility as
Standing Committee – A Standing Committee a neutral player, and access to global expertise and
consists of the four co-sponsoring agencies, namely knowledge from multiple disciplines and sectors.

Figure 7. TDR governance organigram

Joint Coordinating
Board (JCB)

Standing
Committee

Scientific and
Technical Advisory
Committee (STAC)

TDR secretariat

Strategic Advisory
Groups (SAGs)
TDR PERFORMANCE ASSESSMENT FRAMEWORK - PART 3 21

WHO's performance assessment by the The Portfolio and Programme Management (PPM)
World Health Assembly unit is responsible for facilitating the performance
TDR contributes to two of the thirteen WHO stra- assessment process in consultation with the Direc-
tegic objectives (SOs) highlighted in the Eleventh tor’s office, TDR staff and stakeholders, including
General Programme of Work, 2006-2015 – A Global donors and partners. It fosters the utilization of
Health Agenda: (i) SO1 – to reduce the health, social monitoring and evaluation findings for continuous
and economic burden of communicable diseases; improvement through portfolio analysis, and for
and (ii) SO2 – to combat HIV/AIDS, tuberculosis and providing the basis for policy advice and decision-
malaria. TDR’s technical and financial progress making. PPM facilitates organizational learning,
towards achieving the specific expected results information management and risk management in
contributing to these two SOs is compiled in WHO's close collaboration with other relevant units.
annual Performance Assessment Report, which is
reviewed by the Executive Board and the World Team leaders and project managers are responsible
Health Assembly. for coordinating technical activities. They lead the
development and implementation of expected
WHO has developed the Twelfth General results and related activity indicators in consulta-
Programme of Work, 2014-2019. As of 2014, TDR will tion with PPM, advisory committees and major
contribute mainly to Category 1 (Communicable stakeholders within and outside of WHO. Team
Diseases), with strong linkages to the other four leaders and project managers are also responsible
categories (work on maternal and child health, for integrating systematic performance assessment
outbreaks, health systems, etc.). and risk management within the activities of the
teams.
WHO internal audits
TDR's operational, administrative and financial Stakeholders have been extensively engaged in
procedures and practices are subject to audit by the development, implementation and revision of
WHO's internal auditors, who perform ad hoc audits the Performance Assessment Framework. Resource
following the schedule and procedures established contributors provided input into the design of the
for WHO as a whole. M&E matrix and helped define and revise TDR’s
results chain. Study investigators, consultants and
institutions are under contract to manage activi-
1.3. Roles and responsibilities ties, monitor their progress and evaluate results
TDR Director provides leadership in promoting prior to independent review. Partners assist TDR in
performance assessment and supporting its use identifying collective outcomes and impact, and
in the management cycle. The Director has overall help develop means to jointly measure such indica-
responsibility for the Programme's performance. tors. External advisers such as advisory committee
members evaluate relevance, quality and achieve-
The Senior Management Group (SMG) and team ment of the activities, teams and the Programme as
leaders are engaged in the implementation and a whole.
review of the Performance Assessment Framework.
The SMG has a critical role in promoting and lead- Governing bodies, including representatives from
ing continuous performance improvement at all disease endemic countries, review the Programme’s
levels of the Programme, utilizing the monitoring expected results and performance and request
and evaluation data and contributing to organiza- periodic external reviews and ad hoc independent
tional learning. evaluations on specific issues as needed.
22 PART 3 - TDR PERFORMANCE ASSESSMENT FRAMEWORK

2. Independent programme evaluation

2.1. External and independent review 2.2. External audits

The JCB regularly requests an independent external TDR financial statements are certified annually
review of TDR, usually every five to seven years. by the Comptroller of WHO.
These reviews have been a key component in guid-
ing TDR's development. TDR’s new strategy was TDR’s revenue, expense and fund balance figures
developed following the interim external review of are part of WHO financial data, which are audited
2011. The Performance Assessment Framework was annually by independent external auditors in accor-
instrumental for the review. dance with the International Standards on Audit-
ing. The report and statement of the audit of WHO
are made available to the World Health Assembly
each year.

The audit report and statement of the external

auditor, as well as the TDR financial statements,
are made available to JCB each year.
TDR PERFORMANCE ASSESSMENT FRAMEWORK - PART 4 23

PART IV: HOW APPLYING

THE FRAMEWORK HELPS TDR

1. Optimizing the framework as needed

Implementation of the framework is an incremen- Internal and external review systems are used to
tal process starting at the Programme level, then facilitate a systematic TDR monitoring and evalua-
integrated, step-by-step, at team and activity levels. tion process. These indicators have been selected
The framework builds on systems that already ex- to reflect progress on the strategic plan 2012-2017.
ist. As the framework is being implemented at team Consideration was given to selecting a limited
and activity levels, it is optimized to facilitate its number of indicators that are sensitive enough
application and to fit the needs of the Programme. and easy to measure.

2. Utilizing monitoring and evaluation findings to

learn, share and improve

Organizational learning is critical if the process of Regular progress monitoring and performance
performance assessment is to lead to performance evaluation provide a good understanding of where
improvement. the Programme lies in achieving the expected
results. They help clarify the factors underlying
Fig. 8 shows how a monitoring and evaluation pro- these achievements, make informed decisions and
cess fits into the overall management cycle of TDR readjust the plans accordingly.
and how the related findings are utilized to learn,
share and make informed decisions at individual
and organizational levels.
24 PART 4 - TDR PERFORMANCE ASSESSMENT FRAMEWORK

Figure 8. Use of monitoring and evaluation findings for organizational learning

Planning

Making informed
decisions
Implementing

PERFORMANCE
IMPROVEMENT
(review meetings
and lunch seminars)
Organizational
Monitoring &
learning
Evaluation

Reporting

Described below are various opportunities at TDR reviewed. Progress on expected results (outputs
to discuss collectively the monitoring and evalua- and outcomes) is assessed. The indicators present-
tion findings. ed in the framework are reviewed and milestones
highlighted. The review allows for reflection and
Monthly staff meetings provide a good opportunity discussion on past experiences. Risk management
for updates and sharing experiences. actions are being followed up on and additional
measures identified as needed.
Bi-weekly team leaders meetings discuss progress
made and any issues encountered that need special The governance structure and review processes
attention. The meetings are also opportunities to through the advisory committees greatly facilitate
review new processes, systems and policies ahead performance improvement. Recommendations are
of those being implemented at Programme level. carefully analysed and addressed.

Lunchtime seminars are organized regularly to Follow-up on recommendations is coordinated

discuss technical issues and share lessons learned. at the bi-weekly team leaders and SMG meetings.
Innovative processes and systems to facilitate
At the quarterly management review meetings, organizational learning are being investigated.
the performance of teams and units is internally
TDR PERFORMANCE ASSESSMENT FRAMEWORK - PART 4 25

3. Main challenges

Performance assessment and related monitoring of international norms, standards and guidelines.
and evaluation activities are recognized as criti- In its efforts to optimize performance assessment,
cal elements in global health initiatives and in the TDR is seeking to harmonize with international
development sector. They give programmes the practices and engage with stakeholders.
chance to highlight their results and their contri-
bution towards global health, to ensure strategic Engagement of TDR's management, leadership and
relevance and to identify what does and does not staff in the performance assessment process has
work. However, measuring the specific outcomes been critical for its success. Expanding the focus
and impact of a single programme is challenging, to outcomes and impact required a major culture
as improvements made in global health are often change within TDR, but it is now facilitating the
synergistic among stakeholders and seldom implementation of the new strategy 2012-2017.
achieved by a single programme.

The need for coherence between the various stake-

holders requires harmonization of monitoring and
evaluation practices. Various international groups
and networks have been leading the development
26 TDR PERFORMANCE ASSESSMENT FRAMEWORK

Adopting common terminologies

This section provides the definition of common Equity – Absence of avoidable or remediable
terms adopted by TDR. The monitoring and differences among groups of people, whether
evaluation terms used in this document are aligned those groups are defined socially, economically,
with those adopted by TDR co-sponsors and other demographically or geographically.
international organizations4.
Evaluation – The systematic and objective
Accountability – Obligation towards beneficiaries, assessment of the relevance, effectiveness,
resource contributors and other stakeholders, to efficiency, impact and sustainability of an ongoing
demonstrate that activities have been conducted or completed activity, a team, a policy or the
in compliance with agreed rules and standards and Programme. Evaluation can also address specific
to report fairly and accurately on the achievement issues and answer specific questions to guide
of objectives vis-à-vis mandated roles and/or plans. decision-makers and managers and to provide
It involves taking into account the needs, concerns, information on the underlying factors influencing
capacities and disposition of affected parties, and a change.
explaining the meaning of, and reasons for, actions
and decisions. Expected results - Expected results are outputs,
outcomes and/or impact that TDR intends to
Activity – A set of interrelated actions necessary
produce through its portfolio of activities.
to deliver specific outputs towards achieving the
objectives. In TDR, the activity level encompasses Impact – Positive or negative, primary or secondary
all actions under a team, including contracting for long-term change produced by an activity or a
research grants and services. set of activities directly or indirectly, intended or
unintended. It is the ultimate change in public
Attribution – The direct causal link between ob-
health to which outcomes are linked or contribute.
served (or expected) changes and a specific activity.
Indicator – See performance indicator.
Baseline data – Indicator data that describes the
situation at the beginning of the TDR strategy Input – Financial, human and material resources
implementation, against which progress can be used for activities.
assessed or comparisons made. Baselines may not
Key performance indicator – Performance
be available when measurements are complex and
indicator that is shared across the Programme and
expensive. In such cases the first measurement to
can be aggregated from the activity level to the
be carried out through this framework will serve as
team and from the team level to the Programme
the baseline level.
level.
Contribution – The indirect causal link between
observed (or expected) changes and a specific Milestone – Performance indicator related to
activity or set of activities. It is implied that the processes and used to track progress towards
change cannot be produced by the activity or set of achievements of outputs. Milestones are key
activities specific to the Programme alone but will events, achievements or decisions in workplans.
be achieved through the output of the Programme They map out the main steps of the workplan
combined with outputs resulting from the activi- implementation.
ties of partners and other players. Monitoring – A continuing function that
Disease endemic country (DEC) – A low-, middle- aims primarily to provide managers and main
income5 or least developed6 country in which stakeholders with regular feedback and early
infectious diseases (whether endemic or epidemic) indications of progress or lack thereof in the
contribute to the overall burden of disease7 or achievement of intended results. Monitoring
mortality and/or a major public health problem. tracks the actual performance or situation against
TDR PERFORMANCE ASSESSMENT FRAMEWORK 27

what was planned or expected according Results chain – Causal sequence of the expected
to pre-determined standards. Monitoring results to achieve objectives and contribute to
generally involves collecting and analysing the broader impact. The TDR results chain reflects
data on specified performance indicators and the causal sequence of the programme's expected
recommending corrective measures. results to achieve the Programme’s objectives.

Neglected priorities – Priority research needs Review – An assessment of the performance of

that are not adequately addressed by others. activities, team or Programme, periodically or on an
ad hoc basis.
Outcome – The likely or achieved short-term
and medium-term effects of an activity’s output. Stakeholder – Governments, agencies,
Outcomes are short- and medium-term changes organizations, institutions, groups or individuals
derived from outputs. As the outcomes are also who have a direct or indirect interest in TDR's
influenced by actions implemented by partners activities or evaluation.
and external factors, they cannot be fully attrib- Sustainability – The continuation of benefits after
uted to TDR and are not under the Programme’s major guidance and support have been completed.
control.
Target – Targets provide a desirable level of
Output – Products and services resulting from achievement at a given time. Outcome targets
activities. allow for a span of a few years after the current
strategy period.
Partnership – Formalized relationship between
TDR and one or more country, region, organiza-
tion, institution, company or foundation around
an activity or set of activities in which there are
well- defined common objectives and shared
benefits, where both TDR and the strategic
partner make continuing contributions in one or
more strategic area, such as technical expertise,
financial contribution, technology or product.

Performance – The degree to which an activity,

team or programme operates, according to spe-
cific standards and guidelines, aligns with the
Programme's core values or achieves results in
accordance with stated objectives and plans. 4. Glossary of key terms in evaluation and results based management.
Paris, OECD Development Assistance Committee, 2002 (http://www.
Performance indicator – Quantitative or qualita- oecd.org/findDocument/0,2350,en_2649_34435_1_119678_1_1_1,00.
html, accessed on 14 May 2013); Standards for evaluation in the UN
tive factor or variable that provides a simple
system. New York, The United Nations Evaluation Group, updated in
and reliable means to measure achievement, to 2012 (http://www.uneval.org/normsandstandards/index.jsp?doc_
cat_source_id=4, accessed on 14 May 2013); Norms for evaluation in
reflect the changes connected to an interven- the UN System. The United Nations Evaluation Group, 2005, updated
tion, or to help assess performance. in 2012 (http://www.uneval.org/normsandstandards/index.jsp?doc_
cat_source_id=4, accessed on 14 May 2013).
Programme – Programme refers to the TDR 5. As per the World Bank classification (http://data.worldbank.org/
about/country-classifications/country-and-lending-groups, accessed
Programme as a whole.
on 14 May 2013).

Result – The output, outcome or impact 6. As per UN classification (http://unstats.un.org/unsd/methods/m49/

m49regin.htm#least, accessed on 14 May 2013).
(intended or unintended, positive and/or
7. As per WHO statistics (http://www.who.int/gho/publications/world_
negative) of a set of activities. health_statistics/EN_WHS2012_Full.pdf, accessed 14 May 2013).
28 TDR PERFORMANCE ASSESSMENT FRAMEWORK

Related documents

Organization for Economic Co-operation and United Nations Children's Fund (UNICEF) - Programme
Development (OECD) - Development Assistance policy and procedures manual: programme opera-
Committee, Working Party on Aid Evaluation – tions, chapter 6. Programming tool, section 06. Inte-
Glossary of Key Terms in Evaluation and Results Based grated Monitoring and evaluation plan (http://www.
Management, 2002 (http://www.oecd.org/findDocum ceecis.org/remf/Service3/unicef_eng/module2/part1.
ent/0,2350,en_2649_34435_1_119678_1_1_1,00.html, html, accessed on 14 May 2013).
accessed on 14 May 2013).
United Nations Children's Fund (UNICEF) - Understand-
Organization for Economic Co-operation and Develop- ing Results Based Programme Planning and Manage-
ment (OECD) – Development Assistance Committee, ment, Evaluation Office and Division of Policy Plan-
Principles for Evaluation of Development Assistance, ning, 2003 (http://www.unicef.org/evaluation/files/
1991, reprinted in 2008 (http://www.oecd.org/datao- RBM_Guide_20September2003.pdf, accessed on 14
ecd/31/12/2755284.pdf, accessed on 14 May 2013). May 2013).

Organization for Economic Co-operation and Results-Based Management in the United Nations
Development (OECD) – Development Assistance Development System: Progress and Challenges (http://
Committee, Evaluation Quality Standards, 2006 www.un.org/esa/coordination/pdf/rbm_report_10_
(http://www.oecd.org/document/30/0,3343, july.pdf , accessed on 14 May 2013).
en_21571361_34047972_38903582_1_1_1_1,00.html, ac-
The World Bank – Ten Steps to Results-Based Monitor-
cessed on 14 May 2013).
ing and Evaluation System, A Handbook for Develop-
United Nations Development Programme (UNDP) – ment Practitioners; Jody Zall Kusek, Ray C. Rist, 2004.
Handbook on Monitoring and Evaluation for Results,
The World Bank – Annual Report on Operations Evalu-
2002 updated in 2009.
ation - Appendix A: Overview of Monitoring and Evalu-
United Nations Development Programme (UNDP) – ation in the World Bank, 2006 (http://web.worldbank.
RBM in UNDP: Selecting Indicators, 2002 (http://www. org/WBSITE/EXTERNAL/EXTOED/EXTANNREPOPEEVA
undp.org/cpr/iasc/content/docs/MandE/UNDP_RBM_ L/0,,contentMDK:21123422~menuPK:4635630~pagePK
Selecting_indicators.pdf, accessed on 14 May 2013). :64829573~piPK:64829550~theSitePK:4425661,00.html,
accessed on 14 May 2013).
UNEG, Ethical Guidelines for Evaluation (2007) (http://
www.unevaluation.org/ethicalguidelines, accessed on The World Bank – Sourcebook on Emerging Good
14 May 2013). Practices - Emerging Good Practice in Managing For
Development Results, 2006 (http://www.mfdr.org/
United Nations Evaluation Group (UNEG) - Principles of
Sourcebook.html, accessed on 14 May 2013).
Working Together, 2007 updated in 2012 (http://www.
unevaluation.org/papersandpubs/documentdetail. World Health Organization (WHO) – 11th General
jsp?doc_id=96, accessed on 14 May 2013. Programme of Work 2006-2015 (http://whqlibdoc.who.
int/publications/2006/GPW_eng.pdf, accessed on 14
United Nations Evaluation Group (UNEG) – Standards
May 2013).
for Evaluation in the UN System, 2005, updated in 2012
(http://www.uneval.org/normsandstandards/index.
jsp?doc_cat_source_id=4, accessed on 14 May 2013).

United Nations Evaluation Group (UNEG) – Norms for

Evaluation in the UN System, 2005, updated in 2012
(http://www.uneval.org/normsandstandards/index.
jsp?doc_cat_source_id=4, accessed on 14 May 2013).
TDR PERFORMANCE ASSESSMENT FRAMEWORK - PART 3 29

Annexes
Annex 1 – Engaging stakeholders in the Development of this framework
Annex 2 – Reporting
Annex 3 – TDR Monitoring and Evaluation Matrix
30 ANNEX
PART
1 - TDR
3 - TDR
PERFORMANCE
PERFORMANCE
ASSESSMENT
ASSESSMENT
FRAMEWORK
FRAMEWORK

ANNEX 1 Engaging stakeholders in

the development of this framework

The development of the initial TDR Performance Switzerland; International Development Research
Assessment Framework was conducted through a Centre, Canada; Academy for Educational
collective effort led by Drs Beatrice Halpaap and Development, USA; Department for International
Fabio Zicker involving TDR staff and stakeholders. Development, UK; and the Swedish International
Internal and external consultations helped to Development Cooperation Agency, Sweden.
develop ownership, capture the perspectives of • The Secretariat for the Global Strategy and Plan
various stakeholders and enhance harmonization of Action for Innovation, Public Health and
with international practices. Intellectual Property.
• World Intellectual Property Organization.
A small internal working group representing TDR's
strategic functions was established in order to
An external advisory group with representation
assist in the development of an initial draft and
from research and training funding programmes,
subsequent revisions. This group was supported by
development agencies, research institutions
four additional internal groups to help develop key
in disease endemic countries and individual
performance indicators which are used to measure
researchers, met in December 2009 to review
and reflect TDR's performance. The groups worked
the TDR Performance Assessment Framework
in consultation with the following stakeholders:
and made recommendations to TDR's Director.
• WHO research programmes, including the The external advisory group was composed of the
Initiative for Vaccine Research, Research Policy following individuals:
and Cooperation Department; WHO Ethics,
• Dr Alejandro CRAVIOTO, Executive Director,
Equity, Trade and Human Rights Department;
International Centre for Diarrhoeal Disease
and the Special Programme of Research,
Research (ICDDR,B), Dhaka, Bangladesh.
Development and Research Training in Human
Reproduction, co sponsored by UNDP, UNFPA, • Professor Alan FAIRLAMB, Professor and Head,
UNICEF, WHO and the International Bank for Division of Biological Chemistry and Drug
Reconstruction and Development. Discovery, School of Life Sciences, Wellcome
Trust Biocentre, University of Dundee, Dundee,
• WHO regional offices for Africa, the Americas,
UK.
the Eastern Mediterranean, Europe, South-East
Asia and the Western Pacific. • Dr Linda KUPFER, Acting Director Division
of International Science Policy, Planning &
• TDR co-sponsors' evaluation and/or policy
Evaluation, NIH/Fogarty International Centre,
offices: UNICEF, UNDP (Global Environment
Bethesda, USA.
Facilities) and the World Bank.
• Professor Mary Ann D LANSANG (Chair),
• Research institutions including the International
University of the Philippines, Manila, Philippines;
Centre for Medical Research (CIDEIM),
seconded as Director, Knowledge Management
Colombia; the Trypanosomiasis Research Center,
Unit, Global Fund to Fight AIDS, Tuberculosis
Kenya; International Centre for Diarrhoeal
and Malaria, Geneva, Switzerland.
Disease Research (ICDDR,B), Bangladesh;
Fundação Oswaldo Cruz (FIOCRUZ), Brazil; and • Ms Jo MULLIGAN, Health Advisor, Department
University of Dundee, UK. for International Development, London, UK
• Research funding institutions and development • Dr Zenda OFIR (Rapporteur), Evaluation
agencies, including the Wellcome Trust, UK; Specialist, Johannesburg , South Africa.
Fogarty International Center, USA; National • Dr Claude PIRMEZ, Vice-President of Research
Research Foundation, South Africa; the Global and Reference Laboratories, Fundação Oswaldo
Fund to Fight AIDS, Tuberculosis and Malaria, Cruz (FIOCRUZ), Rio de Janeiro, Brazil.
TDR PERFORMANCE ASSESSMENT FRAMEWORK - ANNEX
PART 3 1 31

• Dr Ana RABELLO, Laboratory of Clinical

Research, Leishmaniasis Reference Centre,
Centro de Pesquisas René Rachou, Fundação
Oswaldo Cruz (FIOCRUZ), Belo Horizonte,
Brazil.
• Dr Daisy SELEMATSELA, Executive Director,
Knowledge Management & Evaluation, National
Research Foundation, Pretoria, South Africa.
• Dr Val SNEWIN, International Activities
Manager, The Wellcome Trust, London, UK.
• Dr David ZAKUS, Senior Program Specialist,
Governance Equity & Health Program,
International Development Research Centre,
Ottawa, Canada.

The PAF is revised regularly, taking into

consideration specific donor requirements.
This current revision led by Drs Michael Mihut
and Béatrice Halpaap benefited from in-depth
discussions and consultation with:
• Claire Kairys, Junior Professional Associate,
Health, Nutrition, & Population, The World
Bank.
• Beth Scott, Health Advisor, Human Development
Team, Research & Evidence Division [RED],
UK Department for International Development
(DFID).

The final draft of the framework, developed with

feedback from various consultations, was reviewed
and endorsed by TDR's governing bodies.
32 ANNEX
PART
2 - TDR
3 - TDR
PERFORMANCE
PERFORMANCE
ASSESSMENT
ASSESSMENT
FRAMEWORK
FRAMEWORK

ANNEX 2 Reporting
Types of report Scope Frequency Target Audience

Each team presents:

• progress on technical and financial
implementation towards expected results;
Quarterly team
planned activities Quarterly TDR staff
progress report
• follow-up on JCB and STAC
recommendations
• follow-up on significant risks action plans.

Progress towards the achievement of the Annually or Grant donors;

Grant progress
grant/contract objectives (technical and finan- as required TDR manage-
report
cial). If relevant, specific plans and budget for by grant ment; Related
(grants/contracts)
upcoming years. agreements SAG, if relevant

Annual consolidation of the Programme's STAC; resource

Work area
progress towards the achievement of Annually contributors;
Annual Report
objectives in each work area. stakeholders

Progress towards the achievement of expect-

TDR results report TDR manage-
ed results, application of TDR core values
ment; STAC; JCB;
(Published on the and efficiency in management. This report
Annually resource con-
TDR website) includes a description of performance using
tributors; stake-
key performance indicators and related quali-
holders
tative description.

TDR annual report TDR manage-

Provides TDR contributors and stakeholders ment; STAC;
(Published on the
with an update on progress, strategic Annually JCB; resource
TDR website)
direction and planned activities. contributors;
stakeholders

Analysis of results achieved by the WHO

WHO Programme secretariat, as measured against the expected
budget performance results for the biennium reviewed, is provided Biennial, plus
assessment report by the WHO Planning, Resource Coordination mid-term WHA
(Published on the and Performance Monitoring Department. review
WHO website) The report is reviewed by the World Health
Assembly.

Programme-wide review commissioned by JCB

External Programme Every
which also establishes the terms of reference JCB
review report 5–7 years
of the review.

TDR's operational, administrative and financial

WHO internal audit WHO Director-
procedures and practices are reviewed by a Ad hoc
report General; WHA
WHO internal auditor.

TDR's financial report and statement are

TDR financial report Annually WHA, JCB
certified by the WHO financial controller.
ANNEX 3. TDR monitoring and evaluation matrix

Target Frequency of
Expected results Key performance indicators Source of data
(2017) measurement

Technical expected results

OUTCOME: 1. Number and proportion of new/improved solutions, 30 Publications, annual reports, Measured annually,
Infectious disease implementation strategies or innovative knowledge ≥ 75% interviews, surveys cumulative over 6 years
knowledge, solutions successfully applied in developing countries.
and implementation
strategies translated
2. Number of tools and reports that have been used to inform
into policy and practice
policy and/or practice of global/regional stakeholders or Publications, annual reports, Measured annually,
in disease endemic 7
major funding agencies. interviews, surveys cumulative over 6 years
countries
TDR PERFORMANCE ASSESSMENT FRAMEWORK - ANNEX
PART 3 3

MAIN OUTPUT: 3. Number and proportion of new/improved solutions, imple-

New and improved mentation strategies or innovative knowledge developed
35 Publications, annual reports, Measured annually,
solutions and in response to requests from WHO control programmes
≥87% interviews, surveys cumulative over 6 years
implementation and/or disease endemic countries.
strategies that respond
to health needs of
4. Number of peer-reviewed publications supported by TDR
disease endemic ≥150/year
and percentage published in open access journals. Bibliographic analysis Measured annually
countries developed 100%

FEEDER OUTPUTS: 5. Number and evidence of new/improved tools, case-man-

High quality agement, control or implementation strategies generated Publications, annual reports, Measured annually,
40
intervention and through TDR facilitation with systematic quality review by interviews, surveys cumulative over 6 years
implementation external committees.
research evidence
6. Proportion of peer-reviewed publications supported by
produced ≥70% Bibliographic analysis Measured annually
TDR with first author from DEC institutions.
33
 34

ANNEX 3 TDR monitoring and evaluation matrix

Target Frequency of
Expected results Key performance indicators Source of data
(2017) measurement

Enhanced research and 7. Number of DEC institutions and/or networks demonstrat-

knowledge transfer ing expanded scope of activities and/or increased funding Publications, annual reports, Measured annually,
5
capacity within disease from alternative sources thanks to TDR support. interviews, surveys cumulative over 6 years
endemic countries
8. Number of TDR grantees/trainees and proportion demon- Database, interviews, Measured on cohorts
strating career progression and/or increased scientific 150
surveys, social networks 3-5 years after training
productivity. ≥80%
ended

Key stakeholders in 9. Number and evidence of research-related agendas,

Publications, annual reports, Measured annually,
disease endemic countries recommendations and practices agreed by stakeholders 9
interviews, surveys cumulative over 6 years
engaged in setting the at global, regional or country level.
research agenda and
ensuring research reflects 10. Proportion of TDR outputs produced with key DEC
Publications, annual reports,
ANNEX

stakeholder active involvement. 100% Measured annually

their needs interviews, surveys
PART
3 - TDR

Application of core values

3 - TDR

11. Proportion of TDR grants/contracts awarded to

Equity WHO financial data,
institutions or individuals in DECs (total count and 75% Measured annually
TDR database
Social and economic: total dollar amount).
PERFORMANCE

12. Proportion of experts from DECs on TDR advisory WHO financial data,
PERFORMANCE

committees. 60% Measured annually

TDR database

Gender: 13. Proportion of women among grantees/contract recipients WHO financial data,
50% Measured annually
ASSESSMENT

(total count and total amount). TDR database

ASSESSMENT

14. Proportion of women on TDR advisory committees. WHO financial data,

50% Measured annually
TDR database

15. Proportion of women as first author of peer-reviewed Bibliographic analysis,

FRAMEWORK

50% Measured annually

publications supported by TDR (within a calendar year). TDR database
FRAMEWORK
 16. Resources leveraged as direct contributions (co-funding, Quarterly and annual
Effective partnerships services or in-kind) to TDR projects (examples). tbd reviews, project progress Measured annually
reports, interviews

17. Number of effective public health tools and strategies Measured annually, two
Sustainability of developed which have been in use for at least two years. 67 Annual reports, publications
outcomes years after adoption

18. Proportion of project final reports found satisfactory Committee meeting minutes
Quality of work by peer- review committees. >80% Measured annually
and recommendations

Management performance

19. Percentage of approved biennial budget successfully TDR JCB-approved budget, Measured in the second
Effective resource funded. ≥100%
WHO financial data year of each biennium
mobilization

20. Percentage of income received from multi-year WHO financial data, TDR Measured in the second
tbd
agreements. agreements year of each biennium
TDR PERFORMANCE ASSESSMENT FRAMEWORK - ANNEX
PART 3 3

21. Percentage of staff workplans and performance reviews

Effective management (including personal development plan) completed on time. ≥90% ADG report Measured annually

22. Proportion of expected results on track. Quarterly and annual

≥80% reviews, project progress Measured annually
reports, interviews

23. Proportion of significant risk management action plans Quarterly reviews, risk moni-
≥80% Measured annually
that are on track. toring tool
35
36 PART 3 - TDR PERFORMANCE ASSESSMENT FRAMEWORK
 TDR/STRA/14.2

This Framework is a key element in the implementation of TDR’s

2012-2017 strategy. It has the following objectives:
• Promote continuous performance improvement through orga-
nizational review, learning and informed decision-making.
• Enhance accountability to stakeholders, including beneficia-
ries, partners and resource contributors.
• Ensure strategic relevance and coherence of TDR’s activities
to meet the aspirations expressed in the vision, mission and
strategy.
• Ensure TDR’s performance assessment is harmonized and con-
sistent with international practices.

TDR/World Health Organization

20, Avenue Appia
1211 Geneva 27
Switzerland

Fax: (+41) 22 791-4854

tdr@who.int
www.who.int/tdr

TDR, the Special Programme for Research and Training in Tropical

Diseases, is a global programme of scientific collaboration that
helps facilitate, support and influence efforts to combat diseases
of poverty. TDR is hosted at the World Health Organization
(WHO), and is sponsored by the United Nations Children’s Fund
(UNICEF), the United Nations Development Programme (UNDP),
the World Bank and WHO.

World Bank