A framework to measure

the impact of investments in health research

OECD Blue Sky II Forum, September 25, 2006

Alan Bernstein, Vern Hicks, Peggy Borbey, Terry Campbell

Abstract
This paper describes the approach taken by the Canadian Institutes of Health Research
(CIHR) to develop a framework and indicators to measure the impact of health research.
The development process included national and international consultations. Key
methodology challenges and measurement requirements were identified. The framework
that has resulted from this process includes definitions of key concepts, methodology
guidelines, identification of the different stakeholders for impact information and the
individual concerns of each stakeholder group. Indicators are classified within five
categories that encompass a broad range of impacts. Sources of information and issues
in attributing research impacts are discussed. An analysis of issues suggests that impact
measurement and performance measurement are complementary activities, with the
former focused on broad impacts of the health research sector and the latter on the
degree of success achieved by funding agencies in contributing to the process of
knowledge development and uptake.

Introduction
Accountability and value received for the use of public funds have become high priorities
for governments around the world. Quantifying the value of publicly funded health
research is a challenge for many countries. This paper describes the approach taken by
the Canadian Institutes of Health Research (CIHR), Canada’s national health research
funding agency, to develop a framework to measure the impact of its investments in
health research and to begin to identify robust indicators that can be used to establish
benchmarks and track progress through time.

CIHR was created in 2000 with a broad mandate:

‘to excel, according to international standards of scientific excellence, in the
creation of new knowledge and its translation into improved health, more effective
health services and products, and a strengthened Canadian health care system.’

The Government of Canada has increased investments in health research from $289
million in 1999-2000 (the last year of CIHR's predecessor, the Medical Research
Council) to $700 million in 2005-2006. A description of the CIHR operating model is
shown in the text box on page 2. CIHR has taken a proactive approach to foster
research that is relevant to important health issues and to anticipate the future demand
for health research. In addition, it has committed to actively encouraging collaboration in
multidisciplinary research. CIHR consists of a central office that sets strategic directions,
engages in national initiatives and administers peer review for grant funding; and 13
Institutes, each with a unique research focus. Each Institute is headed by a scientific
director and hosted by an academic or research institution. Each Institute provides a
nexus for researchers in specific fields to collaborate and identify research priorities.
CIHR funds both investigator-initiated (open) and strategic research. The Institutes play

1
a pivotal role in identifying strategic research priorities and launching funding
opportunities in priority areas.

Background and Development of the Framework

While CIHR now regularly evaluates its individual programs and has documented the
benefits of individual CIHR-funded research projects, there has not yet been any
comprehensive or overall assessment of their outcomes in terms of scientific, social and
economic impacts. In early 2005, CIHR convened a group of international and Canadian
experts to review the present state of knowledge about measuring the impact of health
research and to provide advice on the creation of a conceptual framework. The
framework that resulted from this process was vetted at a consensus conference of high-
level stakeholders in May, 2005. The framework identifies the key expected impacts of
health research as well as a set of preliminary indicators and processes for evaluating
impact. It includes dimensions that address the concerns of identified stakeholder
groups and identifies potential sources of information.

The framework and draft indicators were published on the CIHR website in January
2006. 1 Since that time CIHR has been experimenting with populating some of the
indicators. This has led to some refinements and updates, which are included in this
paper.

This paper provides a brief description of the conceptual framework and then discusses
specific indicators. The discussion centres on the potential use of each indicator,
availability of data and approaches that can be used to develop data that are not
presently available. Illustrative examples for specific indicators are included, where
possible. The next section includes a brief review of methodology challenges in
measuring impact. The third section describes the CIHR framework and the last section
discusses indicators chosen for the initial years of an ongoing development process.

CIHR Operating Model

CIHR Institutes

Aboriginal Peoples’ Health Human Development, Child and Youth

Aging Health
Cancer Research Infection and Immunity
Circulatory and Respiratory Health Musculoskeletal Health and Arthritis
Gender and Health Neurosciences, Mental Health and
Addiction
Genetics
Nutrition, Metabolism and Diabetes
Health Services and Policy Research
Population and Public Health

Research Themes (Pillars)

Biomedical research Health services and policy research

Clinical research Population and public health research

2
 Core Outcomes

Research & the creation of new knowledge

Translating research knowledge into applications
Training & career development

Mandate – overall goals

Improved health for Canadians

More effective health services and products
Strengthened Canadian health care system

Objectives of health research funding and measurement challenges

Experts from funding agencies in the UK (Wellcome Trust) and Australia (National
Health and Medical Research Council) participated in the expert review, which identified
similarities in the major objectives of funding agencies. Although the language used to
describe objectives varies, the three most important objectives for the UK, Australian
and Canadian funding agencies are shared: knowledge creation, knowledge translation
and human resource development. The first two of these objectives also align with
objectives of the US National Institutes of Health.

There was also agreement about the main challenges in identifying the impacts of health
research.
• Linkages between health research outputs and outcomes are difficult to trace
where knowledge develops incrementally over time.
• Health research outcomes are often intangible, e.g. improved health, longevity
and the capacity to achieve human potential.
• Attribution of credit for research accomplishments can be difficult as impacts
often result from a number of research projects carried out either collaboratively
or independently in different countries.
• Priorities differ across stakeholder groups – for example, commercial returns are
important to industry and government but may play a subordinate role in the
value systems of researchers and the public.

A number of methodology requirements for an appropriate framework and indicators

were identified:
• New methodologies for measuring impact should build on existing performance
measurement work in order to produce a continuum between the two types of
activity.
• A variety of approaches and measures are required to address CIHR’s mandate
and research themes.
• Methodologies should consider short term and long term impacts of research.
• Where appropriate, methodologies should distinguish between social and
commercial rates of return.

3
 • Indicators should be updated regularly.
• Involvement of other research funding agencies, both domestic and international,
is desirable in order to maximize insight, achieve efficiencies available by pooling
efforts, and allow for comparative analysis.

Framework Structure
The impact framework builds on a base of ongoing performance measurement activities
that measure success in achieving corporate objectives. New indicators will seek to
understand how research contributes to health, social and economic progress.
Dimensions of the framework include definitions of key concepts, methodology
guidelines (above), identification of the stakeholders for impact information and the
concerns of each stakeholder group.

Stakeholders
The stakeholders for health research impact information are summarized in Table 1.
Some concerns are unique to particular stakeholders while others are shared across
many or all sectors. For example, academic excellence is a concern found mainly in the
higher education sector while improved health status is a concern shared by all
stakeholders.

The framework first identifies generic types of impact information to address stakeholder
concerns and then seeks to identify indicators to measure benchmarks and progress.
For example, heath status impacts will be oriented to a system of health status indicators
that has been developed during the last decade and will be tracked annually. 2,3

Table 1
Cross Classification of Stakeholders for Impact Information and Concerns

Stakeholders Concerns
Higher education sector • Academic Excellence
• Knowledge production
• Capacity building
Health professionals & • New treatments and diagnostic potential
administrators • Productivity of resources used in health systems
Society • Improved health status
• New treatments for disease
• Response to public health threats
• Efficiency and sustainability
Business sector • Commercial potential
Government • Public health and responses to health threats
• Health status
• Contribution to macroeconomic growth and
productivity
• Efficiency and sustainability of both public and
private health systems

4
Types of impact
There are five categories within which to measure impact in the framework. These
categories were adapted from the Payback Framework developed by Dr. Martin Buxton
and colleagues at the Health Economics Research Group, Brunel University, UK. Dr.
Buxton was an active participant in development of the CIHR framework. The impact
categories, as adapted for the CIHR framework, consist of:
• Knowledge Production, usually measured through contributions to the scientific
literature.
• Research targeting and capacity, in which targeting is defined as the ability of
research projects to inform subsequent research agendas and methods and
target future research. Contributions to capacity can be defined as development
of research skills, including the capability to use existing research.
• Informing Policy includes impacts of research in the areas of clinical,
administrative and government policy.
• Health and health sector benefits measure impact in terms of advances in
prevention of ill health, improvements in public health and treatment.
• Economic benefits are defined within four categories developed in a recent
literature review by Dr. Buxton and colleagues. 4 The economic categories define
benefits to the economy from commercialization of discoveries, direct cost
savings to the health care system, human capital gains and the value of life and
health.

Social benefits are not included as a separate category. The framework categories of
health and health sector benefits and economic benefits have been identified as
important measures of social benefit. 5,6

Indicators
Indicators within each framework category are shown in Table 2. Their merits, data
availability and approaches to developing data sources are subsequently discussed.

5
 Table 2
Preliminary Indicators of Health Research Impact and Potential Sources of Information

Knowledge Production
1. Number and impact of publications resulting from CIHR- Bibliometric Studies
supported research. End-of-Grant reports
2. Citation Impact Analysis.

Research Targeting and Capacity

1. Extent to which Institutes have influenced the research, Evaluations every 3 – 5 years.
policy and/or practice agendas in their communities. End-of-Grant reports
2. Percentage of Research Chairholders attracted or retained
in Canada. Canada Research Chairs database.
3. Number and types of PhD graduates in Canada by year
4. % of PhD graduates in Canada planning postdoctoral Statistics Canada
fellowship or research associateship in health
Informing Policy
1. Public policies influenced by CIHR and CIHR-funded Case studies.
research
End-of-Grant reports
2. Clinical practice guidelines by disease area influenced by
CIHR-funded research. Evaluations every 3 – 5 years.

Health and Health Sector Benefits

Public health:
Strategic research initiatives and their outcomes. Case studies

Health impacts: End-of-Grant reports

Impact of health research on Potential Years of Life Lost
(PYLL) for target disease categories (e.g. cancer, circulatory Statistics Canada data.
disease). Special studies to establish links to
health research.
Economic Impacts
Commercialization: End-of-Grant reports
1. Number and nature of patents, spin-off companies and
licenses for intellectual property (IP) generated from CIHR- Statistics Canada
funded research.
2. Income from IP commercialization. Statistics Canada
3. Case studies and follow-up surveys of commercial use of
Special studies
research funded by CIHR’s commercialization programs.

Cost savings:
Estimates of the value of high impact innovations developed Technology assessment. Special studies
through health research in Canada.
Human capital: Collaborative studies with Health Canada
Reduction in productivity lost through illness or injury due to and Statistics Canada
innovations from research.

6
Several of the indicators identified in Table 2 will need to be developed further through
special studies as the information required is not routinely collected. These studies will
vary in complexity and the degree of collaboration required from other agencies. Case
studies of strategic research initiatives, for example, can be carried out by CIHR
Institutes at suitable intervals following the completion of a strategic research project.
Other indicators will be addressed in end-of-grant surveys and external reviews. Still
other indicators will need a research oriented approach. The immediate importance of
the approach illustrated in Table 2 is to clarify the types of information that are desirable,
identify how they fit within a conceptual framework and identify potential sources of
information.

Attribution Issues
Credit for the direct benefits of research can rightly be claimed by individual researchers
or research teams that develop new knowledge. Research funding plays an enabling
role in knowledge creation and its subsequent impacts. The question of how much credit
for beneficial impacts can be claimed by funding agencies involves special challenges.
Funding agencies can assume either an active or passive role in terms of shaping the
national research agenda. CIHR plays a very active role, using its resources and insights
to encourage research that has a high priority from scientific or national policy
perspectives. CIHR allocates approximately 30% of its research funding for strategic
competitions

CIHR is not the only source of health research funding in Canada, a fact that introduces
additional complexity to efforts to identify impacts from funded research activities (see
text box page 8).

Finally, many of the impacts of health research (for example changes in health status)
are long-term and are the results of a combination of factors, including application of
new research knowledge and, for example, environmental factors.

In view of these issues, it seems impractical to identify the impacts of any single agency,
except when dealing with specific examples. Instead, agencies have tended to establish
performance measures that track success in achieving intermediate outputs rather than
final impacts. Key CIHR performance measurement indicators are summarized in
Table 3.

7
 Health Research Funding in Canada

Canada has three federal granting councils to fund and promote research. CIHR is
responsible for health research, although some health research funding is also provided
by the other two granting councils and national agencies dedicated to health or
innovation1. Funding for research infrastructure and a share of indirect costs are provided
by two separate agencies.

Most Canadian provinces also have health research funding agencies. The not-for-profit
(NFP) sector includes many philanthropic and disease-specific agencies. Funding
agencies from all three sectors often collaborate to sponsor research within their
mandates.

Research grants from the three sectors do not include investigator salaries or indirect
costs and most of these costs are borne by research institutions.

The distribution of federal, provincial and NFP health research funding by sector is shown
in the accompanying graph.2 Estimates of gross expenditures for research and
development (GERD), as defined by Statistics Canada, include these three sectors as well
as expenditures or in-kind contributions from higher education and expenditures by
business enterprise and foreign sources.

___________________
1
Other national agencies that provide some health research funding are Genome Canada Ltd.,
Health Canada, the National Research Council and the Canadian Health Services Research
Foundation.
2
Source: Balance of Funding in Canadian Health Research and Future Funding Requirements.
CIHR, forthcoming

Distribution of Health Research Funding 2003-04

Public & NFP Sectors

NFP
10%
Provincial CIHR
24% 36%

Other
federal
30%

Source: CIHR

8
 Table 3
CIHR Performance Indicators

Indicator topic Variables

Distribution of budget Expenditure by type of funding program by
expenditures research area, and operating expenses
Institute evaluations Relevance, effectiveness and delivery of Institute
mandates and priorities.
Research grants Number, value and success rates of grants by
research area.
Investigators supported Number of investigators by type of grant or
award.
Collaboration Percent of grants that feature interdisciplinary
research and size of investigator team.
Capacity building Number, value and success rates for training and
salary awards by research area.
Commercialization Awards and amount of funding by
commercialization program. For the POP
program annual reports provide #patents, #IP
licensed and companies formed.
Partnerships Number of partners and value of partner
contributions by sector.
Knowledge Translation Percent of grants that include or undertake
knowledge translation in their research activities
Notes: Indicators are obtained from CIHR administrative databases, evaluations, surveys of
stakeholders and surveys of funding recipients.
Source: CIHR Report to International Review Panel, February 2006.

Impact Indicators
The impact indicators in the CIHR framework (Table 2) include both performance
measurement indicators and indicators that measure results achieved by the Canadian
health research community. Each type of indicator is discussed below, along with
potential sources of information for future activities.

Knowledge Production
Number and impact of publications resulting from CIHR-supported research. CIHR is
designing an end-of-grant report to measure knowledge production specifically from
grant funds. CIHR currently has data on publications produced by CIHR-funded
researchers – recognizing that a portion of these would not have been directly
attributable to CIHR funding. General publication trends for health research in Canada
overall is being used until more CIHR-specific data is available. A recent study in the UK
found that Canada ranks sixth among the nations of the world in its share of total
publications and citations in 8,000 journals between 1993 and 2003. 7,8 Indicators are
shown below.

9
Indicator Definition Results
Canadian publications / Total publications Canada produces about 5% of the world’s
health research literature at an annual rate
of 14,000-15,000 publications. In 2004 this
was roughly 6.7% of total health
publications in the G8 nations. 9
Citation intensity (Citations/GDP) Canada is among the top 8 countries in
compared to wealth intensity citation intensity, all of which are above the
(GDP/population) curve that defines citation intensity as a
function of wealth intensity. Canada and
the UK are the only two G8 countries
above the curve. 10

The two indicators above vary in terms of sophistication. We would expect the second to
be a more powerful measure of scientific output since it takes into account a country’s
level of economic development. But sophistication can come at the cost of reduced
comprehension to a broad audience. More sophisticated bibliographic indicators can be
used for an academic audience, for example, the ISI Journal Citation Reports impact
indicators 11 for specific journals to weight publications.

Research Targeting and Capacity

Extent to which Institutes have influenced the research, policy and/or practice agendas
in their communities. Case studies will be the most usual form of analysis for this
indicator. Follow-up surveys or focus groups can be designed to determine how well
specific research has informed policy. These activities will be complementary to ongoing
efforts to determine how effective knowledge translation has been in affecting practice or
policy.

Researchers attracted or retained in Canada. Sources of information include Statistics

Canada surveys and periodic evaluations of the Canada Research Chairs program
(CIHR is responsible for 700 chairholders under the program).

10
Indicator Definition Results
Extent to which Institutes have influenced A recent survey of funded researchers
the research, policy and/or practice found:
agendas in their communities • broad agreement that the Institute with
which they are affiliated has
contributed to developing capacity in
terms of people (84%), the research
environment (82%) and research
excellence (91%),

• 93% of funded researchers believe

CIHR has succeeded in setting a
national research agenda and that this
agenda is the appropriate one.
Number and types of PhD graduates in Canada ranks fifth for proportion of the
Canada by year. population with PhDs1.

% of PhD graduates in Canada planning

A Statistics Canada survey (July 31, 2003
postdoctoral fellowship or research
to June 30, 2004 found that 64% of
associateship in health
graduates in the life sciences planned to
pursue a postdoctoral fellowship2

1
2004, King, Scientific Impact of Nations (average for 1998-2000).
2
Data taken from Gluszynski, Tomasz and Valerie Peters, Survey of Earned Doctorates: A
Profile of Doctoral Degree Recipients, Statistics Canada and Human Resources Development
Canada, p.14.

Informing Policy
The two indicators listed in the policy section of the Framework (policies and clinical
practice guidelines affected by CIHR and CIHR-funded research) will need to be put in
context. In other words, what policies were affected and how important were they in
public policy formation? Similar questions can be asked regarding clinical practice
guidelines. The box below describes two recent examples of CIHR activities that have
informed important medical policy issues in Canada.

Indicator Definition Results

Evidence based standards for medical CIHR commissioned a series of research
waiting times syntheses in 2005 to establish evidence-
based benchmarks for medically
acceptable wait times in five priority areas
that were agreed to in the 2004 federal,
provincial, territorial Health Accord.
Scientific and public policy guidelines • Guidelines for Human Embryonic Stem
Cell Research
• Privacy and Confidentiality in Health
Research

11
The types of information in the examples above are qualitative indicators of success in
meeting policy requirements. Success in changing policy or practices can be viewed as
a form of intermediate outcome. Final outcome can be defined at successively higher
levels depending on the policy issue that is being addressed. For example, adoption of
minimum wait times achieves an important public policy objective. Beyond that there are
concerns about the degree of compliance by regional institutions with minimum wait
times and ultimately the extent to which wait-time standards affect overall health status.

Health and Health Sector Benefits

Health and health sector benefits fall into two broad groups in the framework: public
health and health impacts. The impacts of research in public health can often be related
to strategic initiatives. For example, at the height of the SARS epidemic in 2003, CIHR
launched a three-part strategic research initiative, which involved (1) funding research
on the causes and consequences of SARS; (2) funding research on Canada’s health
system preparedness and response to the SARS outbreak; and (3) the creation of the
Canadian SARS Research Consortium to coordinate, promote and support Canadian
research on SARS and newly emerging infectious diseases. Success in containing
SARS was achieved through this balanced and coordinated approach and through
adherence to sound public health principles.

The second dimension of health and health sector benefits includes impacts on both life
expectancy and quality of life through medical advances. Attribution of credit for these
impacts can be even more difficult than for public health advances. Successful treatment
results from a chain of circumstances, beginning with: 1) new knowledge about how to
prevent or treat illness and proceeding through 2) the development of optimal treatment
guidelines, 3) training of professionals in the use of new innovations and 4) widespread
adoption of best practice protocols for prevention and treatment. The latter three steps of
this process involve knowledge translation, a major priority in health research.

The four step process illustrates two key points in understanding the impacts of health
research on population health:
1. New knowledge will not, by itself, have widespread impacts on health.
2. Research is pervasive – it plays a key role in each step of the process that links
discovery to gains in health status.
In these circumstances many actors will be able to claim some credit for success. An
attempt to apportion credit to any agent, or even any one step in the process, seems
inappropriate since no single link in the chain would, by itself, be sufficient to bring about
the end result. It is typically a complex process involving multiple funders, multiple
institutions, and long time lines.

This reality calls for an approach to impact measurement that focuses on health status
gains and their causes rather than specific contributions. There is a growing realization
that those responsible for research and knowledge translation should ‘celebrate’ success
rather than attempt to divide up credit for it.8

The logical approach to impact measurement in these circumstances is to seek ways to

measure health gain and to describe the processes that led to that gain. In this context,
individual organizations, such as CIHR, can point to the fact that they have effectively
contributed to health status improvements both by funding and encouraging appropriate

12
research that created the right environment to either advance and/or implement a
necessary research discovery. In effect, organizational claims to impact include
demonstrating both the ultimate impacts of new knowledge and the organization’s
effectiveness in contributing to that process. This approach brings together impact
measurement and performance measurement.

The health impact measure suggested in Table 2 is Impact of health research on

Potential Years of Life Lost (PYLL) for target disease categories. This type of indicator is
not suggested as a single best measure; the choice is based on more pragmatic
considerations. PYLL is an objective measure and it is part of an ongoing series of
health status indicators compiled by Statistics Canada. Approximately 460,000 potential
years of life were lost to cancer and 250,000 to circulatory disease in 2001 12 , indicating
that a large potential exists to reduce losses through mortality from these conditions. In a
more advanced evolution of health status measurement, it would be desirable to add a
dimension of quality as well as life expectancy - e.g. health adjusted life expectancy
(HALE) or quality adjusted life years (QALYs). Widespread population-based
measurement of these indicators has yet to be developed.

Reductions in PYLL can be related to advances in knowledge although there will usually
be a margin of uncertainty, especially where several factors contribute (e.g. prevention,
non-health determinants and treatment). In the case of cardiovascular disease, CIHR is
considering a qualitative approach used in the UK12 called a witness seminar or narrative
approach, which brings together a range of experts to discuss the key influences that
contributed to an advance in a health outcome.

Economic Impacts
The classification of economic impacts in the CIHR Framework follows a categorization
scheme in a recent critical review of the economic benefits of research by Buxton and
colleagues3. Economic impacts are closely related to the concept of return on
investment. In the case of funding for health research, however, the two concepts differ
in important respects.
• The economic benefits from research tend to be diffused throughout society, for
example enhanced earnings and productivity as the result of gains in workforce
health.
• In many cases, research produces public goods, which are not patented and not
traded in economic markets – enhanced understanding of the health benefits of
exercise and diet are examples, as are new medical and surgical procedures
adopted into publicly financed health care systems.
• Where discoveries or intellectual property results in patented products and spin-
off companies, financial benefits will usually accrue to the researchers who made
the discoveries and the institutions that hosted the research [in Canada, usually
public universities or hospitals], as well as the downstream industries that use
them and persons employed in spin-off companies. Research funding agencies
in Canada receive no direct financial benefit from successful commercialization
of intellectual property.
These considerations argue for an approach to measuring economic impacts that is
conceptually similar to the approach recommended for health impacts: measure the
overall economic impacts of research and then demonstrate that a funding agency has
contributed effectively by encouraging research.

13
Commercialization
Statistics Canada conducts surveys of intellectual property commercialization 13 and
biotechnology. 14 The latter are part of an effort by the Canadian Biotechnology Strategy
group to develop a set of biotechnology statistics for Canada. CIHR, Statistics Canada
and other organizations are partners in the Strategy. Statistics produced to date show
impressive results in terms of the number of new biotechnology companies and annual
earnings, almost half of which are in the health field. Two papers on biotechnology
indicators are to be presented at the Bluesky Conference.

While there are no comprehensive measures at present of value added from the
discovery and adoption of new technology in Canada, issues of how to do so are being
considered. 15,16 CIHR plans to monitor and when possible participate in these activities
with a view to collaborative development of appropriate indicators of commercialization.

CIHR also has targeted commercialization programs such as The Small and Medium-
Sized Enterprise Research Program (SME) and Proof of Principle (POP) grants. The
SME is a partnership between CIHR and biotechnology companies that supports
university spin-off companies and new commercial ventures. The POP program supports
university based researchers to establish the marketability of an invention or discovery
and then move it toward commercial viability. Follow-up studies of the results of these
programs can be carried out and documented to obtain direct measures of commercial
results from funding initiatives. For example, since 2001, more than 160 projects have
been funded by the POP program. Of the projects that have matured sufficiently to be
evaluated, 63%, or 49 projects, resulted in new patents being funded; 21%, or 16
projects had intellectual property licensed; and 14%, or 11 projects, contributed to new
company formation.

Direct cost savings

Health technology assessments have the potential to document savings from the use of
specific technologies. The Canadian Agency for Drugs and Technologies in Health
(CADTH) provides a national focus for the study of cost-effectiveness of new
technologies and medications. Collaboration with CADTH appears to be a promising
approach to defining indicators of cost savings as a result of the adoption of new
technologies that, for example, arose from CIHR-funded research.

Human Capital
The human capital approach to economic valuation measures the value of potential
earnings lost through ill health or accidents. Health Canada has published estimates of
the burden of disease in Canada using a human capital approach. 17 Work to update the
estimates is currently being pursued by Health Canada, Statistics Canada and the
Canadian Institute for Health Information. Once the work is complete, it seems feasible
to extend the estimates to provide measures of the value of illness or accidents avoided,
or disability periods shortened as a result of advances made possible by research.

Value of life and health

Recent work in the US and Australia has sought to measure the value of medical
advances in terms of the value of life or potential contributions to GDP. 18,19 The expert
group that discussed this Impact Framework thought that such an undertaking would be
beyond the scope of CIHR’s framework. From a conceptual point of view, the group
pointed out that GDP is affected by many factors and there is a well established system

14
in place to classify and measure the economic determinants of GDP growth. The
contributions of indirect factors, such as a healthy population, are not measured in
present accounting systems.

Conclusions
It is feasible to develop indicators to measure the impacts of health research and health
research funding. A conceptual framework is important to identify the stakeholders who
require impact information and understand the nature of evidence that will be
appropriate for each stakeholder group. Indicators can then be identified within this
framework.

The framework developed by CIHR facilitates the identification of areas where specific
initiatives can be evaluated to identify impacts and areas where a more nuanced
approach is required. In broad areas, particularly health impacts and economic benefits,
the most promising approach to impact measurement will be to measure the value to
society of innovations in prevention or treatment. Often, credit for breakthroughs will be
shared by many researchers and by those who support them, including the institutions in
which they operate and the agencies that fund their work. Funding agencies, such as
CIHR, can demonstrate that they have played an integral part in success by assuming a
leadership role in encouraging appropriate research, by recognizing the potential of
discovery in its early stages and by undertaking and encouraging the ongoing
assessment of impacts. This approach brings together impact measurement and
performance evaluation.

National and International collaboration between agencies and organizations that have
stakes in health research and national statistical and evaluation agencies are the most
promising route to continue the development and population of a robust and credible
suite of indicators.

This paper has emphasized the complex challenges in devising credible approaches to
the measurement of both impact and performance. This complexity means that no
single approach will satisfy all stakeholders or be sufficient to cover all the activities of an
agency with a broad mandate, such as CIHR. Nevertheless, as a publicly funded
organization, we have an obvious and clear responsibility to demonstrate both high
performance and high impact in everything that we do.

15
References:

1
Developing a CIHR Framework to Measure The Impact of Health Research. http://www.cihr-
irsc.gc.ca/e/documents/meeting_synthesis_e.pdf
2
National Consensus Conference on Population Health Indicators - Final Report. Canadian
Institute for Health Information, Ottawa, 1999.
3
Healthy Canadians: A Federal Report on Comparable Health Indicators, 2004. Health Canada,
Ottawa.
4
Buxton M, S Hanney, T Jones 2004. Estimating the economic value to societies of the impact of
health research: a critical review. Bulletin of the World Health Organization. 82 (10):733-739.
5
Sharpe A, Smith J. (2005). Measuring the Impact of Research on Well-being: A Survey of
Indicators of Well-being. Centre for the Study of Living Standards Report 2005-02.
6
Torjman, S, F Minns (2005). Sustainable Development Framework for Science and
Technology Research: Social and Cultural Dimensions. Caledon Institute of Social Policy.Ottawa.
7
PSA target metrics for UK research base. Office of Science and Technology, UK Depy. of Trade
and Industry, October 2004. (www.ost.gov.uk/research/psa_target_metrics_oct2004.pdf)
8
King, DA, 2004. The scientific impact of nations. Nature. 430: 311-316.
9
Observatoire des Sciences et des Technologies, 25 Years of Health Research in Canada,
Analysis produced for CIHR, November 2005
10
King, DA, 2004. The scientific impact of nations. Nature. 430: 311-316.
11
Thomson Scientific Journal Citation Reports. http://www.thomsonisi.com/
12
Statistics Canada, Cansim Table 102-0311.
13
2003 Survey of Intellectual Property Commercialization in the Higher Education Sector.
Statistics Canada Catalogue 88F0006XIE2005018 . Science, Innovation and Electronic
Information Division (SIEID). Statistics Canada.. Nov. 2005.
14
Biotechnology Use and Development Survey. Preliminary results for 2003 reported in
Innovation Analysis Bulletin. Statistics Canada Catalogue 88-003-XIE, V7,1. February, 2005 and
V7.3, October 2005.
15
Gault, Fred and Susan McDaniel. Forthcoming. Summary: Joint Statistics Canada – University
of Windsor Workshop on Intellectual Property Commercialization Indicators, Windsor,
November, 2004. SIEID Working paper. Statistics Canada.
16
Earl, Louise, Fred Gault and Michael Bordt. Forthcoming. Summary: Meeting on
Commercialisation Measurement, Indicators, Gaps and Frameworks, Ottawa, December, 2004.
SIEID Working paper. Statistics Canada.
17
Economic Burden of Illness in Canada, 1993. Moore R, Mao Y, Zhang Y, Clarke, K. Laboratory
Centre for Disease Control. Health Canada. Ottawa, 1997.
18
Measuring the Gains from Medical Research An Economic Approach. Murphy K. M. Topple,
R.H. eds. The University of Chicago Press. 2003.
19
Nordhaus, W.D. The Health of Nations: The Contribution of Improved Health to Living
Standards (pg. 29, 30). In Measuring the Gains from Medical Research An Economic Approach.
Murphy K. M.Topel, R.H. eds. The University of Chicago Press. 2003.

16