International Journal for Quality in Health Care 2003; Volume 15, Supplement 1: pp. i5–i11 10.

1093/intqhc/mzg084

Developing evidence-based clinical

indicators: a state of the art
methods primer
JAN MAINZ
The National Indicator Project and University of Aarhus, Denmark

Downloaded from https://academic.oup.com/intqhc/article/15/suppl_1/i5/1796945 by guest on 23 May 2023

Abstract
Objective. To describe steps in developing and testing clinical indicators based on state of the art methods in previous literature
and experience in the Danish National Indicator Project.
Analysis. The development process includes a planning phase, where the clinical area to be evaluated is chosen and the
measurement team selected and organized. The planning phase is followed by a development phase where clinical indicators
are prioritized and selected by the measurement team on the basis of documentation and knowledge from the scientiWc literature.
When clinical indicators have been selected, speciWc measure speciWcations should be designed, including inclusion and exclusion
criteria for the target population, description of a risk adjustment strategy, identiWcation of data sources, description of data
collection procedures, and an analytical plan for data analyses. Before clinical indicators are implemented they should be tested
for reliability and validity. Preliminary tests may identify areas requiring further modiWcations and speciWcations of the indicators.
Conclusion. Using clinical indicators for quality assessment represents an important approach to documenting the quality of
care. Consumers of indicator information (clinicians, administrators, purchasers, regulators, and patients) need reliable and
valid information for benchmarking, making judgments, and determining priorities, accountability and quality improvement.
This underlines the fact that clinical indicators must be developed and tested with scientiWc rigor in a transparent process.
Keywords: clinical indicators, outcome measures, performance measures, quality improvement, quality of care

In most health care systems, a consensus is emerging that Each of these steps is discussed below. SpeciWc examples
there is a need for quality measures. Various audiences may wish of how they apply to clinical indicators for different diseases
to use them to document the quality of care, make comparisons are provided to make the concepts more concrete, to illustrate
(benchmarking), make judgments and determine priorities, some of the choices faced by developers of clinical indicators,
support accountability, support quality improvement, and and to facilitate understanding of the process of developing
provide transparency in health care [1,2]. clinical indicators.
Using clinical indicators for performance and outcome
measurement is one way of measuring and monitoring the qual-
Choosing the clinical area to evaluate
ity of care and services. In a companion paper in this journal,
clinical indicators have been deWned and characterized [3]. There are many potential users of clinical indicators (clinicians,
It is imperative that clinical indicators are meaningful, administrators, purchasers, regulators, and patients). Each
scientiWcally sound, generalizable, and interpretable. To achieve user must clarify the purpose of the quality measurement
this, clinical indicators must be developed, tested, and imple- effort. Several purposes may motivate quality measurement,
mented with scientiWc rigor [4]. such as regulation, purchasing, or quality improvement.
This paper focuses on the development and testing of clinical Administrative quality improvement may arise from an organ-
indicators. ization’s mission, values, and commitment, in response to
patient complaints, or in response to payer or regulatory
demands. The purposes will dictate the focus on particular
Steps in developing and testing indicators clinical areas. Prioritizing among clinical areas for assessment
may be based on various criteria, including the importance of
The different steps required to develop and test clinical indicators the health care problem or disease and the opportunity for
are summarized in Table 1. clinical interventions [2,4–6].

Address reprint requests to J. Mainz, Lyseng Allé 1, 8270 Hoejbjerg, Denmark. E-mail: jmz@ag.aaa.dk

International Journal for Quality in Health Care vol. 15 Supplement 1

© International Society for Quality in Health Care and Oxford University Press 2003; all rights reserved i5
J. Mainz

Table 1 Steps in the development and testing of clinical indicators

Planning phase 1. Choose the clinical area to evaluate

Establish importance (high volume, cost, variation)
Identify opportunities for clinical intervention
2. Organize the measurement team
Select group participants
Organize and divide tasks
Development phase 3. Provide an overview of existing evidence and practice
Present documentation and knowledge from the scientiWc literature for potential
indicators
Establish consensus about existing knowledge and practice
4. Select clinical indicators and standards

Downloaded from https://academic.oup.com/intqhc/article/15/suppl_1/i5/1796945 by guest on 23 May 2023

Select process indicators
Select outcome indicators
Identify prognostic factors (risk adjustment)
Establish consensus and rating procedures
5. Design measure speciWcation
DeWne indicators and standards
Identify target population
Determine inclusion and exclusion criteria
Devise risk adjustment strategy
Identify data sources
Describe data collection procedures
Develop a plan
6. Perform pilot testing

The importance of the health care problem. McGlynn and others actions are available to improve the quality of care. Conditions
have suggested that a health care problem or a disease is with high volumes might be chosen for indicator monitoring.
important if it has a high volume, and is associated with high Rare conditions have often not had many clinical trials for
morbidity and mortality and is costly to treat. logistical reasons, and do not have evidence-based guidelines
Local or national epidemiological data can be used to developed. Therefore, it is difWcult to know which processes
determine the prevalence of disease in a population. Mortality might be improved by monitoring. In the Danish National
rates, signiWcant use of health services, and costly treatment Indicator Project, stroke and lung cancer were selected, partly
are other criteria that have been cited to support a focus on a because evidence-based clinical guidelines were available.
particular condition for quality measurement [2,4–6]. Disease- Greater priority should be given to clinical areas where there is
speciWc mortality rates for a variety of conditions are available evidence that the quality of care is either variable or substandard,
in most countries [1,5,6]. For example, in Denmark, such data so that areas with a substantial potential for quality improvement
are available from the Central Patient Registry [6]. Health are chosen. In the Danish National Indicator Project, lung cancer
services utilization rates for a particular condition and costs of was included because mortality rates indicated that Denmark
treatment have also been cited by McGlynn and others as had a high mortality rate compared with other countries [6].
important indicators of importance and, when available, may
be used as additional criteria for choosing conditions for quality
Organize the measurement team
measurement [1,5,6].
The higher the prevalence and incidence of a condition or Select team members. The measurement team may be stronger if it
frequencies of procedures or outcomes, the more likely it is represents different perspectives. Clinicians can ensure that
that it will be possible to identify an adequate number of cases appropriate clinical indicators are selected together with stand-
for quality measurement. ards to evaluate whether desirable performance or outcome rates
are obtained. Often it may be relevant to select a multidisciplinary
Opportunities for clinical interventions. Another important criterion for team of clinicians, including, for example, doctors from different
choosing the clinical area for review is the opportunity for specialties, nurses, physiotherapists, and occupational therapists.
interventions related to the health care problem or disease. Clinicians being evaluated will have more conWdence in the
Clinical indicators are most useful if the processes and out- indicators developed if measurement team members include
comes being assessed can be inXuenced by clinical interventions clinicians who are widely recognized and respected. Relevant
in terms of quality improvements efforts. For each health credentials include appropriate professional training and an
problem or disease it is therefore important to consider what active role in professional societies.

i6
 Evidence-based clinical indicators

For instance, when selecting clinicians for a multidiscipli- evidence is obtained by meta-analysis of randomized controlled
nary team to develop clinical indicators for lung cancer, it trials and evidence from at least one randomized controlled
might be relevant to include thoracic surgeons, internists trial (‘A’-evidence). ‘B’-evidence is obtained for controlled
specializing in pulmonary diseases, oncologists, nurses, physi- studies without randomization or quasi-experimental studies.
otherapists, and psychologists [6]. When clinical indicators for ‘C’-evidence relates to different epidemiological studies such
schizophrenia are developed, psychiatrists, nurses specializing as case-control studies. Finally, ‘D’-evidence refers to evidence
in psychiatry, psychologists, and social workers might be relevant based on different expert opinions. Using a rating scheme to
to include in the measurement team [6]. summarize the strength of evidence enables the measurement
If the quality measurement is conducted in a nation, a team to describe the evidence of clinical indicators [9].
region, state, or county, it might be best to draw the team Literature databases such as the Cochrane Collaboration or
members from different geographical areas, from urban and Medline, and the compendium ‘Clinical Evidence’ are impor-
rural locales, and from different types and sizes of organization. tant sources for determining the strength of evidence for
The size of the team, however, should be kept small, so clinical indicators (available online at http://www.cochrane.org,

Downloaded from https://academic.oup.com/intqhc/article/15/suppl_1/i5/1796945 by guest on 23 May 2023

that discussions and teamwork do not become unwieldy. In http://www.ncbi.nlm.nih.gov/PubMed, and http://www.
the Danish National Indicator Project, the teams selected consist clinicalevidence.org).
of eight to 14 professional clinicians appointed by different
scientiWc societies [6].
Select clinical indicators and standards
Quality of care researchers together with clinical epidemio-
logical expertise can help to ensure methodological integrity A clinical indicator is a measure that assesses a particular
of the clinical indicators and a sound approach to data collec- health care process or an outcome [3]. In other words a clinical
tion and data analysis. Like multi-center clinical trials, multi- indicator is a tool for producing a quantitative measurement
organizational measurement involves methodological issues of quality of care. However, simply knowing the level of an
that require highly trained scientists. indicator does not reveal whether or not it is acceptable. A
According to the selected clinical area, it might be relevant judgment of the acceptability of a performance or an out-
to include patient or administrator representatives. Patients come rate must be made in relation to the purpose for which
and administrator representatives can contribute to the selection it should be used. The Wnal stage in measuring health care
and prioritization of some clinical indicators. For complex quality is applying a standard of quality that embodies accept-
medical technical indicators, their contribution will be less. ability of a particular performance or outcome rate. If a
The Clinical Standards Board of Scotland usually has patients desired attribute of care falls below the standard or an unde-
and organizational representatives ‘on board’ their selecting sired attribute of care rises above this level, further evaluation
teams [7]. or action is triggered. There may be instances where standards
are not established and must be based on a preliminary data
Organize teamwork. Once the measurement team has been collection when benchmarks are available. Before the estab-
selected, the work in the group can be organized, including lishment of standards, providers or populations can be compared
planning relevant meetings, the creation of smaller working to determine relative quality of care, but adherence to a standard
subgroups, and the delegation of tasks to individual group par- cannot be determined. The strength of evidence for both the
ticipants. In the Danish National Indicator Project we found it clinical indicators and related standards of care can be
helpful that all meetings in the measurement team were fol- described. Multiple indicators can be used to evaluate the
lowed by exhaustive summaries of decisions made by the group. quality of care for most conditions. It is best to select clinical
In nationwide measurement projects, reimbursement of clin- indicators supported by evidence indicating that they reXect
ical departments for clinicians, time used in meetings, and other the process and outcome being evaluated. If no scientiWc
tasks may facilitate the work within the measurement team [6]. evidence is available, clinical indicators can be selected on the
basis of consensus among health professionals.
Provide an overview of existing evidence and
Clinical process indicators. Measuring the quality of a process of
practice
care requires determination of whether clinicians are adhering
Clinical indicators should be based on scientiWc evidence or to practices to achieve the best outcomes for patients. The
consensus among health professionals. Preferably, clinical linkage of a process to outcomes must ideally have been dem-
indicators should be based on research evidence rather than onstrated scientiWcally.
on expert opinions or clinical experience alone. The level of To determine whether to measure a speciWc process of care
evidence supporting each clinical indicator is transparent it is helpful to review the strength of scientiWc literature
when the strength of that evidence is described before the supporting the fact that inclusion of this process in the process
indicators are selected. measure will affect outcomes. The processes of care measured
Providing an overview of existing knowledge from the should be those demonstrated to cause a higher probability of
scientiWc literature and practice allows the measurement team achieving a desired outcome. The strength of the evidence for
to take into account the strength of evidence when choosing an indicator will determine its scientiWc soundness or the like-
clinical indicators. There are numerous systems for rating the lihood that improvements in the clinical indicator will produce
strength of scientiWc evidence [8,9]. The highest level of consistent and credible improvement in quality of care. Clinical

i7
J. Mainz

Table 2 Examples of clinical process indicators

Indicator concept DeWnition of Type Standard of care Time of Level of evidence

indicator intervention/
treatment
.....................................................................................................................................................................................................................................................

Organization of Proportion of Process More than 90% of patients <24 hours after A
treatment (stroke) patients treated/ with acute stroke should admission
rehabilitated in be treated and rehabilitated in
stroke units a stroke unit
Treatment (lung cancer) Proportion of Process Treatment rate ≥70% Discharge B
patients treated
Resection (lung cancer) Proportion of Process Resection rate ≥25% Discharge B

Downloaded from https://academic.oup.com/intqhc/article/15/suppl_1/i5/1796945 by guest on 23 May 2023

patients resected

A, evidence from at least one randomized controlled trial; B, evidence from quasi-experimental or non-randomized controlled studies.

process indicators based on A-evidence are most credible, but Although the presented indicators for lung cancer are asso-
there might be arguments for selecting indicators with a lower ciated with B- and C-evidence, they might be regarded as impor-
strength of evidence. tant by a measurement team when evaluating the treatment
Initially, when evidence links a process to better outcomes for lung cancer, because it is important from a clinical
it may appear that the standard for a proportion of patients so perspective to evaluate this clinical practice even though no
treated should be 100%. However, there are reasons why this randomized controlled studies have been conducted [6].
is not always the case, depending on how well the denominator For most process indicators, risk adjustment plays a
of eligible patients can be deWned. smaller role than it does for outcome measurement. For some
Table 2 illustrates an example of a clinical process indicator process measures, however, risk adjustment may reveal that
for stroke. The indicator measures the proportion of patients patient factors are inXuencing a measure. The more closely an
treated and rehabilitated in a stroke unit [6]. The evidence indicator measures the actual process of care delivered rather
supporting this indicator is strong (A-evidence), since meta- than patient adherence or other factors, the less risk adjustment
analyses of randomized controlled trials have demonstrated the will be needed [4].
effects of stroke units on outcomes of care [10,11]. Compared
with treatment in departments of internal medicine, treatment Clinical outcome indicators. Multiple factors contribute to
in stroke units was associated with lower mortality [odds health care outcomes [3]. When evaluating outcome indica-
Ratio (OR)=0.83; 95% conWdence interval (CI) 0.71–0.97]. tors, the adequacy of controls for differences in case mix
On the basis of this evidence it has been recommended that and the adequacy of controls for other covariates are impor-
all patients with acute stroke be treated at specialized stroke tant criteria.
units [10,11]. Table 2 illustrates, however, that the selected Case mix or severity-of-illness adjustment allow for a ‘fair’
standard suggests that >90% of patients with acute stroke comparison of health outcomes to ensure that any observed dif-
should be treated at stroke units. The reason for this discrepancy ferences can be attributed to the health care interventions and
is that a smaller subgroup of patients (∼10%) with acute stroke not to differences between the populations included [5]. Patients
will not beneWt from treatment at specialized stroke units who die or recover more slowly may not have received poorer
because they are deceased by the time of admittance to hospi- quality care, but have been at higher risk for the outcomes before
tal or intensive care unit [6]. treatment.
Table 2 also illustrates two clinical process indicators for Table 3 lists different outcome indicators. Intermediate
lung cancer: proportion of patients who are actively treated outcome indicators reXect changes in biological status that
and proportion of patients who are resected. The active treat- affect subsequent health outcomes, and can be regarded as
ment rate expresses the proportion of patients who were short-term outcomes [3]. For example, large RCTs in people
offered a concrete treatment (e.g. resection, chemotherapy, or with type 1 and type 2 diabetes have found that risk of devel-
radiation therapy). opment or progression of complications increases progressively
The scientiWc literature indicates that active treatment for as hemoglobin A1c (HbA1c) increases above the non-diabetic
all patients with lung cancer is important with regard to survival range (≥7.0 mmol/l) (A-evidence) [15]. HbA1c therefore reX-
and the patients’ quality of life (B-evidence) [12–14]. According ects important health outcomes that can only be measured
to the literature, >70% of patients with primary lung cancer after years. It is considered desirable for most diabetics
should be offered an active treatment (C-evidence). (≥90%) to have a HbA1c <7.0 mmol/l [15].
Cohort studies have shown that a high resection rate is The last outcome indicator presented in Table 3 refers to
associated with high survival (B-evidence) [12–14]. The studies 30-day mortality for stroke patients. Stroke is associated with
indicate that >25% of patients should be resected. high mortality. The literature suggests that case fatality should

i8
 Evidence-based clinical indicators

Table 3 Examples of outcome indicators

Indicator concept Indicator deWnition Type Standard of care Time of Level of

intervention/ evidence
treatment
................................................................................................................................ .....................................................................................................................

Blood glucose Proportion of diabetics with Intermediate ≥90% should have Every third A
control HbA1c <7.0 mmol/l outcome HbA1c <7.0 mmol/l month
Mortality (stroke) 30-day and 3-, 6- and Outcome <20% should have 30 days after B
12-month mortality a 30-day mortality stroke

A, evidence from at least one randomized controlled trial; B, evidence from quasi-experimental or non-randomized controlled studies.

Downloaded from https://academic.oup.com/intqhc/article/15/suppl_1/i5/1796945 by guest on 23 May 2023

be <20%. Prognostic factors according to the literature are based on these measures would be expressed as means for the
age, sex, previous stroke, diabetes, and social status [16]. relevant population. All details of the indicator should be
speciWed. For instance, for the above-mentioned indicator,
The selection process. The measurement team selects clinical indi- the concept ‘stroke unit’ should be deWned. A stroke unit can
cators in a consensus process. For a speciWc clinical area, team be described as a hospital unit or part of a hospital unit that
members can suggest multiple clinical indicators. The team may solely, or almost solely, treats or rehabilitates patients with
use several methods for selecting among them. Criteria may stroke [6].
include the indicator’s importance, strength of evidence for When the indicators have been deWned, standards describing
the indicator and the indicator’s validity and reliability, and what are acceptable performances or outcomes should be
the Xexibility of obtaining the indicator data. The team may speciWed. As described above, standards can be derived from
apply various processes to make decisions based on these criteria, the academic literature or from consensus within the meas-
including consensus, majority vote, or prioritization according urement team. The combination of evidence from the scientiWc
to average ratings derived using formal techniques such as those literature and clinical experience may be helpful for the meas-
used by RAND for appropriateness criteria [17]. The Danish urement team to derive relevant and realistic standards.
National Indicator Project has chosen to select indicators by
team consensus in order to enhance the credibility of indicators Identify target population. The target population refers to the
among intended users [6]. Each indicator should be considered patient group whose care the clinical indicator is designed to
by the measurement team according to importance, strength of assess. SpeciWc inclusion and exclusion criteria have to be
evidence, validity and reliability, and feasibility. Within the selec- deWned. It should also be decided whether the selection
tion of indicators, different rating procedures are available [17]. should be based on conWrmed diagnoses, or symptoms or
signs. Whether prevalent or incident cases (or both) are
included should also be taken into consideration. It might be
Design measure specifications
relevant to describe upper or lower age limits. Finally, decisions
When potential indicators have been selected, the next step is about the time period for measurement should be taken.
to design a reliable and valid measure that can be implemented
consistently. It is helpful when measure speciWcations are Determine risk adjustment strategy. As described above, prognos-
described in speciWc manuals in order to minimize inter-rater tic factors have to be identiWed and described together with
variation [6]. clinical outcome indicators. The data required for risk adjust-
ment are indicated above. SpeciWc data deWnitions should be
DeWne the clinical indicator and standard. The Wrst step is to described for all included prognostic factors (patient charac-
describe each indicator in detail. Examples of indicators are teristics and comorbidities).
described in Tables 2 and 3. Some measures are dichotomous
where the answer is either ‘yes’ or ‘no’, such as whether a patient Identify data sources. Once the clinical indicators have been
with lung cancer has undergone resection. Dichotomous deWned, the prognostic factors for risk adjustments derived,
measures, when aggregated for the relevant population to and the target population identiWed, the measurement team
which a quality indicator is applied, are expressed as propor- must state how the data should be obtained.
tions, with a given numerator and denominator, such as the Different data sources are used for quality measurement.
proportion of patients with lung cancer who have had resection Administrative data are generated to support reimbursement
(Table 2). Dichotomous measures can also be used to generate activities. This data source is readily available and inexpensive to
rates, which are proportions within a given time period. An collect, but it might be unreliable because of coding problems,
example of a rate is the proportion of stroke patients treated and therefore lacks speciWcity and detail. Clinical data include
in a stroke unit within 24 hours (Table 2) [18]. medical record data and data from clinical research databases.
Other measures are continuous, such as the number of Medical record data are the most complete sources of infor-
minutes from arrival at the health care facility until a myocardial mation on diagnosis, treatment, and clinical outcomes. They
infarct patient received thrombolytic treatment. Indicators are, however, expensive to obtain. Data from clinical research

i9
J. Mainz

databases are often valid and reliable, if such databases are aspect of quality of care, are compared. Measuring inter-rater
available in the chosen clinical area. Primary data, in terms of reliability, internal consistency, and test–re-test reliability
prospectively collected clinical data that are collected for a allows users to determine if the data collection methods are
particular quality measurement purpose, are the most speciWc precise enough to provide reproducible results. These meth-
and can deWne exactly what data are required. Primary data ods assess data quality powerfully and identify whether the
can also include survey data, from patients to access attitudes, measure and data collection procedures are well speciWed.
behavior, knowledge, and outcome. Since primary data represent Validity determines the degree to which an indicator measures
data that are not readily available, such data are expensive to what it is intended to measure, that is whether the results of a
collect, but are often a valid and reliable information source. measurement corresponds to the true state of the phenomenon
In quality measurement, data collection can be made part of being measured [2–4]. Validity can be tested by conWrming that
routine care by standardizing the documentation of patient char- the scores of a measure are linked to speciWc outcomes, and that
acteristics and care delivery that clinicians and administrators are the measure can reXect good and bad quality.
already recording while delivering care and services. Rubin et al.

Downloaded from https://academic.oup.com/intqhc/article/15/suppl_1/i5/1796945 by guest on 23 May 2023

have noted that this is particularly useful for the efWcient use of
process indicators [2,4], and may reduce missing data and the Conclusion
additional cost of data collection. When electronic medical
records become widespread, standard data speciWcations useful This paper has outlined the steps in a state-of the-art develop-
for quality assessment can be incorporated into these systems. ment process of clinical indicators. Performance and outcome
This would eliminate duplicative clinical data collection for the indicators represent the only way to obtain quantitative data
purposes of clinical care and quality assessment [2,4]. on the quality of care for quality improvement.
Clinical areas should be selected according to the importance
Describe data collection procedures. Description of detailed speciWca- of health care problems and known opportunities for clinical
tions for data collection allows different health institutions (e.g. interventions. In the development process, representatives
hospitals) to implement quality measures in a consistent way that can contribute signiWcantly should be involved. Because
so that results may be fairly compared. clinical indicators measure the cornerstones of clinical practice,
A protocol for scoring the measures can be developed recognized and respected clinicians play a central role. Nation-
according to the measure speciWcations. A plan for handling wide or regional quality measurement efforts may wish to
missing data or data outside of a logical range must also be include team members from different geographical areas for
included in the speciWcations. increased acceptance and credibility.
A development process based on scientiWc principles will
Develop an analytical plan. As part of the development process, create more valid indicators and will also increase acceptance
Rubin et al. recommend developing a detailed plan for how and credibility of the resulting data. For each selected indicator
the measures are to be analyzed and how statistical and clinical and related standard of care, transparency about measures,
signiWcance should be determined [2,4]. The analytical plan whether evidence or consensus based, and the level of evidence
could include a detailed description of the population, an available, will assist users in assessing how much the resolving
assessment of the distribution of the data, how missing data data matter to them.
are handled, and the statistical analyses of tests to be used. For some process measures and for all outcome measures,
The measurement team may wish to consider what are prognostic factors (patient characteristics and comorbidities)
clinically signiWcant differences among groups rather than are generally needed to deWne a risk adjustment strategy.
simply statistically signiWcant differences. Comparing the As part of the development process, exhaustive and exclusive
clinical signiWcance of differences among groups helps measure speciWcations should be described, including speciWc
answer a question that data users frequently ask: ‘what is good deWnitions of the clinical indicators and standards, identiWcation
quality of care?’. of the target population and data sources; and development
of an analytical plan with description of statistical and clinical
signiWcance of the results will be assessed when comparing
Preliminary testing
groups or comparing a group to a standard.
Before clinical indicators are implemented they should be Before implementation of clinical indicators, preliminary
thoroughly tested. Preliminary tests may identify areas that tests for reliability and validity can provide users with infor-
require further speciWcations of the quality measures. mation about the trustworthiness and usefulness of data.
Reliability of a clinical indicator expresses the extent to Indicator measurement can be made most efWcient when
which repeated measurements of a stable phenomenon by incorporated into routine patient care as part of clinicians’
different providers and instruments, at different times and and administrators’ documentation of required information
places, obtain similar results. Reliability is important for on patient characteristics and care delivery, which is already
comparing groups or comparing the same group over time recorded for clinical purposes.
periods. Reliability can be tested as inter-rater reliability, When electronic medical records become widespread, data
where different people or methods provide data on the same speciWcation for quality assessment can be incorporated into
indicator. Reliability can also be tested as internal consistency, such systems, which will avoid duplicative clinical data collec-
for which two indicators, expected to measure the same tion and reduce additional cost.

i10
 Evidence-based clinical indicators

References 10. Langhorne P, Dennis M, eds. Stroke Units: An Evidence-Based

Approach. London: BMJ Books, 1998.
1. US Department of Health and Human Services, Public Health 11. Stroke Unit Trialists’ Collaboration. Organised Inpatient (Stroke
Service, and Agency for Health Care Policy and Research. Unit) Care for Stroke (Cochrane Review). In The Cochrane Library,
AHCPR Publication No. 95-0046, March 1995. Issue 1. Oxford: Update Software, 2001.
2. Rubin HR, Pronovost P, Diette GB. The advantages and disad- 12. Breathnach OS, Freidlin B, Conley B et al. Twenty-two years of
vantages of process-based measures of health care quality. Int J phase III trials for patients with advanced non-small-cell lung
Qual Health Care 2001; 13: 469–474. cancer: sobering results. J Clin Oncol 2001; 19: 1734–1742.
3. Mainz J. DeWning clinical indicators for quality improvement. 13. Engeland A, Bjorge T, Haldorsen T, Tretli S. Prognosis of
Int J Qual Health Care 2003, 523–530. patients with lung cancer diagnosed in Norway, 1954–93. Cancer
4. Rubin HR, Pronovost P, Diette GB. From a process of care to a Causes Control 1998; 9: 57–65.
measure: the development and testing of a quality indicator. Int J 14. Laroche C, Wells F, Coulden R et al. Improving surgical resection
Qual Health Care 2001; 13: 489–496. rate in lung cancer. Thorax 1998; 53: 445–449.

Downloaded from https://academic.oup.com/intqhc/article/15/suppl_1/i5/1796945 by guest on 23 May 2023

5. McGlynn EA, Asch SM. Developing a clinical performance 15. The Diabetes Control and Complications Trial Research Group.
measure. Am J Prov Med 1998; 14: 14–21. The effect of intensive treatment of diabetes on the development
6. Mainz J, Bartels PD, Laustsen S et al. The National Indicator and progression of long-term complications in insulin-dependent
Project to monitoring and improving of the medical technical diabetes. N Engl J Med 1993; 329: 977–986.
care. Ugeskr Laeger 2001; 163: 6401–6406. 16. Aboderin I, Venebles G. Pan European consensus meeting on
7. Clinical Standards Board of Scotland. Colorectal Cancer Services. stroke management. J Intern Med 1996; 240: 173–180.
Edinburgh: CSBS, 2002. 17. Brook RH, McGlynn EA, Cleary PD. Measuring quality of care.
8. West S, King V, Cary TS et al. Systems to Rate the Strength of Scien- N Engl J Med 1996; 335: 960–970.
tiWc Evidence. Evidence Report/Technology Assessment No. 47. 18. Sackett D, Haynes RB, Guyatt GH, Tugwell P. Clinical Epidemiology.
AHRQ Publication No. 02-E016. Rockville, MD: Agency for A Basic Science for Clinical Medicine. Boston, MA: Little, Brown and
Healthcare Research and Quality, April 2002. Company, 1991.
9. Eccles M, Fremantle N, Mason J. North of England evidence-
based guidelines development project: methods of developing
guidelines for efWcient drug use in primary care. Br Med J 1998;
316: 1232–1235. Accepted for publication 18 August 2003

i11
Downloaded from https://academic.oup.com/intqhc/article/15/suppl_1/i5/1796945 by guest on 23 May 2023