270

Qual Saf Health Care: first published as 10.1136/qhc.11.3.270 on 1 September 2002. Downloaded from http://qualitysafety.bmj.com/ on May 25, 2023 by guest. Protected by copyright.
QUALITY IMPROVEMENT RESEARCH

Evaluation of quality improvement programmes

J Øvretveit, D Gustafson
.............................................................................................................................

Qual Saf Health Care 2002;11:270–275

In response to increasing concerns about quality, many greater consequences for patient safety and other
countries are carrying out large scale programmes clinical outcomes. Yet we know little of their
effectiveness or relative cost effectiveness, or how
which include national quality strategies, hospital to ensure they are well implemented.
programmes, and quality accreditation, assessment and Decision makers and theorists have many
questions about these programmes:
review processes. Increasing amounts of resources are
• Do they achieve their objectives and, if so, at
being devoted to these interventions, but do they ensure what cost?
or improve quality of care? There is little research • Why are some more successful than others?
evidence as to their effectiveness or the conditions for • What are the factors and conditions critical for
maximum effectiveness. Reasons for the lack of success?
evaluation research include the methodological • What does research tell us about how to
improve their effectiveness?
challenges of measuring outcomes and attributing
Some anecdotal answers come from the reports of
causality to these complex, changing, long term social consultants and participants, and there are theo-
interventions to organisations or health systems, which ries about “critical success factors” for some types
themselves are complex and changing. However, of programme. However, until recently there was
little independent and systematic research about
methods are available which can be used to evaluate effectiveness and the conditions for effectiveness.
these programmes and which can provide decision Indeed, there was little descriptive research which
makers with research based guidance on how to plan documented the activities which people actually
undertook when implementing a programme.
and implement them. This paper describes the research Research has made some progress in answering
challenges, the methods which can be used, and gives these questions, but perhaps not as much as was
examples and guidance for future research. It hoped, in part because of the methodological
challenges. This paper first briefly notes some of
emphasises the important contribution which such the research before describing the challenges and
research can make to improving the effectiveness of the research designs which can be used. It
finishes with suggestions for developing research
these programmes and to developing the science of in this field.
quality improvement.
.......................................................................... RESEARCH INTO QUALITY IMPROVEMENT
PROGRAMMES
The most studied subcategory of quality pro-

A
quality programme is the planned activities grammes is hospital quality programmes, particu-
carried out by an organisation or health larly US hospital total quality management
system to improve quality. It covers a range programmes (TQM), later called continuous
of interventions which are more complex than a quality improvement programmes (CQI). Several
single quality team improvement project or the non-systematic reviews have been carried out
quality activities in one department. Quality pro- (box 1).2–6
grammes include programmes for a whole organ- There is evidence from some studies that
isation (such as a hospital total quality pro- certain factors appear to be necessary to motivate
gramme), for teams from many organisations (for and sustain implementation and to create condi-
example, a “collaborative” programme), for exter- tions likely to produce results. The most com-
nal reviews of organisations in an area (for exam- monly reported are senior management commit-
ple, a quality accreditation programme), for ment, sustained attention and the right type of
changing practice in many organisations (for management roles at different levels, a focus on
example, a practice guidelines formulation and customer needs, physician involvement, sufficient
implementation programme), and for a national resources, careful programme management, prac-
or regional quality strategy which itself could tical and relevant training which personnel can
See end of article for include any or all of the above. These programmes use immediately, and the right culture.4–13 These
authors’ affiliations create conditions which help or hinder smaller demanding conditions for success raise questions
....................... quality improvement projects. about whether the type of quality programmes
Correspondence to: Quality improvement programmes are new which have been tried are feasible for health care.
Dr J Øvretveit, The Nordic “social medical technologies” which are increas- These limited conclusions appear similar across
School of Public Health, ingly being applied. One study noted 11 different public and private, and across nations. However,
Box 12133, Goteborg,
S-40242 Sweden;
types of programmes in the UK NHS in a recent 3 there is little research for non-US clinics and hos-
jovret@aol.com year period.1 They probably consume more re- pitals, for public hospitals, or systematic compara-
....................... sources than any treatment and have potentially tive investigation to support this impression.

www.qualityhealthcare.com
Evaluation of quality improvement programmes 271

Qual Saf Health Care: first published as 10.1136/qhc.11.3.270 on 1 September 2002. Downloaded from http://qualitysafety.bmj.com/ on May 25, 2023 by guest. Protected by copyright.
concluded that there was some evidence that quality collabo-
Box 1 Non-systematic reviews of hospital quality ratives can help some teams to make significant improve-
programmes ments quickly if the collaborative is carefully planned and
managed, and if the team has the right conditions. It
The general conclusions of non-systematic reviews of hos- suggested that a team’s success depended on their ability to
pital quality programmes are: work as a team, their ability to learn and apply quality meth-
• The label given to a programme (for example, “TQM”) is no ods, the strategic importance of their work to their home
guide to the activities which are actually carried out: organisation, the culture of their home organisation, and the
programmes with the same name are implemented very dif- type and degree of support from management. This can help
ferently at different rates, coverage, and depth in the teams and their mangers to decide whether they have, or can
organisation. create, the conditions to be able to benefit from taking part in
• Few hospitals seem to have achieved significant results and what can be a costly programme.
little is known about any long term results. There is therefore little research into quality programmes
• Few studies describe or compare different types of hospital which meets rigorous scientific criteria, but some of the
quality programmes, especially non-TQM/CQI pro-
research which has been done does provide guidance for deci-
grammes.
• Most studies have severe limitations (see later).
sion makers which is more valid than the reports of
consultants or participants. There is clearly a need for more
evaluations and other types of studies of quality programmes
which answer the questions of decision makers and also build
With regard to research methods, studies have tended to theory about large scale interventions to complex health
rely on quality specialists or senior managers for information organisations or health systems. The second part of this paper
about the programme and its impact, and to survey them once considers the designs and methods which could be used in
retrospectively. Future studies need to gather data from a future research.
wider range of sources and over a longer period of time. Data
should also be gathered to assess the degree of implementa- RESEARCH CHALLENGES
tion of the programme. Implementation should not be These interventions are difficult to evaluate using experimen-
assumed; evidence is needed as to exactly which changes have tal methods. Many programmes are evolving, and involve a
been made and when. Outcomes need to be viewed in relation number of activities which start and finish at different times.
to how deeply and broadly the programme was implemented These activities may be mutually reinforcing and have a
and the stage or “maturity” of the programme. To date, for synergistic effect if they are properly implemented: many
most studies the lack of evidence of impact may simply reflect quality programmes are a “system” of activities. Some quality
the fact that the programmes were not implemented, even programmes are implemented over a long period of time;
though some respondents may say they had been. Assessing many cannot be standardised and need to be changed to suit
the degree of implementation could also help to formulate the situation in ways which are different from the way in
explanations of outcomes. There is a need for studies of which a treatment is changed to suit a patient.
organisations which are similar apart from their use of quality The targets of the interventions are not patients but whole
methods and ideas, as well the need for more studies to use organisations or social groups which vary more than the
the same measures—for example, of results, of culture, or of physiology of an individual patient: they can be considered as
other variables. Many of these points also apply to research complex adaptive social systems.27 There are many short and
into other types of quality programmes. long term outcomes which usually need to be studied from the
perspectives of different parties. It is difficult to prove that
Other quality improvement programmes these outcomes are due to the programme and not to
Few other types of quality improvement programmes have something else, given the changing nature of each type of
been systematically studied or evaluated; there are few studies programme, their target, the environment, and the time scales
of national or regional programmes such as guideline involved. They are carried out over time in a changing
implementation or of the effectiveness of quality review or economic, social, and political climate which influences how
accreditation processes.14 Managers have reported that organi- they are implemented.28
sations which received low scores (“probation”) on the US One view is that each programme and situation is unique
Joint Commission for Accreditation of Healthcare Organisa- and no generalisations can be made to other programmes
tions assessment were given high scores 3 years later but had elsewhere. This may be true for some programmes, but even
not made substantive changes.6 Few studies have described or then a description of the programme and its context allows
assessed the validity or value of the many comparative quality others to assess the relevance of the programme and the find-
ings to their local situation. However, at present researchers do
assessment systems,15–18 of external evaluation processes,19–24 or
not have agreed frameworks to structure their descriptions
have studied national or regional quality strategies or
and allow comparisons, although theories do exist about
programmes in primary health care.25
which factors are critical.
More evaluation research is also being undertaken into
Quasi-experimental designs can be used29 30: it may be pos-
quality improvement collaboratives. This is part of a new wave sible to standardise the intervention, control its implementa-
of research which is revealing more about the conditions tion, and use comparison programmes within the same envi-
which organisations and managers need to create in order to ronment in order to exclude other possible influences on
foster, sustain and spread effective projects and changes. Col- outcomes. One issue is that many programmes are local inter-
laboratives are similar to hospital quality programmes in that pretations of principles; many are not standardised specific
they usually involve project teams, but the teams are from dif- interventions that can be replicated. Indeed, they should not
ferent organisations. The structure of the collaborative and the be: flexible implementation for the local situation appears to
steps to be taken is more prescribed than most hospital qual- be important for success.5 TQM/CQI is more a philosophy and
ity programmes. set of principles than a specific set of steps and actions to be
One study has drawn together the results of evaluations of implemented by all organisations, although some models do
different collaboratives.26 This study provides knowledge come close to prescribing detailed steps.
which can be used to develop collaboratives working on other
subjects, helps to understand factors critical to success, and RESEARCH DESIGNS
also demonstrates other research methods which can be used The difficulties in evaluating these programmes do not mean
to study some types of quality programmes. The study that they cannot or should not be evaluated. There are a

www.qualityhealthcare.com
272 Øvretveit, Gustafson

Qual Saf Health Care: first published as 10.1136/qhc.11.3.270 on 1 September 2002. Downloaded from http://qualitysafety.bmj.com/ on May 25, 2023 by guest. Protected by copyright.
Box 2 A qualitative evaluation of external reviews of Box 3 Example of a theory testing comparative design
clinical governance
The first comprehensive studies of effectiveness of
One example which illustrates the use of qualitative meth- TQM/CQI programmes in health care also tried to estab-
ods is a study of the UK government’s programme of exter- lish which factors were critical for “success”.8–10 The meth-
nal review of clinical governance arrangements in public ods used in these studies were to survey 67 hospitals, some
healthcare provider organisations.35 Members of the with programmes and some without, and later 61 hospitals
review team as well as senior clinicians and managers with TQM programmes, asking questions about the
were interviewed in 47 organisations before and after the programme and relating certain factors to quality perform-
review. A qualitative analysis identified themes and issues ance improvement. The findings were that, after 3 years,
and reported common views about how the review process the hospitals could not report clear evidence of results and
could be improved. that few had tackled clinical care processes.
Although most interviewees thought the reviews gave a A later study tested hypotheses about associations
valid picture of clinical governance, much of the between organisation and cultural factors and
knowledge produced was already known to them but had performance.11 Interviews and surveys were undertaken in
not been made explicit. It concluded that major changes in 10 selected hospitals. Performance improvements were
policy, strategy, or direction in the organisations had not found in most programmes in satisfaction, market share,
occurred as a result of the reviews, and suggested that the and economic efficiency as measured by length of stay,
use of the same process for all organisations was “at best unit costs, and labour productivity. Interestingly, culture
wasteful of resources and perhaps even positively was only found to influence the patient satisfaction
harmful”. This study provided the only independent performance. It was easier for smaller hospitals with fewer
description of the review process and of different complex services to implement CQI. Early physician
stakeholders’ assessments as to its value and how the involvement was also associated with CQI success, a find-
process could be improved. The findings were useful to the ing reported in other studies.6 7
reviewers to refine their programme. One of the limitations This set of studies has a practical value. The findings give
of the study was that it did not investigate outcomes further managers a reliable foundation for assessing whether they
than the interviewees’ perceptions of impact: “measuring have the conditions which are likely to result in a success-
impact reliably is difficult and different stakeholders may ful programme. Another strength of this study was to assess
have quite different subjective perceptions of impact”.35 the “depth” of implementation by using Baldridge or
EFQM award categories.19 21 Limitations of the study were
that: precise descriptions of the nature of the different hos-
number of designs and methods which can and have been pital programmes were not given; only one site data gath-
used: these are summarised below and discussed in detail ering visit was undertaken; and less than 2 years was
elsewhere.28–34 taken for the investigation so that the way the programmes
changed and whether they were sustained could not be
Descriptive case design gauged. Follow up studies would add to our knowledge of
This design simply aims to describe the programme as imple- the long term evolution of these programmes, any long
mented. There is no attempt to gather data about outcomes, term results, and explanations about why some hospitals
but knowledgeable stakeholders’ expectations of outcome and were more successful than others.
perceptions of the strengths and weaknesses of the pro-
gramme can be gathered. Why is this descriptive design
sometimes useful? Some quality programmes are prescribed intervention. Outcomes are considered as the differences
and standardised—for example, a quality accreditation or between the before and after data collected about the target.
external review. In these cases a description of the interven- The immediate target is the organisation and personnel; the
tion activities is available which others can use to understand ultimate targets are patients.
what was done and to replicate the intervention. However, Comparative before-after designs produce stronger evi-
many programmes are implemented in different ways or not dence that any outcomes were due to the programme and not
described, or may only be described as principles and without to something else. If the comparable unit has no intervention,
a strategy. For the researcher a first description of the this design allows some control for competing explanations of
programme as implemented saves wasting time looking for outcomes if the units have similar characteristics and
impact further down the causal chain (for example, patient environments. These are quasi-experimental or “theory
outcomes) when few or no activities have actually been testing” designs because the researcher predicts changes to
implemented. the one or more before-after variables, and then gathers the
data before and after the intervention (for example, personnel
Audit design attitudes towards quality) to test the prediction. However,
This design takes a written statement about what people when limited to studying only before-after (or later)
should do, such as a protocol or plan, and compares this with differences, these designs do not generate explanations about
what they actually do. This is a quick and low cost evaluation why any changes occurred (box 2).
design which is useful when there is evidence that following a
programme or protocol will result in certain outcomes. It can Retrospective or concurrent evaluation designs: single
be used to describe how far managers and health personnel case or comparative
follow prescriptions for quality programme interventions and In these designs the researcher can use either a quasi-
why they may diverge from these prescriptions. “Audit” experimental “theory testing” approach or a “theory building”
research of quality accreditation or review processes can help approach. An example of the former is the “prediction testing
managers to develop more cost effective reviews.35 survey” design. The researcher studies previous theories or
empirical research to identify theorised critical success
Prospective before-after designs: single case or factors—for example, sufficient resources, continuity of man-
comparative agement, aspects of culture—and then tests these to find
The single case prospective design gathers specific data about which are associated with successful and unsuccessful
the target of the intervention before and after (or during) the programmes (box 3).

www.qualityhealthcare.com
Evaluation of quality improvement programmes 273

Qual Saf Health Care: first published as 10.1136/qhc.11.3.270 on 1 September 2002. Downloaded from http://qualitysafety.bmj.com/ on May 25, 2023 by guest. Protected by copyright.
(2) Validating “implementation assessment”
Box 4 Example of an action evaluation comparative
design (3) Wider outcome assessment
(4) Longitudinal studies
A 4 year comparative action evaluation study of six Nor- (5) More attention to economics
wegian hospitals provided evidence about results and (6) Explanatory theory
critical factors.4 7 36 It gave the first detailed and long term
description about what hospitals in a public system (7) Common definitions and measures
actually did and how the programmes changed over time. (8) Tools to predict and explain programme effectiveness
The study found consistencies between the six sites in the
factors critical for success: management and physician Assessing or measuring the level of implementation of
involvement at all levels, good data systems, the right the intervention
training, and effective project team management. A 9 year Studies need to assess how “broadly” the programme
follow up is planned. penetrated the organisation (did it reach all parts?), how
“deeply” it was applied in each part, and for how long it was
applied. One of the first rules of evaluation is “assume nothing
In contrast, a “theory building” approach involves the has been implemented—get evidence of what has been imple-
researcher in gathering data about the intervention, context, mented, where and for how long”.30 There is no point looking
and possible effects during or after the intervention (box 4). To for outcomes until this has been established. Instruments for
describe the programme as it was implemented, the re- assessing “stage of implementation” or “maturation” need to
searcher asks different informants to describe the activities be developed such as the adaptation of the Baldridge criteria
which were actually undertaken.30 The validity of these used in the study by Shortell et al5 or other instruments.
subjective perceptions can be increased by interviewing a cross
Validating “implementation assessment”
section of informants, by asking informants for any evidence
Survey responses are one data source for assessing level of
which they can suggest which would prove or disprove their
implementation and are useful for selecting organisations for
perceptions, and by comparing data from difference sources to
further studies. However, these responses need to be gathered
identify patterns in the data (box 4).30 32 33
from a cross section of personnel, at different times, and sup-
The choice of design depends on the type of quality
plemented by site visits and other data sources to improve
programme (short or long term, prescribed or flexible, stable
validity.
or changing), for whom the research is being undertaken, and
the questions to be addressed (Was it carried out as planned? Wider outcome assessment
Did it achieve its objectives? What were the outcomes? What With regard to short term impact, data need to be gathered
explains outcomes or success or failure?). Descriptive, audit, from a wide cross section of organisational personnel and
and single case retrospective designs are quicker to complete other stakeholders and from other data sources. Most studies
and are cheaper but do not give information about outcomes. also need to gather data about long term outcomes and to
Comparative outcome designs can introduce some degree of assess carefully the extent to which these outcomes can be
control, thus making possible inferences about critical factors attributed to the programme. The outcome data to be gathered
if good descriptions of the programmes and their context are should be determined by a theory predicting effects, which
also provided. builds on previous research, or in terms of the specified objec-
tives of the programme, and these links should be made clear
IMPROVING FUTURE RESEARCH in the report.
Some of the shortcomings of research into quality pro-
grammes have been presented earlier. The five most common Longitudinal studies
are: Retrospective single surveys provide data which is of limited
• Implementation assessment failure: the study does not use. We need more prospective studies which follow the
examine the extent to which the programme was actually dynamics of the programme over long timescales. Many future
carried out. Was the intervention implemented fully, in all studies will need to investigate both the intervention and the
areas and to the required “depth”, and for how long? outcomes over an extended period of time. Very little is known
• Outcome assessment failure: the study does not assess any about whether these programmes are continued and how they
outcomes or a sufficiently wide range of outcomes such as might change, or about long term outcomes.
short and long term impact on the organisation, on
More attention to economics
patients, and on resources consumed.
No studies have assessed the resources consumed by a quality
• Outcome attribution failure: the study does not establish improvement programme or the resource consequences of the
whether the outcomes can unambiguously be attributed to outcomes. The suspected high initial costs of implementation
the intervention, or whether something else caused the would look different if more was known about the costs of
outcomes. sustaining the programme and about the possible savings and
• Explanation failure: there is no theory or model which economic benefits.37 Long term evaluations may also uncover
explains how the intervention caused the outcomes and more outcomes, benefits, or “side effects” which are not
which factors and conditions were critical. discovered in short studies.
• Measurement variability: different researchers use very dif-
Explanatory theory
ferent data to describe or measure the quality programme
For hospital programmes there is no shortage of theories
process, structure, and outcome. It is therefore difficult to
about how to implement them and the conditions needed for
use the results of one study to question or support another
success, but few are empirically based. For both practical and
or to build up knowledge systematically.
scientific reasons, future studies need to test these theories or
Future evaluations would be improved by attention to the fol- build theories about what helps and hinders implementation
lowing: at different stages, and about how the intervention produces
(1) Assessing or measuring the level of implementation of the any discovered outcomes. For other types of quality pro-
intervention grammes there is very little theory of any type. Innovation

www.qualityhealthcare.com
274 Øvretveit, Gustafson

Qual Saf Health Care: first published as 10.1136/qhc.11.3.270 on 1 September 2002. Downloaded from http://qualitysafety.bmj.com/ on May 25, 2023 by guest. Protected by copyright.
programmes. Decision theory models could be used to create
Box 5 Steps for studying a quality improvement
such tools, as could tools which effectively predict the
programme outcomes of particular improvement projects.39
In addition there is a need for overviews and theories of
The methods used depend on who the research is for (the quality improvement programmes; we have not described the
research user), the questions to be addressed, and the type full range of interventions which fall within this category and
of programme. An example of one action evaluation have only given a limited discussion of a few. Future research
research strategy is presented here.30 36 studies need to describe the range of complex large scale
• Conceptualise the intervention. At an early stage, form a quality interventions increasingly being carried out and their
simple model of the component parts of the programme and characteristics—for example, to describe and compare na-
of the activities carried out at different times. This model can
tional or regional quality programmes. More consideration is
be built up from programme documents or any plans or
descriptions which already exist, or from previous theories needed of the similarities and differences between them, of
about the intervention. what can be learned from considering the group as a whole,
• Find and review previous research about similar pro- and of how theories from organisation, change management,
grammes and make predictions. Identify which factors are sociology, and innovation studies can contribute to building
suggested by theory or evidence to be critical for the theories about these interventions (box 5).
success of the programme. Identify which variables have
been studied before and how data were collected. CONCLUSIONS
• Identify research questions which arise out of previous
Although there is research evidence that some discrete quality
research and/or which are of interest to the users of the
research. team projects are effective, there is little evidence that any
• Consider whether the intervention can be controlled in its large scale quality programmes bring significant benefits or
implementation (would people agree to follow a prescribed are worth the cost. However, neither is there strong evidence
approach or have they done so if it is a retrospective that there are no benefits or that resources are being wasted.
study?). If not, design part of the study to gather data to The changing and complex features of quality programmes,
describe the programme as implemented and to assess the their targets, and the contexts make them difficult to evaluate
level of implementation. Consider whether comparisons using conventional medical research experimental evaluation
could be made with similar or non-intervention sites—for methods, but this does not mean that they cannot be
example, to help exclude competing explanations for evaluated or investigated in other ways. Quasi-experimental
outcomes or to discover assisting and hindering factors.
evaluation methods and other social science methods can be
• Plan methods to use to investigate how the programme was
actually carried out, the different activities performed, and used. These methods may not produce the degree of certainty
to assess the level of implementation. Gather data about the that is produced by a triple blind randomised controlled trial
sequence of activities and how the programme changed of a treatment, but they can give insights into how these proc-
over time. Use documentary data sources, observation, esses work to produce their effects.
interviews, or surveys as appropriate describing how Conclusive evidence of effectiveness may never be possible.
informants or other data sources were selected and possible At this stage a more realistic and useful research strategy is to
bias. Note differences between the planned programme describe a programme and its context and discover factors
and the programme in action, and participants’ explana- which are critical for successful implementation as judged by
tions for this as well as other explanations.
different parties. In a relatively short time this will provide
• Plan methods to gather data about the effects of the
programme on providers and patients if possible. Data may useful data for a more “research informed management” of
be participants’ subjective perceptions, or more objective these programmes.
before and after data (for example, complaints, clinical A science is only as good as its research methods. The
outcomes), or both. Use data collected by the programme science of quality improvement is being developed by research
participants to monitor progress and results if these data are into how changes to organisation and practice improve patient
valid. Consider how to capture data about unintended side outcomes. However, insufficient attention has been given to
effects—for example, better personnel recruitment and methods for evaluating and understanding large scale
retention. programmes for improving quality. As these programmes are
• Consider other explanations for discovered effects apart increasingly used, there is particular need for studies which do
from the programme and assess their plausibility.
not only assess effectiveness, but also examine how best to
• To communicate the findings, create a model of the
programme which shows the component parts over time, implement them.
the main outcomes, and factors and conditions which
appear to be critical in producing the outcomes. Specify the .....................
limitations of the study, the degree of certainty about the Authors’ affiliations
findings, and the answers to the research questions. J Øvretveit, Professor of Health Policy and Management, The Nordic
School of Public Health and The Karolinska Institute, Sweden, and The
Faculty of Medicine, Bergen University, Norway
D Gustafson, Robert Ratner Professor of Industrial Engineering &
adoption38 and diffusion theories are one source of ideas for Preventive Medicine, University of Wisconsin, Madison, WI 53705, USA
building explanatory theories, for understanding level of
implementation, and for understanding why some organisa- REFERENCES
tions are able to apply or benefit more from the intervention 1 West E. Management matters: the link between hospital organisation
than others.38 and quality of patient care. Qual Health Care 2001;10:40–8.
2 Bigelow B, Arndt, M. Total quality management: field of dreams. Health
Care Manage Rev 1995;20:15–25.
Common definitions and measures 3 Motwani J, Sower V, Brasier L. Implementing TQM in the health care
Most studies to date have used their own definitions and sector. Health Care Manage Rev 1996;21:73–82.
measures of effects of quality programmes. This is now limit- 4 Øvretveit J, Aslaksen A. The quality journeys of six Norwegian
hospitals. Oslo: Norwegian Medical Association, 1999.
ing our ability to compare and contrast results from different 5 Shortell S, Bennet C, Byck G. Assessing the impact of continuous quality
evaluation studies and to build a body of knowledge. improvement on clinical practice: what will it take to accelerate progress.
Milbank Quarterly 1998;76:593–624.
Tools to predict and explain programme effectiveness 6 Blumenthal D, Kilo C. A report card on continuous quality improvement.
Milbank Quarterly 1998;76:625–48.
Future research needs to go beyond measuring effectiveness 7 Øvretveit J. The Norwegian approach to integrated quality
and to give decision makers tools to predict the effects of their development. J Manage Med 2001;15:125–41.

www.qualityhealthcare.com
Evaluation of quality improvement programmes 275

Qual Saf Health Care: first published as 10.1136/qhc.11.3.270 on 1 September 2002. Downloaded from http://qualitysafety.bmj.com/ on May 25, 2023 by guest. Protected by copyright.
8 Shortell S, O’Brien J, Hughes H, et al Assessing the progress of TQM in 23 Shaw C. External quality mechanisms for health care: summary of the
US hospitals: findings from two studies. Quality Leader 1994;6:14–17. ExPeRT project on visitatie, accreditation, EFQM and ISO assessment in
9 Shortell M, O’Brien J, Carman J, et al. Assessing the impact of European Union countries. Int J Qual Health Care 2000;12:169–75.
continuous quality improvement/total quality management: concept 24 Øvretveit J. Quality assessment and comparative indicators in the
versus implementation Health Serv Res 1995;30:377–401. Nordic countries. Int J Health Planning Manage 2001;16:229–41.
10 Carman JM, Shortell SM, Foster, RW, et al. Keys for successful 25 Wensing M, Grol R. Single and combined strategies for implementing
implementation of total quality management in hospitals. Health Care changes in primary care: a literature review. Int J Qual Health Care
Manage Rev 1996;21:48–60. 1994;6:115–32.
11 Boerster H, Foster E, O’Connor, et al. Implementation of total quality 26 Øvretveit J. How to run an effective improvement collaborative. Int J
management: conventional wisdom versus reality. Hospital Health Serv Health Care Qual Assur 2002;15:33–44.
Admin 1996;41:143–59. 27 Plsek P, Wilson T. Complexity science: complexity, leadership, and
12 Gustafson D, Hundt A. Findings of innovation research applied to management in healthcare organisations. BMJ 2001;323:746–9.
quality management principles for health care. Health Care Manage Rev 28 Øvretveit J. Evaluating hospital quality programmes. Evaluation
1995;20:16–24. 1997;3:451–68.
13 Gustafson D, Risberg L, Gering D, et al. Case studies from the quality 29 Cook T, Campbell D. Quasi-experimentation: design and analysis issues
improvement support system. ACHPR Research Report 97-0022. for field settings. Chicago: Rand McNally, 1979.
Washington: US Department of Health and Human Services, 1997. 30 Øvretveit J. Action evaluation of health programmes and change: a
14 Shaw C. External assessment of health care. BMJ 2001;322:851–4. handbook for a user focused approach. Oxford: Radcliffe Medical Press,
15 Thompson R, McElroy H, Kazandjian V. Maryland hospital quality 2002.
indicator project in the UK. Qual Health Care 1997;6:49–55. 31 Øvretveit J. Evaluating health interventions. Milton Keynes: Open
16 Cleveland Health Quality Choice Program (CHQCP). Summary University Press, 1998.
report from the Cleveland Health Quality Choice Program. Qual Manage 32 Yin R. Case study research: design and methods. Beverly Hills: Sage, 1994.
Health Care 1995;3:78–90. 33 Jick T. Mixing qualitative and quantitative methods: triangulation in
17 Rosenthal G, Harper D. Cleveland health quality choice. Jt Comm J action. In: Van Maanen J, ed. Qualitative methodology. Beverly Hills:
Qual Improve 1994;8:425–42. Sage, 1983.
18 Pennsylvania Health Care Cost Containment Council (PHCCCC). 34 Ferlie E, Gabbay J, FitzGerald F, et al. Evidence-based medicine and
Hospital effectiveness report. Harrisburg: Pennsylvania Health Care Cost organisational change: an overview of some recent qualitative research.
Containment Council, 1992. In Mark A, Dopson S, eds. Organisational behaviour in healthcare: the
19 National Institute of Standards and Technology (NIST). The Malcum research agenda. London: Macmillan, 1999.
Baldridge national quality award 1990 application guidelines. 35 Walshe K, Wallace L, Freeman T, et al. The external review of quality
Gaithersburg, MD: National Institute of Standards and Technology, improvement in healthcare organisations: a qualitative study. Int J Qual
1990. Health Care 2001;13:367–74.
20 Hertz H, Reimann C, Bostwick M. The Malcolm Baldridge National 36 Øvretveit J. Integrated quality development for public healthcare. Oslo:
Quality Award concept: could it help stimulate or accelerate healthcare Norwegian Medical Association, 1999.
quality improvement? Qual Manage Health Care 1994;2:63–72. 37 Øvretveit J. The economics of quality: a practical approach. Int J Health
21 European Foundation for Quality Management (EFQM). The Care Qual Assur 2000;13:200–7.
European Quality Award 1992. Brussels: European Foundation for 38 Rogers E. Diffusion of innovation. New York: Free Press, 1983.
Quality Management, 1992. 39 Gustafson D, Cats-Baril W, Alemei F. Systems to support health policy
22 Sweeney J, Heaton C. Interpretations and variations of ISO 9000 in analysis. Ann Arbor: Health Administration Press, University of Michigan,
acute health care. Int J Qual Health Care 2000;12:203–9. 1992.

www.qualityhealthcare.com