210

QUALITY IMPROVEMENT RESEARCH

Methods for evaluation of small scale quality

improvement projects
G Harvey, M Wensing
.............................................................................................................................

Qual Saf Health Care 2003;12:210–214

Evaluation is an integral component of quality purpose. Evaluations of small scale projects may
improvement and there is much to be learned from the encompass one or more of the following aims:
evaluation of small scale quality improvement initiatives (1) to monitor the success or impact of a local
quality improvement project over time—for ex-
at a local level. This type of evaluation is useful for a ample, to make sure the project is achieving the
number of different reasons including monitoring the desired results and to demonstrate the impact of
the project to others;
impact of local projects, identifying and dealing with
(2) to identify issues or problems as they arise
issues as they arise within a project, comparing local within the project so that actions can be taken to
projects to draw lessons, and collecting more detailed change or redesign the project while it is in
information as part of a bigger evaluation project. progress;
Focused audits and developmental studies can be used (3) to compare similarities and differences in a
number of local projects to draw out common les-
for evaluation within projects, while methods such as sons learnt and develop hypotheses for future
multiple case studies and process evaluations can be research;
used to draw generalised lessons from local experiences (4) to collect more detailed information about the
and to provide examples of successful projects. processes and outcomes of implementing a local
quality improvement initiative as part of a bigger
Evaluations of small scale quality improvement projects evaluation research project to help to explain the
help those involved in improvement initiatives to findings of this project.
optimise their choice of interventions and use of Broadly speaking, the reasons for evaluation
relate to two main types of learning—learning
resources. Important information to add to the within the project (points 1 and 2 above) and
knowledge base of quality improvement in health care more generalised learning about the implementa-
can be derived by undertaking formal evaluation of tion of quality improvement (points 3 and 4
above). The first type of learning is associated
local projects, particularly in relation to building theory with the processes of clinical audit and quality
around the processes of implementation and increasing improvement, while the second type is associated
understanding of the complex change processes with research. This paper will outline a number of
approaches and methods for the evaluation of
involved. quality improvement at a local level. Table 1 high-
.......................................................................... lights the four main approaches that will be pre-
sented.

M
any questions can be raised about the
impact of quality improvement pro- TYPES OF EVALUATIONS IN SMALL SCALE
grammes in health care. Do they work? IMPROVEMENT PROJECTS
How can they be improved? What factors promote Focused audit studies
or inhibit their success? What can we learn from Local quality improvement projects typically
our local experiences? Why do they work in some involve implementing one or more specific
settings and not in others? Different research changes that are designed to bring about im-
designs are needed depending on the focus of the provements on a focused topic, such as a new way
specific question the research is trying to answer, of treating a particular condition or a different
often involving the setting up of an external way of organising delivery of care. Examples
research project. But what about quality improve- include a quality improvement project to ensure
ment initiatives that take place on a small scale the provision of evidence-based pain manage-
such as a local ward, unit or departmental level: a ment to patients following gastrointestinal sur-
clinical audit project, a process redesign effort or a gery or a project to introduce more clinically and
See end of article for unit that is participating in a breakthrough cost effective ways of organising patient-centred
authors’ affiliations collaborative—should these be evaluated and, if stroke services at a district or regional level.
....................... Within projects such as these, evaluation should
so, how?
Correspondence to: Evaluations of small scale quality improvement comprise an integral part of the quality improve-
Dr G Harvey, Royal projects (defined as projects in a specific ward, ment process linked to an explicit assessment of
College of Nursing unit or practice) can help both those who under- the effect of implementing planned changes in
Institute, Radcliffe Infirmary,
Woodstock Road, Oxford take such projects and researchers of quality practice. For example, in models of continuous
OX2 6HE, UK improvement interventions. An important first quality improvement the third phase of the Plan-
....................... step in any evaluation is the clarification of its Do-Study-Act cycle1 involves collecting data to

www.qshc.com
Methods for evaluation of small scale quality improvement projects 211

Table 1 Types of evaluations in small scale improvement projects

Research designs Aims Approaches

Focused audit studies Monitor impact of the activities over time Evaluation as a component of quality improvement
Developmental studies Identify issues and intervene when necessary, develop Evaluation linked to action by participants in the case of action research
hypotheses
Multiple case studies Draw lessons and develop hypotheses Case reports and comparisons across a number of local projects
Process evaluations Explain the findings of a bigger research project In depth analyses of projects as part of a bigger research project

evaluate whether changes introduced during the “Do” phase ies may comprise sampling of cases, such as patient records, so
have actually realised improvements in practice or patient that statistical techniques can be used to indicate the reliabil-
care. Similarly, in models of clinical audit the process typically ity of figures. Generalisation to a larger population of
includes an audit cycle in which a key stage involves evaluat- clinicians or practices is, however, not sought. Focused audit
ing how practice compares with expected standards and studies help to close the loop of the quality improvement cycle,
implementing changes accordingly. These changes are then an area where many projects have been shown to fail in the
re-evaluated by a process of re-audit.2 The example illustrated past.5 Furthermore, information on the impact of the project
in box 1 shows the role of evaluation within a project designed aids learning from the local project, which is the aim of the
to improve the repeat prescribing process in a general practice approaches described next.
setting.3
Measurements should be valid but simple.4 Chart reviews, Developmental studies
surveys among patients, or simple observations of events are Evaluation may also be beneficial with ongoing quality
all examples of possible data collection methods. The relative improvement projects to help assess what actions may be
simplicity of the measurements is perhaps most visible in the needed to refine or improve the design of the project, or
absence of complex case mix adjustments, as these would specific interventions within the project. Evaluation mecha-
often require extensive additional data collection. Audit stud- nisms can be built into a local improvement project through
both informal and formal methods. At an informal level, this
might involve observation and discussion with colleagues
about the process of how the project is going. Alternatively, the
Box 1 Role of evaluation within a project designed to evaluation may employ a more formal developmental research
improve the repeat prescribing process in a busy method, particularly where there is a need to provide support,
general practice setting3 feedback, or help to the project team.6 One method is action
research, which is broadly defined as an approach to research
This project was established within a general practice in that actively involves participants and which has an explicit
the UK to improve the service to patients in relation to focus on promoting and facilitating change.7 It is an approach
ordering repeat prescriptions. A 48 hour target for that has been used in a range of healthcare settings in the UK
processing repeat prescriptions was set. A multiprofes- and has been the subject of a recent review to define the
sional team was established to work on the quality approach more clearly and assess its impact in practice.8 From
improvement initiative, using continuous quality improve- this review a number of factors key to the success of action
ment methods and supported by an external facilitator. research were highlighted, including participation, maintain-
Following the steps of the Plan-Do-Study-Act cycle, the team ing a “real world” focus, resources, and project management.
began by gathering information to assess their current Developmental approaches to evaluation may be particu-
practice and plan the necessary changes. This included the larly useful within the context of organisational learning9 and
preparation of flow charts of the repeat prescribing proc- learning by professionals10 because of their action-orientated
ess, and a baseline audit over a 1 month period to assess approach and the focus on personal and professional develop-
how many prescriptions were actually ready for collection ment. Within a quality improvement project, developmental
within 48 hours and to identify the number that required research may form part of a flexible intervention
medical records to be checked before they could be programme—for example, a tailored educational approach to
signed. implement clinical guidelines, enabling actions to be planned
Information gained from the flow charts and the initial on the basis of insight into the barriers for change. Box 2
audit results helped the team to identify those areas where illustrates the use of action research to introduce new wound
they could introduce changes that would have the most management practices in a community nursing
impact and to identify the measures they would use to organisation.11 This example also illustrates the use of a
evaluate the change process. Once planned, the changes focused audit to assess the impact of the project as an integral
were implemented in practice and repeat audits were part of the study design. The type of knowledge generated by
undertaken at 6, 12 and 24 months. The resulting data developmental approaches is seen to be practical and
were presented in two main ways: a comparison of results propositional,8 and the focus is on generating and refining
at baseline, 6, 12 and 24 months; and graphs plotting the interpretations through inductive processes within repeated
turnaround times for consecutive prescriptions over time. cycles of action research. As quality improvement projects
Analysis of the results helped the team to understand studied through action research do not usually involve
more clearly what was happening. Although 95% of random or purposeful sampling, the generalisability of the
repeat prescriptions were available within 48 hours at the knowledge generated may be limited to associations between
baseline audit, the graphs illustrated considerable different variables within the project under study.
variation which led to frustration among staff. Repeated
audits demonstrated improvements in turnaround times, Multiple case studies
significant reductions in the number of records that needed In the approaches described above the focus has mainly been
to be checked, and much greater staff satisfaction as the on learning within and about individual quality improvement
process became more consistent and more effective. projects. However, to draw out common experiences and
lessons for the purpose of more generalised learning about

www.qshc.com
212 Harvey, Wensing

Box 2 An action research approach to introduce new Box 3 Key steps in the comparative case study
wound management practices in a community approach
nursing organisation11
• Select individual cases relevant to the issues to be studied.
This project was set up to establish and encourage an • Collect data within individual sites using a range of quanti-
improved approach to wound management in a commu- tative and qualitative methods.
• Analyse the data within individual sites using appropriate
nity nursing organisation in South Australia. Within the quantitative and qualitative methods of analysis—for exam-
organisation about 50% of client visits were related to ple, descriptive statistics, thematic analysis of qualitative
wound care, hence the importance of promoting best data.
practice in this area of care. Following an initial survey of • Compare data analyses across sites to draw more general
wound management practices, participatory action re- conclusions and/or generate hypotheses for further testing.
search groups were established to address some of the
issues identified.
Each group followed an action research approach with can still compare across projects to draw out some more gen-
its three phases of planning, action, and evaluation being eralisable findings. One approach often used in these
undertaken as part of a cyclical process. Volunteers were situations is the multiple case or comparative case study
sought for the participatory action research groups, and method.12 13
core principles of action research including the group’s Increasingly, the comparative case study approach is being
responsibility for agenda setting, decision making about applied in health care, notably within the field of evidence
appropriate actions, and reaching consensus were based practice and quality improvement. Here the focus is
emphasised. One group elected to focus specifically on often on “why” questions, such as “why and under what con-
evidence based practice relating to the care of leg ulcers, ditions clinical professionals decide to adopt an innovation or
particularly appropriate methods for cleansing chronic leg change their clinical practice”.13 Recently published studies
ulcers. This involved comparing the use of tap water addressing questions such as this include an evaluation of the
cleansing to an aseptic technique with sterile saline impact of guidelines on the management of adult asthma,14
solution. the uptake of evidence based practice in elective
As part of the planning phase, an initial review of the lit- orthopaedics,15 the management of glue ear,16 an evaluation of
erature was undertaken which highlighted the fact that the the ‘Promoting Action on Clinical Effectiveness’ initiative
evidence base underpinning cleansing practice was across 16 sites in England,17 and an evaluation of the six
limited and inconclusive. However, from this review and projects forming the Welsh Clinical Effectiveness National
their own clinical experience, the group reached the con- Demonstration Project.18 Box 3 summarises some of the key
clusion that there was no evidence to suggest that tap steps involved in the comparative case study approach.
water cleansing was ineffective. It also had the advantage Purposeful selection of cases to be included in the study
of being more cost effective. Moving on to the action contributes to its validity because a relevant diversity of cases
phase of the research cycle, the group examined the cur- is studied.13 15 In reality, however, the range of cases studied
rent cleansing practices used by their colleagues and may be determined by what cases are available. The case study
reasons underpinning their chosen approach. This approach is not characterised by one specific method for data
highlighted concerns around infection influencing the collection. Instead, a key feature is the use of data from a range
choice of the aseptic technique, so the group ran of sources which are often collected using both quantitative
educational sessions to disseminate the research evidence and qualitative methods—for example, questionnaire surveys,
on cleansing wounds. A repeat survey was subsequently semi-structured interviews, analysis of written documents,
carried out which showed an increase in the use of the and direct observations. Combining data from multiple
clean tap water technique. As a spin-off from the action sources to study specific variables (known as “triangulation”)
research and the identification of a lack of evidence to is recommended as it increases the validity of the data.19 It
inform cleansing practices, a randomised controlled trial may, however, be expensive or impossible to achieve triangula-
was subsequently set up to compare the use of warmed tion for all the variables studied.
sterile saline with warm tap water for cleansing chronic leg The data analysis in multiple case studies is not character-
ulcers. ised by one specific technique but by its overall approach. It is
recognised that the cases are heterogeneous, so the analysis
usually takes two approaches. Firstly, the cases are described
quality improvement, it is most helpful to compare experi- in depth—comparable to detailed case reports of complex
ences across a number of local improvement projects to iden- patients—including, for instance, both factual descriptions
tify similarities and differences. This presents particular chal- and the views of the participants. A systematic approach may
lenges in terms of identifying an appropriate research then be used to derive lessons from such case reports—for
methodology for a number of reasons: instance, by verifying ideas on cases other than the one on
• each local project may be focused on a different topic for which the idea was originally based.20 Secondly, multiple case
improvement and have different targets; studies can be used to examine associations between variables
and hypotheses on determinants of success, although formal
• there may be considerable variation in the processes of
statistical testing may be impossible. This requires that infor-
implementation as well as external influences across
mation on the impact of the projects is available from, for
sites—for example, reasons for introducing the quality
instance, focused audit studies.
improvement initiative, membership of the quality im-
Box 4 describes a project in which a number of hypotheses
provement team, use of an internal/external facilitator or
were developed a priori and then tested on the basis of the
change agent;
data available. Testing hypotheses is only valid for a limited
• process and outcome indicators used to audit the progress number of predefined factors; if too many factors are studied,
and impact of the project are likely to be specific to each some associations will be found by chance. The associations
individual site. found should be interpreted as hypothesis generating rather
Dealing with these context-specific issues requires an than testing. Although the heterogeneity of cases means that
approach that is able to take account of local differences yet data cannot be pooled by more traditional methods such as

www.qshc.com
Methods for evaluation of small scale quality improvement projects 213

ness of a particular approach to quality improvement. Within

Box 4 Multiple case study approach to evaluate the
the design of the RCT, the research team may decide to collect
implementation of 10 programmes to increase more detailed qualitative data from a sample of the study sites
physical exercise in older adults21 involved in the trial to examine more fully what happens dur-
ing the implementation process. This, in turn, may inform
Physical exercise improves the health status of adults, their subsequent understanding of the relationships between
including older adults, but many adults perform very little process and outcome data and provide information that helps
physical exercise. A range of programmes in the Nether- to explain the trial findings in more detail. Another aim may
lands which focuses on walking, dancing, and aerobics be to provide examples of successful sites (“success stories”)
aim to encourage older adults to become physically active that can be used to disseminate the message of the trial to a
for at least 30 minutes per day, at least five days a week. wider audience. A potential problem that needs to be consid-
The clinical effectiveness of many of these programmes has ered, however, is the effect that additional measurements
been proven, so the focus is now on effective implementa- (collected as part of the in-depth evaluation) may have on the
tion in terms of setting up programmes and optimal partici- subjects participating in the quality improvement project, as
pation of older adults in these programmes. these may be undesirable in the context of a controlled trial. If
A multiple case study project has been undertaken to this is the case, it is important to find the right balance
evaluate the implementation of 10 physical exercise between learning about the programme and avoiding the test
programmes. This study has taken two approaches. Firstly, effect. Process evaluations of quality improvement have been
structured descriptions of the programmes were made and discussed in detail in an earlier paper in this series.23
showed, for instance, that a variety of methods were used
to improve participation in the programmes such as DISCUSSION
personal contact in case of absence, obligatory indication All practitioners of quality improvement need to know the
of check out, and provision of drinks to enhance social impact of specific programmes and possible ways to improve
interaction. Furthermore, project leaders were asked to their effectiveness. Focused audit studies and developmental
describe the most important barriers and facilitators to the studies are designs that can help to structure these evaluations
success of the programme. Many mentioned, for example, and provide information to determine the optimal choice of
the problem of convincing municipalities and welfare interventions and use of resources for quality improvement.
organisations of the relevance of the programme. These Although the generalisability of the findings may be limited to
data were used to make structured descriptions of the the programmes evaluated, such evaluations can help to shed
cases. light on the more promising quality improvement methods
Secondly, the study team proposed about 25 hypoth- and approaches.
eses on factors that influenced the success of implementa- An issue which is often debated is the extent to which clini-
tion. For instance, it was hypothesised that the programme cians and others who undertake quality improvement projects
was more successful if there was a local tradition of at a local level should use rigorous evaluation methods. For
collaboration between different organisations and if the instance, how many cases should they study to get a reliable
physical exercise was three times a week (rather than five). figure, should they adjust for case mix severity, and how
Structured questionnaires were distributed to individuals extensive should the data collection on each case be? From a
involved in organising or delivering the programmes to research point of view it is tempting to promote the use of rig-
collect data on the variables indicated by the hypotheses. orous approaches, but we believe that it is not realistic or nec-
Where possible, information on the success of implemen- essary to evaluate each and every quality improvement project
tation was derived from evaluations within the projects. with the same level of rigour required by research. Simple
These data were used to test the predefined hypotheses. evaluations can help to identify the methods that are most
The results indicated that successful implementation of acceptable to clinical staff and appear to result in change of
physical exercise programmes was associated with larger clinical performance. The probability that effective methods
investment by organisations in the programme, a prevail- will be rejected on the basis of such evaluations appears to be
ing view that audit and evaluation were relevant, and a small because rigorous evaluations such as randomised trials
local tradition of innovation in health care services. usually show smaller (and not larger) effects than simple
Although the number of cases is usually much lower than evaluations.
the number of variables, defining hypotheses a priori pro- Evaluations of small scale projects can also contribute to
vides some protection against associations found by more generalised learning and inform scientific knowledge
chance. about quality improvement in health care. They can help to
provide insight into causality if some sort of control is
included in the design. A randomised trial is the ideal type of
systematic reviews or meta-analyses, case study researchers evaluation, but it is inefficient to trial interventions before
are testing methodological approaches to pool results across they have been proved to be promising in small scale
similar studies. For example, Dopson and colleagues22 reported evaluation.24 This is particularly relevant for organisational
an attempt to pool data across a suite of seven related studies and structural changes which require large scale expensive
examining the diffusion of innovations in health care. This trials. Multiple case studies may be particularly useful for
involved a multi-staged approach to critically review and testing the relevance of factors associated with a programme
summarise the findings of individual studies before identify- or its organisational context. Process evaluations help to
ing themes that were common across the studies. These understand the mechanism of causality better and contribute
themes were then verified by independent analysis of the data, to the evidence on a specific intervention in this way. From a
followed by collective discussion and simultaneous analysis. research perspective, these two designs can be used for stud-
ies that are equivalent to early phase studies in pharmaceuti-
Process evaluations cal research and are performed before large clinical trials.25
Methods used for the in-depth study of local projects can also
be helpful when undertaking evaluations of quality improve- CONCLUSIONS
ment initiatives using other research designs which explicitly Implementing change is complex and the processes involved
aim at generalised knowledge. For example, a randomised are still not fully understood. Quality improvement projects
controlled trial (RCT) may be set up to evaluate the effective- are undertaken in many different settings and the knowledge

www.qshc.com
214 Harvey, Wensing

3 Cox S, Wilcock P, Young J. Improving the repeat prescribing process in

Key messages a busy general practice. A study using continuous quality improvement
methodology. Qual Health Care 1999;8:119–25.
4 Solberg LI, Mosser G, McDonald S. The three faces of performance
• Focused audits of the impact of local quality improvement measurement: improvement, accountability and research. Jt Comm J Qual
projects help those involved to learn from the project. Improve 1997;23:135–47.
• In-depth study of local projects using developmental 5 Johnston G, Crombie IK, Davies HTO, et al. Reviewing audit: barriers
and facilitating factors for effective clinical audit. Qual Health Care
approaches to evaluation, case study methodology, or 2000;9:23–36.
process evaluations can provide important insight into how 6 Ovretveit J. Evaluating health interventions. Buckingham: Open
and why programmes work in practice. University Press, 1998.
• These evaluations are characterised by their overall 7 Lewin K. Frontiers in group dynamics: social planning and action
research. Human Relations 1947;1:143–53.
approach rather than by the use of specific techniques for 8 Waterman H, Tillen D, Dickson R, et al. Action research: a systematic
sampling, data collection, or data analysis. review and guidance for assessment. Health Technol Assess 2001;5(23).
• Further development of the methodology for evaluation of 9 Argyris C. On organisational learning. Cambridge, Massachusetts:
local quality improvement projects is recommended. Blackwell Business, 1992.
10 Schon DA. Educating the reflective practitioner. London: Jossey Bass,
1988.
11 Selim P, Bashford C, Grossman C. Evidence-based practice: tap water
cleansing of leg ulcers in the community. J Clin Nurs 2001;10:372–9.
gained from these projects is important to help increase our 12 Yin RK. Case study research: design and methodology. London: Sage,
understanding of implementing effective change. It is useful 1989.
13 Fitzgerald L. Case studies as a research tool. Qual Health Care
to distinguish between evaluation undertaken to enable 1999;8:75.
learning within the project and evaluation that aims to 14 Dawson S, Sutherland K, Dopson S, et al. Changing clinical practice:
contribute to more generalised learning and inform scientific views about the management of adult asthi-na. Qual Health Care
1999;8:253–61.
knowledge about quality improvement in health care. The 15 Ferlie E, Wood M, Fitzgerald L. Some limits to evidence-based medicine:
appropriate methodology for evaluation needs to be elabo- a case study from elective orthopaedics. Qual Health Care
1999;8:99–107.
rated, as not all interventions can or need to be tested in con- 16 Dopson S, Miller R, Dawson S, et al. Influences on clinical practice: the
trolled trials.26 A range of methods can be applied to evaluate case of glue ear. Qual Health Care 1999;8:108–18.
small scale improvement projects, including focused audit 17 Dopson S, Gabbay J, Locock L, et al. Evaluation of the PACE
programme. In: Understanding the role of opinion leaders in improving
studies, developmental research, multiple case studies, and clinical effectiveness. Soc Sci Med 2001;53:745–57.
process evaluations within RCTs. These approaches are 18 Locock L, Chambers D, Surender R, et al. Evaluation of the Welsh
clinical effectiveness initiative national demonstration project. In:
characterised by their overall research approach rather than by Understanding the role of opinion leaders in improving clinical
the specific techniques for data collection or data analysis. effectiveness. Soc Sci Med 2001;53:745–57.
Further development of the methodology for evaluation of 19 Shih FJ. Triangulation in nursing research: issues of conceptual clarity
and purpose. J Advan Nurs 1998;28:631–41.
small scale improvement projects is recommended. 20 Miles MB, Huberman AM. Qualitative data analysis. A sourcebook of
new methods. London: Sage Publications, 1984.
21 Laurant M, Harmsen M, Wensing M. Effective implementation of
..................... physical exercise programmes for older adults. Nijmegen: Centre for
Quality of Care Research, 2001.
Authors’ affiliations 22 Dopson S, Fitzgerald L, Ferlie E, et al. No magic targets! Changing
G Harvey, Royal College of Nursing Institute, Radcliffe Infirmary, Oxford clinical practice to become more evidence based. Health Care Manage
OX2 6HE, UK Rev 2002;27:35–47.
M Wensing, Centre for Quality of Care Research, University Medical 23 Hulscher MEJL, Laurant MGH, Grol RPTM. Process evaluation on quality
Centre St Radboud, 6500 HB Nijmegen, The Netherlands improvement interventions. Qual Saf Health Care 2003;12:40–6.
24 Eccles M, Grimshaw J, Campbell M, et al. Research designs for studies
evaluating the effectiveness of change and improvement strategies. Qual
Saf Health Care 2003;12:47–52.
REFERENCES 25 Freemantle N, Wood J, Crawford F. Evidence into practice,
1 Langley GJ, Nolan KM, Nolan TW, et al. The improvement guide. San experimentation and quasi experimentation: are the methods up to the
Francisco: Jossey Bass Publishers, 1996. task? J Epidemiol Community Health 1998;52:75–81.
2 Morrell C, Harvey G. The clinical audit handbook: improving the quality 26 Black N. Why do we need observational studies to evaluate the
of health care. London: Ballière Tindall, 1999. effectiveness of health care. BMJ 1996;312:1215–8.

www.qshc.com