Booth Taking A Systematic Approach To Your Literature Review

00_Booth_et al_2e_Prelims.
indd 3 19-Apr-16 6:30:38 PM

SAGE Publications Ltd Andrew Booth, Anthea Sutton and Diana Papaioannou 2016
1 Olivers Yard
55 City Road First edition published 2012, reprinted 2013
London EC1Y 1SP This second edition first published 2016
SAGE Publications Inc. Apart from any fair dealing for the purposes of research or
2455 Teller Road private study, or criticism or review, as permitted under the
Thousand Oaks, California 91320 Copyright, Designs and Patents Act, 1988, this publication
may be reproduced, stored or transmitted in any form, or
SAGE Publications India Pvt Ltd by any means, only with the prior permission in writing of
B 1/I 1 Mohan Cooperative Industrial Area the publishers, or in the case of reprographic reproduction,
Mathura Road in accordance with the terms of licences issued by
New Delhi 110 044 the Copyright Licensing Agency. Enquiries concerning
reproduction outside those terms should be sent to the
SAGE Publications Asia-Pacific Pte Ltd publishers.
3 Church Street
#10-04 Samsung Hub
Singapore 049483
Editor: Mila Steele

Editorial assistant: Alysha Owen
Library of Congress Control Number: 2015955561
Production editor: Tom Bedford
Copyeditor: Audrey Scriven
British Library Cataloguing in Publication data
Marketing manager: Ben Griffin-Sherwood
Cover design: Shaun Mercier
A catalogue record for this book is available from
Typeset by: C&M Digitals (P) Ltd, Chennai, India
the British Library
Printed and bound by CPI Group (UK) Ltd,
Croydon, CR0 4YY
ISBN 978-1-4739-1245-8
ISBN 978-1-4739-1246-5 (pbk)
At SAGE we take sustainability seriously. Most of our products are printed in the UK using FSC papers and boards.
When we print overseas we ensure sustainable papers are used as measured by the PREPS grading system.
We undertake an annual audit to monitor our sustainability.
00_Booth_et al_2e_Prelims.indd 4 19-Apr-16 6:30:38 PM

2
TAKING A SYSTEMATIC
APPROACH TO YOUR
LITERATURE REVIEW
in a nutshell
Why be systematic?
The literature review has a long pedigree as an area of academic and research endeavour.
However, review methods can now fulfil an even greater variety of purposes and answer a
greater range of question types.
Reviewing the literature in a systematic way helps the author to be clear, to build confidence
in their work and demonstrate the rigour of their methods.
Recent years have seen significant developments in research synthesis, starting from
within healthcare and migrating across a large number of disciplines and fields.
Systematic is a requirement for any type of research, so a literature review should not be
any different from this standard.
Systematic approaches are used to reduce the potential for bias within a review.
The main types of review can be defined by the extent to which they harness the Search,
AppraisaL, Synthesis and Analysis (SALSA) elements.
INTRODUCTION
Fink (2005) succinctly defines a literature review as a systematic, explicit, and reproduc-
ible method for identifying, evaluating, and synthesising the existing body of completed and
02_Booth_et al_2e_Ch_02.indd 9 19-Apr-16 6:30:34 PM

10 Systematic Approaches to a Successful Literature Review
recorded work produced by researchers, scholars, and practitioners. Particularly noticeable is the
word systematic, key to the title and content of this book. All reviews share the requirement
of original empirical research, namely to be systematic. Different types of review (see Box 2.1)
should differ only in the degree to which they are systematic according to each reviews role and
function and each type should help by telling you what exactly they have and have not done.
box 2.1
Some common types of review

Critical review
Integrative review
Literature review
Mapping review/systematic map
Meta-analysis
Mixed studies review/mixed methods review
Overview
Qualitative systematic review/qualitative evidence synthesis
Rapid review
Realist synthesis
Scoping review
State-of-the-art review
Systematic review
Systematic search and review
Systematised review
Umbrella review
(Note: These different types of review are defined and explained later in this chapter. For the
present we acknowledge a plethora of terms for systematic approaches to a literature review.)
Hart (1998) unpicks the detail of the review process, focusing on the essential components, the
documents themselves:
the selection of available documents on the topic written from a particular standpoint
to fulfil certain aims or express certain views on the nature of the topic and how it is to be
investigated, and the effective evaluation of these documents in relation to the research being
proposed.

Taking a Systematic Approach to Your Literature Review 11
To perform this effectively you need processes to ensure that you complete this task in an
efficient manner and fulfil it to the right quality. By quality we mean appropriate breadth
and depth, rigor and consistency, clarity and brevity, and effective analysis and synthesis
(Hart, 1998).
WHY IS THE LITERATURE REVIEW SO IMPORTANT?

Bem (1995) notes that authors of literature reviews are at risk for producing mind-numbing
lists of citations and findings that resemble a phone book impressive case, lots of numbers,
but not much plot. When we make decisions based on evidence it makes sense to use the best
that is available. By and large the best evidence for many decisions comes from a systematic
review of all the evidence. Mulrow (1995) argues that reviewing in this way is a search for the
whole truth, rather than just one part of it, and is thus a fundamentally scientific activity.
The reviewer uses a specific and reproducible method to identify, select and appraise studies of
a previously agreed level of quality (either including all studies or only those that pass a mini-
mum quality threshold) that are relevant to a particular question. The results of the studies are
then analysed and summarised. Synthesising evidence helps us to find out what we know and
dont know about what works and what doesnt work. A good research synthesis can gener-
ally give us the most trustworthy answer to a specific review question, and it can identify gaps
in our knowledge that require further research. It also communicates the strength of the avail-
able evidence and the quality of included studies, thereby indicating how much confidence
practitioners, service users, managers, policy makers, and the popular media should have in
the results.
A research synthesis can also help us to find out how well a policy, programme, technique
or intervention works in different subgroups of users and inform us about its potential to cause
harm. Some research syntheses shed light on the pros and cons of different ways of organising
or delivering services or policies. A research synthesis that includes considerations of cost can
help shape our judgements about whether a chosen policy or course of action provides good
value for money.
Yet another reason to synthesise the results of different studies of a given intervention is
to learn whether findings are consistent across multiple studies. Light and Pillemer (1984)
write that disagreements among findings are valuable [and that] conflicts can teach us a
lot. Thus we can identify settings in which a particular social policy might succeed, circum-
stances under which an educational programme might work best, or which dose of a drug is
most effective.
A good research synthesis frequently highlights weaknesses in the evidence and argues for
further research. What should service users, policy makers and others decide in the absence of
evidence? Even when a research synthesis shows strong, unambiguous evidence to support one
course of action, politics may make that reviews findings less influential than well-coordinated
lobbying. As Chalmers and colleagues (2002) observe, Research synthesis sometimes yields
unwelcome results that challenge strongly held opinions and other vested interests. Yet even if
the recommendations from a research synthesis are disregarded, its very existence encourages
more transparency about the role of other factors in decision making. No matter how well they
are done, research syntheses are not a panacea for all problems, but they do offer a valuable aid
to decision making.

WHERE DOES A LITERATURE REVIEW FIT

WITHIN THE CONTEXT OF RESEARCH?
Literature reviews are pervasive in academia and policy development. However, a literature
review is typically showcased in three particular contexts:
1. As a major component of a dissertation, thesis or other academic deliverable.

2. As a peer-reviewed publication, typically in a journal or, depending upon the discipline, as a book
chapter.
3. As a report resulting from a funded research project or other commissioned research or
consultancy.
Each of these contexts shapes the final review product. For example, a literature review as
part of a dissertation or thesis should be innovative. A student should be reflexive about their
methods (McGhee etal., 2007) and demonstrate their personal growth through the methodol-
ogy (Daigneault etal., 2014). Students are expected to demonstrate their knowledge about a
particular field of study, including vocabulary, theories, key variables and phenomena, and its
methods and history (Randolph, 2009). Furthermore a student must demonstrate that they
are sensitised to the influential researchers and research groups in the field. Of course a lit-
erature review from a thesis may subsequently become a legitimate and publishable scholarly
document (LeCompte etal., 2003).
In producing a peer-reviewed publication the reviewer faces the challenge of squeezing a
wealth of data into the tight constraints of a journals house style and word limits, occasionally
overflowing into online supplementary materials. Journals, and their parent disciplines, demon-
strate very different approaches to handling reviews. Some journals will not even consider review
articles. In contrast, other journals celebrate annual review-type commissioned overviews and
literature surveys.
Finally if you are reviewing the literature for a funded research project or for consultancy
you may face tight time constraints, a demand for answers rather than issues, and readers
who want to sidestep the methodology and cut straight to the results or findings. Systematic
reviews can yield:
information about the nature and extent of a problem, and the potential benefits, harms,
uncertainties, and costs of interventions and policies. Policymakers may also want to know
about the impact on different groups in various settings [and to] answer questions about
how best to disseminate information and innovations; whether the interventions are appro-
priate to local culture and context; and about the factors influencing study outcomes. (Sweet
and Moynihan, 2007)
Chapter 10 acknowledges that different types of presentation suit different audiences and
purposes. For the moment, we distinguish between reviews for knowledge support and those
for decision support (Mays etal., 2005; Pope etal., 2007). Reviews for knowledge support sum-
marise and synthesise research evidence (focusing on what currently exists and is known about
a topic). They may highlight gaps in the evidence base as a target for future research, just as a
thesis might do. Reviews for decision support go further in bringing the existing evidence to
bear on a particular issue or problem. Gaps in the evidence base lead to the supplementary ques-
tion and what shall we do about this issue or problem in the meantime? Unsurprisingly,

Lavis and colleagues (2005) found that managers are interested in reviews that help them to
decide whether to start, stop or change a programme, how to accommodate new programmes
or services tried elsewhere within their own organisation, and how to bring about change.
The knowledge supportdecision support spectrum is often caricatured within healthcare by
the Cochrane Review at one end and the health technology assessment at the other. In
actuality, Cochrane reviews increasingly strive to contribute to decision making while health
technology assessments look beyond the immediate problem to recommend future commis-
sioned primary research.
WHAT TYPES OF RESEARCH QUESTION ARE

SUITABLE FOR LITERATURE REVIEW?
It is essential that a literature review is question-led. The question, along with the purpose of
the review, the intended deliverables and the intended audience, determines how the data
are identified, collected and presented. Some conclude that a literature review is only useful
where a significant body of literature is already known to exist. However, even if a researcher
believes that they are the first to examine a particular intervention, policy or programme they
should confirm this from the previously published literature. Thus they avoid presenting
islands without continents (i.e. falsely claiming innovation) (Clarke and Chalmers, 1998).
Cross-fertilisation of methods from a related field may similarly save a researcher from having
to develop a solution from scratch.
Although you might well be the first to consider a particular issue most literature reviews
assume that at least one other researcher has at least considered, if not addressed, your ques-
tion. Your question is shaped and influenced by the goal and focus of the review (Hart, 1998;
Randolph, 2009):
Effectiveness questions: What effect does intervention X, compared with intervention Y, have
on outcome Z? What are the relative cost-benefits of X versus Y?
Methodology questions: What research methods have previously been used to investigate
phenomenon X? What are the respective strengths and weaknesses of such methods?
Conceptual questions: How has phenomenon X been identified and defined? Which theories
have been used to explain phenomenon X? Which theory provides the best fit to findings from
empirical studies? What are the main unresolved controversies? What are the underpinning
epistemological and ontological foundations for the discipline?
WHY REVIEW THE LITERATURE?

The internet age has witnessed the so-called information explosion (Major and Savin-Baden,
2010). With increasing numbers of articles being published and improved open access to many
of these articles, it is becoming almost impossible to navigate even the more specialised subject
domains. At an individual level we face information overload. We are overwhelmed by the
volume of information we encounter and unable to retrieve the information we need. Is there
a solution to this situation? One possible approach is to become information literate put

simply this means acquiring the skills covered in Chapters 5 to 10 in this book that will allow us
to locate, evaluate and use information effectively.
Although technologies are always changing, database interfaces and search engines are con-
tinually being upgraded, and new topics emerge with regularity, the skills of information literacy
you acquire during your literature review will equip you beyond the duration of your project or
dissertation and throughout your career and working life.
Many authors agree on the purposes of a literature review (see Box 2.2), irrespective of disci-
pline (Cooper, 1989; Bruce, 1994, 1996; Hart, 1998; Galvan, 1999).
box 2.2
Purposes for a literature review

To place each work in the context of how it contributes to an understanding of the subject
under review.
To describe how each work relates to the others under consideration.
To identify new ways to interpret, and shed light on gaps in, previous research.
To identify and resolve conflicts across seemingly contradictory previous studies.
To identify what has been covered by previous scholars to prevent you needlessly duplicating
their effort.
To signpost the way forward for further research.
To locate your original work within the existing literature.
Which of the above points most accurately capture your reason(s) for conducting a literature
review?
Your reasons may include taking stock of what has gone before and identifying a niche for
your own research. The literature may help you design your own research. You may identify
a theory against which you will explore a specified hypothesis. You may need to select tools,
instruments or scales to help you conduct your research. You may seek to identify gaps that offer
opportunities as future research questions. Reasons for reviewing the literature are mirrored
within the following brief history of research synthesis.
A BRIEF HISTORY OF RESEARCH SYNTHESIS

Research synthesis can have no absolute start date. Very early on in history humankind needed
to record what had previously occurred, to compare experiences across cases, and to build up a
knowledge base of what was now known and what remained unknown. Those who chronicle
research synthesis typically identify specific landmarks (see Table 2.1) and then link these with
a dotted non-continuous line.
Probably the most cited account originates from three eminent proponents, Chalmers, Hedges
and Cooper (2002), in A brief history of research synthesis. A more extensive treatment of the

same topic is available in The Handbook of Research Synthesis and Meta-analysis (Cooper et al.,
2009). Both works attest to the fact that research synthesis has migrated from a select number of
disciplines to pervade almost every area of academic activity.
There is nothing particularly novel about research synthesis. In 1753, James Lind, the Scottish
naval surgeon who was instrumental in the first randomised controlled trial, recognised
the value of systematic methods for identifying, extracting and appraising information from
individual studies as a protection against a biased interpretation of research:
As it is no easy matter to root out prejudices it became requisite to exhibit a full and
impartial view of what had hitherto been published on the scurvy by which the sources
of these mistakes may be detected. Indeed, before the subject could be set in a clear and
proper light, it was necessary to remove a great deal of rubbish. (James Lind, cited in
Dunn, 1997)
Gathering the published research, getting rid of the rubbish, and summarising the best of
what remains characterises the science of research synthesis. Subsequent developments in
information retrieval, documentation and document delivery have made it considera-
bly less of a challenge to identify, acquire and interpret the scattered body of published and
unpublished research.
Others trace the origins of research synthesis to seventeenth-century astronomers who com-
bined data from related studies to introduce greater precision to their individual observations
(Petticrew, 2001). However, a more obvious heritage lies with statistician Karl Pearson who
identified the need to bring together multiple small studies to arrive at a definitive opin-
ion of the evidence on inoculations against fever (Pearson, 1904). Three years later, Joseph
Goldberger, a scientist in the United States, reviewed 44 studies of typhoid fever and then
abstracted and pooled data from 26 of the 44 studies (Chalmers etal., 2002).
Similar work was undertaken within agriculture by Ronald Fisher and colleagues in the 1930s.
However, it was not until the 1970s that formal procedures for synthesising studies were labelled
as meta-analysis by Gene Glass (1976) and other social science colleagues.
Table 2.1 Milestones in the history of research synthesis
Date Milestones
1753 James Lind published first systematic review

1904 Pearson published landmark review on effects of vaccines against typhoid
1976 Glass coined term meta-analysis
1984 Light and Pillemer, Summing Up
1987 Mulrow, The medical review article: state of the science
1989 Enkin and colleagues, Effective Care in Pregnancy and Childbirth
1992 Antman and colleagues illustrated value of cumulation of findings
1993 Launch of Cochrane Collaboration
1994 Establishment of the UK NHS Centre for Reviews and Dissemination
2000 Founding of Campbell Collaboration

In 1984 Light and Pillemer published Summing Up: The Science of Reviewing Research, arguing
that new methods of research synthesis applied to many fields, including health, education and
psychology:
Without a clear picture of where things stand now, simply adding one new study to the exist-
ing morass is unlikely to be very useful For science to be cumulative, an intermediate step
between past and future research is necessary: synthesis of existing evidence. (Light and
Pillemer, 1984)
Three years later, Mulrow (1987) delivered a damning verdict on 50 review or progress articles
published in four leading medical journals. Only one of the 50 reviews had clearly specified
methods of identifying, selecting, and validating included information. She concluded that
Current reviews do not routinely use scientific methods to identify, assess, and synthe-
size information. On the contrary, these reviews are often subjective, scientifically unsound,
and inefficient (Mulrow, 1987). Towards the end of the 1980s Iain Chalmers and colleagues
responded to this challenge (Enkin etal., 1989), laying a platform for collaborative synthesis,
from which the Cochrane Collaboration, and its sibling the Campbell Collaboration,
were launched.
At about the same time Oxman and Guyatt concluded from a survey of published reviews
that:
Our data suggest that experts, on average, write reviews of inferior quality; that the greater
the expertise the more likely the quality is to be poor; and that the poor quality may be related
to the strength of the prior opinions and the amount of time they spend preparing a review
article. (Oxman and Guyatt, 1993)
As evidence-based policy and evidence-based practice became popular in the 1990s

other disciplines, such as education, started to acknowledge the importance of research syn-
theses (Evans and Benefield, 2001). Not everyone welcomed such a trend (Hammersley, 2001).
Nevertheless, evidence-based policy and practice gained pace in other fields such as social
care and management (see, for example, Davies et al., 2000; Trinder and Reynolds, 2000).
Governments also began to fund initiatives to support research syntheses, particularly system-
atic reviews (Davies, 2000).
Many other organisations began producing research syntheses during the 1990s. Funding
bodies began to require a systematic review of existing research before considering applications
for funding a primary study. The UK Medical Research Council requires a researcher to demon-
strate that a systematic review has been undertaken before it will commission a new trial (Clark
and Horton, 2010). This ensures that the question has not already been answered, and that the
results of previous research are used in designing the new trial.
Subsequent years have been characterised by the increasing popularity of literature review
variants and derivatives, often within time-constrained policy windows and requiring a flexible
toolkit of systematic approaches. More than anything the proliferation of so many variants
emphasises that the extent to which a review is systematic lies on a continuum running from
brief evidence summary through to gold standard systematic review.

WHAT IS THE PLACE OF THEORY IN LITERATURE REVIEW?

For Webster and Watson a successful literature review:
creates a firm foundation for advancing knowledge. It facilitates theory development, closes
areas where a plethora of research exists, and uncovers areas where research is needed.
(Webster and Watson, 2002, emphasis added)
Initially theory was considered an unwelcome distraction from the fundamentally pragmatic
intent of the science of research synthesis. Pragmatic disciplines such as health services
research are largely atheoretical at least in the sense of not acknowledging a specific theo-
retical contribution. As you move outwards to contiguous disciplines such as public health,
health promotion and nursing, theory is more plentiful. This is similarly the case for such
disciplines as social care, education, management, and even information systems. A more
complex panorama is emerging, particularly within the context of understanding how
complex interventions might be understood to work (De Silva et al., 2014). Reviews may
be broadly characterised as generating, exploring or testing theory (Gough et al., 2012).
Review techniques may range from the hypothesis testing of meta-analysis through to use
of interpretive techniques such as meta-ethnography and critical interpretive syn-
thesis. Other techniques such as realist synthesis explore the application of mid-range
programme theory to a mix of quantitative and qualitative data. Furthermore concept
analysis explicitly seeks to define, expand and extend the theoretical underpinnings
of a target concept, and may prove a useful starting point for a review where definitions
are contested.
Literature reviews offer multiple opportunities to engage and interact with theory (see Table 2.2).
Methods for searching the literature for theory in a systematic way are beginning to be proposed
(Booth etal., 2013b; Booth and Carroll, 2015). An evidence synthesis may scope out theories relat-
ing to a particular issue (Campbell et al., 2014) or seek to consolidate current theory, creating
a meta-model. Alternatively a literature review can be used to generate new, overarching theo-
ries and interpretations (Campbell etal., 2014). Subsequently, the literature may offer a data set
against which existing theories can be examined and modified. Finally, where a literature review is
unable to fully explain differences that exist between apparently similar mechanisms or contexts,
a reviewer may introduce theory in an attempt to analyse such differences.
THE TRADITIONAL NARRATIVE REVIEW, THE SYSTEMATIC

REVIEW AND SYSTEMATIC APPROACHES
All research, literature reviews included, is required to be systematic in its conduct, analysis
and/or presentation:
Who would want reviews to be unsystematic, if by systematic we mean no more than properly
carried out, taking account of all the relevant evidence, and making reliable judgements about
its validity and implications? On this definition, to produce a systematic review is simply to do
the job of reviewing well. (Hammersley, 2002)

Table 2.2 Examples of the interplay of literature review and theory
Type of review Reference Interplay of literature review and theory
Concept analysis Teamwork: a concept analysis. Used Walker and Avants approach to guide analysis (2005). Searched bibliographic
(Xyrichis and Ream, 2008) databases, Internet search engines and hand searches (19762006). Based on analysis,
02_Booth_et al_2e_Ch_02.indd 18
proposed definition for teamwork, and identified essential ingredients for it to take place.
Creation of Fostering implementation of health Used snowball sampling approach to identify published theories and component constructs.
meta-model services research findings into practice: Combined constructs across published theories while removing redundancy or overlap.
a consolidated framework for advancing Created Consolidated Framework for Implementation Research (CFIR) as overarching
implementation science (Damschroder typology for implementation theory development.
etal., 2009)
Meta-ethnography Using meta-ethnography to synthesise Four papers about lay meanings of medicines arbitrarily chosen. Used Noblit and Hares
qualitative research: a worked example seven-step process for meta-ethnography (1988). Six key concepts were derived from
(Britten etal., 2002) interpretations in chosen papers and four reviewer interpretations were constructed to make
sense of these. Produced middle-range theories as hypotheses to be tested by other researchers.
Meta-narrative Storylines of research in diffusion of Identified 13 key meta-narratives from literatures of rural sociology, clinical epidemiology,
review innovation: a meta-narrative approach marketing and organisational studies. Researchers in different traditions had conceptualised,
to systematic review (Greenhalgh explained and investigated diffusion of innovations differently. Reconciled seemingly
etal., 2005) contradictory data, systematically exposing and exploring tensions between research paradigms.
Realist review (to Realist review to understand the Complemented existing Cochrane Review by exploring detailed information on context,
complement an efficacy of school feeding programmes mechanisms, and outcomes of interventions and theories in 18 trials of school feeding
effectiveness review) (Greenhalgh etal., 2007) programmes.
Review of theories Healthcare professionals intentions Reviewed literature on factors influencing health professionals behaviours based on social
and behaviours: a systematic review cognitive theories: 78 studies met inclusion criteria. Most used theory was Theory of
of studies based on social cognitive Reasoned Action or its extension Theory of Planned Behaviour.
theories (Godin etal., 2008)
Review of use of A systematic review of the use of theory Reviewed use of theory in 235 evaluations of guideline dissemination and implementation
theory in the design of guideline dissemination studies (19661998). Classified theory according to type of use (explicitly theory based,
and implementation strategies and some conceptual basis, and theoretical construct used) and stage of use (choice/design of
interpretation of the results of rigorous intervention, process/mediators/moderators, and post hoc/explanation).
evaluations (Davies etal., 2010)
Scoping review Disseminating research findings: what Searched 12 electronic databases to identify/describe conceptual/organising frameworks
should researchers do? A systematic used in guiding dissemination activity. Narrative synthesis undertaken: 33 frameworks met
scoping review of conceptual inclusion criteria underpinned by three theoretical approaches (persuasive communication,
frameworks (Wilson etal., 2010) diffusion of innovations theory and social marketing).
As precursor to The place of the literature review in Explored the role literature review can play in grounded theory methodology.
conceptual work grounded theory research (Dunne, 2011)
19-Apr-16 6:30:35 PM
However, reviewers have not always recognised this fact, as illustrated by our hypothetical recipe
for a traditional literature review:
Take a simmering topic, extract the juice of an argument, add the essence of one filing cabi-
net, sprinkle liberally with your own publications and sift out the work of noted detractors or
adversaries.
Greenhalgh (2014) describes journalistic reviews of college students where, if research did not fit
with their proposed theory, material is simply left out. Bias, or systematic error, may exist at the
identification, selection, synthesis, analysis and interpretation stages of a review process which
may demonstrate implicit, idiosyncratic methods (Mulrow et al., 1997). Frequently the much
maligned narrative review (Mulrow etal., 1997; Greenhalgh, 2014) is referred to as a traditional
review, conventional review, or even more damningly, as a non-systematic review. However,
some non-systematic reviews openly ally themselves to different traditions of reviewing literature.
Consequently a review can be poorly conducted, poorly reported, or both (Shea etal., 2002).
WHY BE SYSTEMATIC?
While many arguments are advanced for the desirability of systematic approaches to reviewing
the literature, we have identified at least three principal considerations, i.e. clarity, validity and
auditability.
Many systematic approaches target the clarity of scholarly communication. The structure of a
systematic review makes it easier to navigate and interpret. Clear methodology makes it easier to
judge what the reviewers have and have not done. A focused question and explicit search strate-
gies help to clarify scope and terminology. Stated inclusion and exclusion criteria allow readers
to recognise why particular articles known to them have not been included. Graphical, textual
and tabular features combine to reveal rather than conceal.
A second consideration addresses internal validity. The review product must be defensible against
potential bias (see Chapter 8). Potential biases include selection bias where a reviewer selects
primary research studies that support his/her prior beliefs. Biases also include publication bias
(where investigators, reviewers or editors differentially submit or accept manuscripts based on
the direction or strength of the study findings) (Gilbody and Song, 2000). Systematic approaches
require that items are selected for inclusion on the basis of their relevance and rigour, not on
whether they report a favourable outcome or whether their results are intrinsically interesting.
Finally an emphasis on transparency leads to concerns with auditability how do we know
that the reviewers conclusions are grounded in the data retrieved from the review process and
not an argument fabricated to support a prior conclusion?
Systematic research syntheses are important, too, as quality control. Peer-review serves more
as a check on a primary studys published report. The original data themselves seldom are
subject to scrutiny. (Rousseau etal., 2008)
The science of research synthesis is thus populated by flowcharts of numbers of included studies,
supplementary documents, and appendices relating to search strategies, sample data extraction
forms, and increasing numbers of reporting standards (Moher etal., 2014).

exercise 2.1
How systematic is this review?

Disciplines differ in the extent to which they have adopted systematic approaches to
research synthesis. Identify a review article within your own subject area or discipline.
(For example, search for review, overview, or meta-analysis in the title or abstract.)
To what extent can your chosen review be described as systematic? Exercise 2.1 sug-
gests that you construct a grid as in the following example and complete it with your own
observations.
1 Features that make this review 2 Features that make this review
appear SYSTEMATIC appear NON-SYSTEMATIC
Briefly reflect on what the term systematic means to you. Do you consider being system-
atic a positive or a negative attribute? Table 2.3 suggests some words associated with being
systematic or unsystematic.
Table 2.3 Words associated with being systematic or unsystematic
Systematic Either Unsystematic

Explicit Creative Implicit
Transparent Comprehensive Opaque
Methodical Imaginative Whimsical
Objective Publishable Subjective
Standardised Stimulating Variable
Structured Topical Chaotic
Reproducible Well-written Idiosyncratic
The availability of time, personnel and money may further constrain the quality of the final
review product (see Chapter 4). A reviewer should select a model of review that is most appro-
priate to the purpose required (Tricco et al., 2011; Kastner et al., 2012). The reviewer should
be explicit about any limitations inherent to the chosen approach. For example, a scoping
review that offers a snapshot of a particular topic (Arksey and OMalley, 2005) does not usually
attempt quality assessment. In contrast, systematic reviews (or overviews):
use (and describe) specific, explicit and therefore reproducible methodological strategies
to identify, assemble, critical appraise and synthesise all relevant issues on a specific topic.
(Carney and Geddes, 2002)
Well-conducted systematic reviews should improve the reliability and accuracy of conclusions,
being clearly allied to the scientific method:

Syntheses systematically identify where research findings are clear (and where they arent), a
key first step to establishing the conclusions science supports. (Rousseau etal., 2008)
The results of systematic reviews are rarely unequivocal and require careful reading and
interpretation (Hopayian, 2001). Narrative reviews often offer a snapshot of prevalent
knowledge at a particular point in time whereas many systematic reviews explicitly aim
to monitor and capture incident (i.e. emerging) knowledge. Systematic reviews may be
designed to be updated periodically to take into account the emergence of new evidence.
Cooper (1988a, 1988b) identifies four possible literature review approaches:
1) Exhaustive coverage, citing all relevant literature.

2) Exhaustive coverage with selective citation.
3) Representative coverage (discussion of works which typify particular groupings in the literature).
4) Coverage of pivotal works.
Coopers choice of exhaustivity often considered synonymous with the systematic review is
to be preferred to the discourse of comprehensiveness, because the former carries the implication
of finite (i.e. review-dependent) resources. Early on in the development of the systematic review
process it was considered desirable to retrieve all studies on a clearly defined topic. Rousseau and
colleagues (2008) state:
Systematic means comprehensive accumulation, transparent analysis, and reflective inter-

pretation of all empirical studies pertinent to a specific question. Reliance upon any sampling
or subset of the literature risks misrepresenting its diversity in findings, outcomes methods,
and frames of reference.
More recently there is increasing recognition that even the most exhaustive (and exhausting!)
search cannot hope to identify the entire universe of studies on the most specific of topics.
Interestingly Coopers third and fourth approaches broaden our toolkit to include a wider range
of sampling approaches, increasingly recognised as a more appropriate response than a one-size-
fits-all comprehensive approach (Suri, 2011). Fitness for purpose is the appropriate aspiration,
underpinned by the trade-off of rigour versus relevance (Bennett etal., 2005). Notably, however,
selective citation a danger inherent in the second of Coopers approaches is regarded as a
potential source of bias (Song etal., 2010). Exercise 2.2 asks you to compare a systematic review
with a traditional review.
exercise 2.2
Compare a systematic and a traditional review

Identify a systematic review in an area of interest to you and also identify a conventional
review in a similar or related topic. Place the two reviews side by side and briefly make a list
of the differences between the two reviews.

Clarification of terminology
Up to now we have used the umbrella term research synthesis unless specifically referring to a
particular type of review. Many authors have attempted a taxonomy of literature review types
(Strike and Posner, 1983; Cooper, 1988a, 1988b; Grant and Booth, 2009; Gough etal., 2012; Par
etal., 2014; Whittemore etal., 2014). Such a task is challenging because a review may be character-
ised across such variables as the purpose of the review (as with a mapping review), the types of
included study (as with a systematic review of randomised controlled trials), the nature of included
data (as with the qualitative systematic review), the type of question being addressed (as with
the effectiveness review), the phenomenon being investigated (as in the case of meta-theory or
meta-method), and the underlying intent (meta-ethnography for theory generation or real-
ist synthesis for theory verification). Other characteristics relate to the context of the review (as
for the rapid evidence assessment) or to the underpinning philosophy regarding subsequent
use of the review (as with best evidence synthesis). This book will attempt to define the char-
acteristics of each type of review wherever possible. However, we would also acknowledge that:
Only a handful of review types possess prescribed and explicit methodologies and many of
the labels used fall short of being mutually exclusive [we recognise] that there is a lack of
unique distinguishing features for the most common review types, whilst highlighting that
some common features do exist. (Grant and Booth, 2009)
Researchers have a considerable incentive to invent a new label or new form of review that
differs slightly from its predecessors rather than to seek standardisation. In Chapters 4 to 10
we focus on the techniques and ingredients of the review process (systematic approaches) as a
counter balance to methodological pigeon-holing.
Recently approaches to the synthesis of the literature have been characterised as either
aggregative or interpretive/configurative (Weed, 2005; Gough etal., 2012; Sandelowski
etal., 2012) (see Table 2.4). Aggregative reviews bring together studies on a similar topic such
that each additional study adds weight to a shared finding. Bringing studies together in this
way necessitates assumptions about how similar studies are to one another (homogeneity). In
practice all studies are different (with regard to the population studied, in how a procedure is
implemented, in how an outcome is measured, etc.). The reviewer, and indeed the reader, have
to judge whether studies are more alike than different.
By implication aggregative approaches can reach a point at which sufficient studies have estab-
lished a finding beyond statistical doubt. Cumulative meta-analyses can demonstrate a point beyond
which subsequent studies possess a certain degree of informational redundancy (Antman etal.,
1992). Nevertheless, in theory at least, additional studies hold the potential to overturn a previous
finding. Aggregative reviews therefore represent an ongoing attempt to identify studies that have pre-
viously been missed, particularly if their absence might reveal a previously neglected systematic bias.
In contrast, interpretive/configurative reviews seek to broaden our understanding of
a particular intervention or phenomenon. Each study holds the potential to contribute addi-
tional insights and also contribute to the overall picture. Of course this potential is more
limited where a broad consensus exists and authors report the same type of insights. This is
analogous to theoretical saturation within primary research (Dixon-Woods etal., 2005). In
theory, theoretical saturation should be less frequent than in primary research as, unlike inter-
viewees, authors are not independent informants (Dixon-Woods etal., 2006). Indeed authors

are incentivised to report innovative insights. To resists theoretical saturation a reviewer will
make particularly strenuous attempts to sample from other fields or types of literature. Whereas
aggregative reviews implicitly value the average result which adds strength to the overall result
from multiple similar studies, interpretive reviews place particular value on identifying the
disconfirming case (Booth et al., 2013a). Interpretive reviews often seek to contribute to
theory (Walsh and Downe, 2005).
Occasionally the integrative review is used synonymously with the interpretive review.
While this terminology was appropriate for as long as quantitative and qualitative reviews were
separate endeavours, the mixed methods review now seeks to harness the power of stories
alongside the power of numbers (Pluye and Hong, 2014). We reserve the integrative review for
cases where both types of data are brought together (Whittemore and Knafl, 2005), typically to
produce a whole that is greater than the sum of its parts (Strike and Posner, 1983).
Table 2.4 Configurative/interpretive or aggregative?
Consider whether the overall intent of your intended review is interpretive or aggregative
Configurative/Interpretive Aggregative
Will my question develop and Is my question fixed and focused, allowing me

change as new insights emerge to decide authoritatively whether studies are
from the literature? relevant or not?
If new papers fail to contribute new insights will I Will I keep searching until I have exhausted all
purposively move on to different types of sources? likely sources?
Is the main value of each new study in Is the main value of each new study in adding
adding a different insight to what I am weight to, or confirming, what has previously
investigating? been found?
Is my principal focus on the exception to the Is my principal focus on the mean or
rule? average overall result?
Will my data be presented primarily as themes Will my data be presented primarily as tables
or models with accompanying commentary? and graphs with accompanying commentary?
INTRODUCING THE SALSA FRAMEWORK

We have chosen to characterise review types against four critical steps in the review process that
we embody within the mnemonic SALSA (Search, AppraisaL, Synthesis and Analysis) (Grant and
Booth, 2009). The strength and quality of each step contribute to the overall signal emitted by
the review, whereas biases increase the distracting noise (Edwards et al., 1998). Thus a scop-
ing review is characterised as a broad-brush approach to finding the most notable studies in
the field, minimal attempts to evaluate them for quality, a rudimentary attempt at synthesis
(perhaps through listing, tabulation or mapping), and an analysis that caricatures the quantity
and distribution of the literature. In contrast a gold standard systematic review, as endorsed
by the Cochrane Collaboration, prescribes an exhaustive search of the literature, checklist-driven
quality assessment, complex synthesis using textual, numerical, graphical and tabular methods
and sophisticated analysis (for example, for differences between subgroups, the differential effects
of study groups, and the likelihood of missing studies). Between these two extremes lie numerous
variants with different levels of input at the four key stages (see Table 2.5).

Table 2.5 Types of review
Type of
Review Description Search Appraisal Synthesis Analysis
Critical Aims to demonstrate extensive Seeks to identify most No. Evaluates by Narrative, Significant component:
review research and critical evaluation significant items in field. contribution. conceptual, seeks to identify
of quality. Goes beyond mere chronological. conceptual contribution to
description to include degree of embody existing or derive
analysis and conceptual innovation. new theory.
Typically results in hypothesis or
model.
Integrative Utilises broadest type of research Exhaustive search to Reports coded Tabular Creativity, critical analysis
review review methods to include both identify maximum number according to (matrices, of data and data displays
experimental and non-experimental of eligible primary quality but not graphs, key to comparison and
research in order to understand sources, using two or necessarily charts, or identification of important
more fully a phenomenon of more strategies. Purposive excluded. networks) patterns and themes.
concern. Integrative reviews sampling may be combined usually
combine data from theoretical and with exhaustive search if according to a
empirical literature. appropriate. framework.
Literature Examines recent or current Possibly exhaustive. Possibly. Narrative. Chronological, conceptual,
review literature. Can cover wide range thematic, etc.
of subjects at various levels of
completeness and exhaustivity. May
include research findings.
Mapping Maps out and categorises existing As time allows. No. Graphical. Characterises quantity
review/ literature from which to commission Tabular. and quality of literature,
systematic further reviews and/or primary perhaps by study design
map research by identifying gaps in and other key features.
research literature. May identify need for
primary/secondary
research.
Meta- Statistically combines results of Exhaustive. May use May determine Graphical. Numerical analysis.
analysis quantitative studies to provide Funnel plot to assess inclusion/exclusion Tabular.
precise effect of results. completeness. and/or sensitivity Narrative.
analyses.
19-Apr-16 6:30:35 PM
Type of
Mixed studies Combines methods that include Sensitive search or Generic appraisal Narrative. May look for correlations
review/ mixed review component (usually separate quantitative and instrument or Tabular. between characteristics
methods systematic). Specifically combines qualitative strategies. separate appraisal Graphical or use gap analysis to
review review approaches such as processes with (to integrate identify aspects absent in
quantitative with qualitative corresponding quantitative one literature but missing
research or outcome with process checklists. and in other.
studies. qualitative
studies).
Overview Surveys literature and describe its Depends on how Depends on Depends Chronological, conceptual,
characteristics. systematic methods are. how systematic on how thematic, etc.
methods are. systematic
methods are.
Narrative.
Tabular.
Qualitative Integrates or compares findings Selective or purposive. Typically to Qualitative, Thematic may include
systematic from qualitative studies. Looks for mediate messages narrative conceptual models.
review/ themes or constructs in or across not for inclusion/ synthesis.
qualitative individual studies. exclusion.
evidence
synthesis
Rapid review Assesses what is already known As time allows, uses As time allows, Narrative. Quantities of literature and
about policy or practice issue. systematic review methods uses systematic Tabular. overall quality/direction of
to search existing research. review methods to effect of literature.
critically appraise
existing research.
(Continued)
19-Apr-16 6:30:35 PM
(Continued)
Type of
Realist Synthesises large and diverse Mainly iterative and Privileges Narrative, Key output is programme
synthesis selection of literature to inform purposive. relevance over causal chains theory/ies of target
policy revision, design effective rigour. and graphical. intervention, specifying
interventions and identify how and why programme/
potentially effective and innovative service is thought to
interventions. cause intended outcomes
(theory building), and
then testing assumptions
against further evidence,
to strengthen and refine it
(theory testing).
Scoping Identifies nature and extent of As time allows. May include No. Narrative. Quantity and quality of
review research evidence (including research in progress. Tabular. literature, perhaps by
ongoing research). study design and other
features. Attempt to
specify viable review.
State-of-the- Addresses current matters. May Exhaustive coverage of No. Narrative. Current state of
art review offer new perspectives on issue or current literature. Tabular. knowledge, priorities
point out area for further research. for future investigation,
research limitations.
Systematic Combines strengths of critical Exhaustive. Possibly. Narrative. What is known,
search and review with exhaustive search Tabular. recommendations for
review process. Addresses broad questions practice.
to produce best evidence synthesis.
Umbrella Summarises results from systematic Exhaustive search for Possibly using a Graphical and What is known and
review reviews on a topic. reviews only. review specific tabular. research gaps for primary
appraisal tool research or further
(for example, reviews.
AMSTAR).
(Adapted from Grant and Booth, 2009)
19-Apr-16 6:30:35 PM
The SALSA approach is anticipated by other authors: for example, Bruce (2001) specifies that
a review requires the analysis and synthesis of previous work in such a manner that new under-
standings of that work are uncovered, and the way is opened for new scholarship or research.
More recently Major and Savin-Baden (2010) highlighted the importance of synthesis, analysis
and interpretation. For us analysis fuses the technical aspects of juxtaposing studies with the
more imaginative and explorative aspects signified by interpretation.
SYSTEMATIC APPROACHES
By systematic approaches we refer to those elements of a literature review that, either indi-
vidually or collectively, contribute to the methods being both explicit and reproducible.
Systematic approaches (see Box 2.3) are evidenced in both the conduct and presentation
of the literature review and epitomised in the formal method of systematic review.
Exercise 2.3 asks you to examine how systematic a review is. Specifically systematic
approaches include:
systematic approaches to literature searching, as seen in the scoping review and the
mapping review (see Chapter 6);
systematic approaches to quality assessment (appraisal) of the literature, as seen in an
integrative review (see Chapter 7);
systematic approaches to synthesis of the literature, as seen in such techniques as
meta-analysis, meta-ethnography, realist synthesis and thematic synthesis (see
Chapters 8 and 9);
systematic approaches to analysis of the robustness and validity of review findings such as sub-
group analysis, either qualitative or quantitative, or sensitivity analysis (see Chapters 8 and 9);
systematic approaches to the presentation of review findings using narrative, tabular,
numerical and graphical approaches (see Chapters 8, 9, and 10).
box 2.3
What does systematic look like?

A priori specification of planned review methods/protocol.
A clearly focused question.
Clear, explicit criteria for inclusion and exclusion.
Documentation of search process: sources and strategies.
Use of tables and boxes to make methods explicit.
Use of tables to summarise study characteristics.
An explicit mechanism to handle quality assessment.
Exploration of assumptions, limitations and areas of uncertainty.
Use of tables and graphics to support interpretation of data.
Appendices including search strategies, sample data extraction and quality assessment tools.
Explicit Declarations of Interest.

exercise 2.3
How systematic is that review?

Look through the following fictional abstract describing a structured review of the literature in
light of what you have already learnt regarding the search, appraisal, synthesis, analysis (SALSA)
framework. Which elements of this abstract provide evidence of a systematic approach?
performing X in Y
A structured review of the literature

Abstract
{Two brief sentences of Background}. A literature search was conducted across {list of
Databases and Internet sources} of studies that evaluated X. Information on the type of activity,
sample and setting, endpoints, and study design were extracted. Studies were classified based
on a modified {Hypothetical Worthy} model. Four categories of activity were identified: actor,
decision-support, involvement and systems. The search strategy and selection criteria yielded
21 articles. Eleven studies used an actor activity; two studies used a decision support activity,
seven used an involvement activity, and one used a systems intervention. The overall quality of
research was uneven: research design nine studies were quasi-experimental in nature, end-
point measures were not consistent three did not perform statistical analysis. Sample char-
acteristics varied dramatically. In conclusion, the number of high-quality studies of X remains
limited. Methodological limitations include measurement of an inappropriate surrogate mea-
sure when measurement of an endpoint would be more valid. Further research is needed to
understand how each type of activity improves the quality of performing X in a Y setting.
SUMMARY
Like all science, research synthesis is evolving and uncertain. For example, the application of
statistical methods for pooling and synthesising the quantitative results of different studies
meta-analysis is steadily improving, though considerable challenges remain (Egger et al.,
2002). While much early development focused on systematic reviews of evidence about health-
care interventions drugs, therapies, technologies the principles of research synthesis remain
the same regardless of the subject matter under review. This chapter has examined the rela-
tive advantages of systematic over more traditional approaches to reviewing the literature. It
has identified the main stages in the review process. Systematic approaches to the literature
can improve the clarity, validity and auditability of an otherwise well-conducted conventional
review. The key steps of search, appraisal, synthesis and analysis (SALSA) can help to characterise
the differences between various types of review. Systematic approaches to the literature address
three problems neatly encapsulated by Rousseau and colleagues (2008):

the misuse of existing research, the overuse of limited or inconclusive findings, and the under
use of research evidence with substantive implications for understanding and working with
organisations.
key learning points
Research synthesis has a long pedigree and in recent years the stimulus of evidence-based
policy and practice has seen it spread across multiple fields and disciplines.
Surveys of research syntheses consistently reveal poor reporting of review methods. All
literature reviews should be systematic but reviews differ in the degree to which they are
systematic and how explicitly their methods are reported.
Systematic approaches to the literature attempt to address known deficiencies by offering
greater clarity, internal validity and auditability.
Different types of review can be characterised by the extent to which they undertake the
four steps of search, appraisal, synthesis and analysis (SALSA).
frequently asked questions (FAQs)
FAQ 2.1 Will it always be possible to be systematic in conducting

a literature review?
Yes. We contend that all types of literature review should contain elements of systematicity.
Any type of research should access an underpinning system. The flexibility of the review
method allows for different degrees of being systematic. An associated challenge relates
to how you describe what you have done, especially where your methods are iterative and
recursive. Describing what you looked for, what you found, what you did with what you found,
and what you concluded from what you found are shared characteristics of any review. Text,
tables and diagrams are different ways to make your review appear more systematic and
your reader will benefit if you are able to communicate what you have done as explicitly as
possible.
FAQ 2.2 If all reviews are systematic should I call my review a

systematic review?
By all means call your review a systematic review if you have followed closely a prescribed
systematic review methodology, such as outlined by the Cochrane Handbook (Higgins and
Green, 2011) or the Centre for Reviews and Dissemination (2009) guidelines. If a review is
truly a systematic review then it will carry extra academic weight and likely be more highly
cited. However, this label must be earned and not bestowed wantonly. If you call your review a
systematic review and this is not the case then it will rightly receive a rough ride from editors,
peer reviewers, supervisors and examiners. Choosing the most accurate label for your review
will ensure that it is viewed appropriately.
(Continued)

(Continued)
FAQ 2.3 Must a minimum amount of literature exist if I am planning

to conduct a literature review?
This depends upon why you are doing your literature review and what your intended audi-
ence is expecting. If your review is trying to establish whether a study exists that has pre-
viously addressed a specific question then it may be appropriate for you to demonstrate
that you have conscientiously combed all likely sources and still not found any eligible study.
Alternatively you may need to show that you have looked in the right places by itemising
studies that are almost but not quite what you were looking for. If your review is required
to inform a particular course of action then you may need to drop requirements for particular
types of study or literature and widen this to include any source of evidence that matches
your review question.
FAQ 2.4 Must the literature be of a particular quality or study

type to be included in my literature review?
Again this depends on why you are doing the literature review and what your intended
audience is expecting. If your review is trying to establish definitively whether or not a par-
ticular intervention works then simply adding inferior study designs or poor quality papers
is not going to advance this issue. Only good quality studies will confirm or deny whether
the intervention has an effect. In essence you are sieving out inferior studies so that only
good-enough studies remain. However, if you are seeking an overall (all-round) picture of
a phenomenon then even a poor quality study may contain a nugget of valuable insight
(see Chapter 7).
FAQ 2.5 Do I need to be skilled in statistics if I am planning to

conduct a systematic review?
Not necessarily. Systematic reviews can analyse either quantitative literature or qualitative
literature, or even combine both types of literature (mixed methods reviews). Generally
speaking you should ensure that you have skills in reading and interpreting the type of lit-
erature that you intend to review, or plan a route by which you may obtain those skills (such
as through reading or training), or identify a resource (for example, a statistician or research
support service) that you can access as and when you need this. Bear in mind that not all
quantitative reviews are suited to pooling in a meta-analysis. They therefore require numer-
acy rather than statistics. You should also recognise that your skills in qualitative research are
potentially equally important even though these are a less common source of anxiety.
FAQ 2.6 Do I need to include all four steps of search, appraisal,

synthesis and analysis in my literature review?
While the time you spend on each step may vary these four steps are fundamental to the
review process. For example, even if you only examine a collection of full-text articles stored
on your computer, or type some words into Google Scholar, you are conducting a form of
search. Similarly even if you decide to include any type of publication or study on your topic
you are making an implicit policy about quality (appraisal) for your review. We would contend
that you must describe what you have done for each step and match each step to your review
purpose. Frequently analysis, although present, is comparatively neglected when squeezed
out by time constraints.

suggestions for further reading
Mulrow, C.D. (1994) Rationale for systematic reviews. BMJ, 309, 5979.
An influential article that outlines why previous narrative approaches to literature review
might be considered unsatisfactory.
Petticrew, M. (2001) Systematic reviews from astronomy to zoology: myths and misconcep-
tions. BMJ, 322, 13, 98101.
An accessible introductory overview to the idea of systematic reviews and what they aim to do.
Petticrew, M.A. and Roberts, H. (2006) Why do we need systematic reviews? In M.A. Petticrew
and H. Roberts (eds), Systematic Reviews in the Social Sciences. Oxford: Blackwell, 126.
The introductory chapter to a favourite textbook, notable for its clarity and excellent
scholarship.
Pickering, C. and Byrne, J. (2014) The benefits of publishing systematic quantitative litera-
ture reviews for PhD candidates and other early-career researchers. Higher Education
Research & Development,33, 3, 53448.
A good starting point if you have yet to be convinced about the value of systematic reviews
for your own research.
Randolph, J. (2009) A guide to writing the dissertation literature review. Practical Assessment,
Research and Evaluation, 14, 13.
Based within the more traditional, as opposed to systematic, review paradigm this article
nevertheless provides compelling pointers to a high quality literature review.
Steward, B. (2004) Writing a literature review. British Journal of Occupational Therapy, 67,
11, 495500.
A concise and understandable guide to the steps of a systematic review
Volmink, J., Siegfried, N., Robertson, K. and Glmezoglu, A. (2004) Research synthesis and
dissemination as a bridge to knowledge management: the Cochrane Collaboration. Bulletin
of the World Health Organisation, 82, 10, 77883.
Of historical interest on the development of the Cochrane Collaboration, locating reviews
within the broader research-practice gap
REFERENCES
Antman, E.M., Lau, J., Kupelnick, B., Mosteller, F. and Chalmers, T.C. (1992) A comparison
of results of meta-analyses of randomized control trials and recommendations of clinical
experts. Treatments for myocardial infarction. JAMA, 268, 2, 2408.
Arksey, H. and OMalley, L. (2005) Scoping studies: towards a methodological framework.
International Journal of Social Research Methodology, 8, 1, 1932.
Bem, D.J. (1995) Writing a review article for Psychological Bulletin. Psychological Bulletin, 118,
2, 1727.
Bennett, J., Lubben, F., Hogarth, S. and Campbell, B. (2005) Systematic reviews of research in
science education: rigour or rigidity? International Journal of Science Education, 27, 4, 387406.

Booth, A. and Carroll, C. (2015) Systematic searching for theory to inform systematic reviews: is
it feasible? Is it desirable? Health Information & Libraries Journal,32, 3, 22035.
Booth, A., Carroll, C., Ilott, I., Low, L.L. and Cooper, K. (2013a) Desperately seeking dissonance:
identifying the disconfirming case in qualitative evidence synthesis. Qualitative Health
Research, 23, 1, 12641.
Booth, A., Harris, J., Croot, E., Springett, J., Campbell, F. and Wilkins, E. (2013b) Towards a
methodology for cluster searching to provide conceptual and contextual richness for sys-
tematic reviews of complex interventions: case study (CLUSTER). BMC Medical Research
Methodology,13, 1, 118.
Britten, N., Campbell, R., Pope, C., Donovan, J., Morgan, M. and Pill, R. (2002) Using meta
ethnography to synthesise qualitative research: a worked example. Journal of Health Services
Research and Policy, 7, 4, 20915.
Bruce, C. (2001) Interpreting the scope of their literature reviews: significant differences in
research students concerns. New Library World, 102, 4/5, 15866.
Bruce, C.S. (1994) Research students early experiences of the dissertation literature review.
Studies in Higher Education, 19, 2, 21729.
Bruce, C.S. (1996) From neophyte to expert: counting on reflection to facilitate complex concep-
tions of the literature review. In O. Zuber-Skerritt (ed.), Frameworks for Postgraduate Education.
Lismore, NSW: Southern Cross University, 23953.
Campbell, M., Egan, M., Lorenc, T., Bond, L., Popham, F., Fenton, C. and Benzeval, M. (2014)
Considering methodological options for reviews of theory: illustrated by a review of theories
linking income and health. Systematic Reviews, 3, 114. DOI: 10.1186/204640533114.
Carney, S.M. and Geddes, J.R. (2002) Systematic Reviews and Meta-analyses. Evidence in Mental
Health Care. Hove: Brunner Routledge.
Centre for Reviews and Dissemination (CRD) (2009) Systematic Reviews: CRDs Guidance for
Undertaking Reviews in Health Care. York: Centre for Reviews and Dissemination.
Chalmers, I., Hedges, L.V. and Cooper, H. (2002) A brief history of research synthesis. Evaluation
and the Health Professions, 25, 1, 1237.
Clark, S. and Horton, R. (2010) Putting research into context revisited. The Lancet, 376, 9734,
1011.
Clarke, M. and Chalmers, I. (1998) Discussion sections in reports of controlled trials published
in general medical journals. Islands in search of continents? JAMA, 280, 2802.
Cooper, H.M. (1988a) The structure of knowledge synthesis: a taxonomy of literature reviews.
Knowledge in Society, 1, 10426.
Cooper, H.M (1988b) Organizing knowledge syntheses: a taxonomy of literature reviews.
Knowledge, Technology and Policy, 1, 1, 10426.
Cooper, H.M. (1989) Integrating Research: A Guide for Literature Reviews, 2nd edition. Newbury
Park, CA: Sage.
Cooper, H.M., Hedges, L. and Valentine, J. (eds) (2009) The Handbook of Research Synthesis and
Meta-Analysis, 2nd edition. New York: The Russell Sage Foundation.
Daigneault, P.M., Jacob, S. and Ouimet, M. (2014) Using systematic review methods within a PhD
dissertation in political science: challenges and lessons learned from practice.International
Journal of Social Research Methodology,17, 3, 26783.
Damschroder, L.J., Aron, D.C., Keith, R.E., Kirsh, S.R., Alexander, J.A. and Lowery, J.C. (2009)
Fostering implementation of health services research findings into practice: a consolidated
framework for advancing implementation science. Implementation Science, 4, 50.
Davies, H.T.O., Nutley, S.M. and Smith, P.C. (eds) (2000) What Works? Evidence-Based Policy and
Practice in Public Services. Bristol: Policy.
Davies, P. (2000) The relevance of systematic reviews to educational policy and practice. Oxford
Review of Education, 26, 34, 36578.

Davies, P., Walker, A.E. and Grimshaw, J.M. (2010) A systematic review of the use of theory in
the design of guideline dissemination and implementation strategies and interpretation of
the results of rigorous evaluations. Implementation Science, 5, 14.
De Silva, M.J., Breuer, E., Lee, L., Asher, L., Chowdhary, N., Lund, C. and Patel, V. (2014) Theory
of change: a theory-driven approach to enhance the Medical Research Councils framework
for complex interventions.Trials,15, 1, 267.
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B. and Sutton, A. (2005) Synthesising qual-
itative and quantitative evidence: a review of possible methods. Journal of Health Services
Research and Policy, 10, 1, 4553.
Dixon-Woods, M., Bonas, S., Booth, A., Jones, D.R., Miller, T., Sutton, A.J., Shaw, R.L. and Young,
B. (2006) How can systematic reviews incorporate qualitative research? A critical perspective.
Qualitative Research, 6, 1, 2744.
Dunn, P.M. (1997) James Lind (171694) of Edinburgh and the treatment of scurvy. Archives of
Disease in Childhood-Fetal and Neonatal Edition, 76, 1, F64F65.
Dunne, C. (2011) The place of the literature review in grounded theory research. International
Journal of Social Research Methodology, 14, 2, 11124.
Edwards, A.G., Russell, I.T. and Stott, N.C. (1998) Signal versus noise in the evidence base for
medicine: an alternative to hierarchies of evidence? Family Practice, 15, 4, 31922.
Egger, M., Ebrahim, S. and Smith, G.D. (2002) Where now for meta-analysis? International Journal
of Epidemiology, 31, 1, 15.
Enkin, M., Keirse, M.J., Renfrew, M. and Neilson, J. (1989) Effective Care in Pregnancy and
Childbirth. Oxford: Oxford University Press.
Evans, J. and Benefield, P. (2001) Systematic reviews of educational research: does the medical
model fit? British Educational Research Journal, 27, 52741.
Fink, A. (2005) Conducting Research Literature Reviews: From the Internet to Paper, 2nd edition.
London: Sage.
Galvan, J.L. (1999) Writing Literature Reviews. Los Angeles, CA: Pyrczak.
Gilbody, S.M. and Song, F. (2000) Publication bias and the integrity of psychiatry research.
Psychological Medicine, 30, 2538.
Glass, G.V. (1976) Primary, secondary and meta-analysis of research. Educational Researcher, 10,
38.
Godin, G., Blanger-Gravel, A., Eccles, M. and Grimshaw, J. (2008) Healthcare professionals
intentions and behaviours: a systematic review of studies based on social cognitive theories.
Implementation Science, 3, 36.
Gough, D., Thomas, J. and Oliver, S. (2012) Clarifying differences between review designs and
methods. Systematic Reviews, 1, 28.
Grant, M.J. and Booth, A. (2009) A typology of reviews: an analysis of 14 review types and asso-
ciated methodologies. Health Information and Libraries Journal, 26, 2, 91108.
Greenhalgh, T. (2014) How to Read a Paper: The Basics of Evidence-based Medicine, 5th edition.
London: BMJ Books.
Greenhalgh, T., Kristjansson, E. and Robinson, V. (2007) Realist review to understand the effi-
cacy of school feeding programmes. BMJ, 335, 7625, 85861.
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O. and Peacock, R. (2005)
Storylines of research in diffusion of innovation: a meta-narrative approach to systematic
review. Social Science and Medicine, 61, 2, 41730.
Hammersley, M. (2001) On systematic reviews of research literatures: a narrative response to
Evans and Benefield. British Educational Research Journal, 27, 5, 54354.
Hammersley, M. (2002) Systematic or unsystematic, is that the question? Some reflections on
the science, art, and politics of reviewing research evidence. Text of a talk given to the Public
Health Evidence Steering Group of the Health Development Agency, October.

Hart, C. (1998) Doing a Literature Review: Releasing the Social Science Research Imagination.
Thousand Oaks, CA: Sage.
Higgins, J.P. and Green, S. (eds) (2011) Cochrane Handbook for Systematic Reviews of Interventions.
Version 5.1.0 (updated March 2011). Available at: www.cochrane-handbook.org (last accessed
9 March 2016).
Hopayian, K. (2001) The need for caution in interpreting high quality systematic reviews. BMJ,
323, 6814.
Kastner, M., Tricco, A.C., Soobiah, C., Lillie, E., Perrier, L., Horsley, T. and Straus, S.E. (2012)
What is the most appropriate knowledge synthesis method to conduct a review? Protocol for
a scoping review.BMC Medical Research Methodology,12, 1, 114.
Lavis, J.N., Davies, H.T.O., Oxman, A.D., Denis, J-L., Golden-Biddle, K. and Ferlie, E. (2005)
Towards systematic reviews that inform health care management and policy-making. Journal
of Health Services Research and Policy, 10, Suppl. 1, S3548.
LeCompte, M.D., Klinger, J.K., Campbell, S.A. and Menke, D.W. (2003) Editors introduction.
Review of Educational Research, 73, 2, 1234.
Light, R. and Pillemer, D. (1984) Summing Up: The Science of Reviewing Research. Cambridge, MA:
Harvard University Press.
Major, C. and Savin-Baden, M. (2010) An Introduction to Qualitative Research Synthesis: Managing
the Information Explosion in Social Science Research. London: Routledge.
Mays, N., Pope, C. and Popay, J. (2005) Systematically reviewing qualitative and quantitative
evidence to inform management and policy-making in the health field. Journal of Health
Services Research and Policy, 10, Suppl. 1, 620.
McGhee, G., Marland, G.R. and Atkinson, J. (2007) Grounded theory research: literature review-
ing and reflexivity.Journal of Advanced Nursing,60, 3, 33442.
Moher, D.,Altman, D.,Schulz, K.,Simera, I. and Wager, E.(eds) (2014) Guidelines for Reporting
Health Research: A Users Manual. London: BMJ Books.
Mulrow, C. (1987) The medical review article: state of the science. Annals of Internal Medicine,
106, 4858.
Mulrow, C.D. (1995) Rationale for systematic reviews. In I. Chalmers and D. Altman (eds),
Systematic Reviews. London: BMJ Publishing Group, 18.
Mulrow, C.D., Cook, D.J. and Davidoff, F. (1997) Systematic reviews: critical links in the great
chain of evidence. Annals of Internal Medicine, 126, 5, 38991.
Noblit, G.W. and Hare, R.D. (1988) Meta-ethnography: Synthesizing Qualitative Studies. Newbury
Park, CA: Sage.
Oxman, A.D. and Guyatt, G.H. (1993) The science of reviewing research. Annals of the New York
Academy of Science, 703, 12533.
Par, G., Trudel, M.C., Jaana, M. and Kitsiou, S. (2014) Synthesizing information systems knowl-
edge: a typology of literature reviews.Information & Management, 52, 18399.
Pearson, K. (1904) Report on certain enteric fever inoculation statistics. BMJ, iii, 12436.
Petticrew, M. (2001) Systematic reviews from astronomy to zoology: myths and misconceptions.
BMJ, 322, 98101.
Pluye, P. and Hong, Q.N. (2014) Combining the power of stories and the power of numbers:
mixed methods research and mixed studies reviews.Annual Review of Public Health, 35, 2945.
Pope, C., Mays, N. and Popay, J. (2007) Synthesizing Qualitative and Quantitative Health Evidence:
A Guide to Methods. Maidenhead: Open University Press, 1315.
Randolph, J. (2009) A guide to writing the dissertation literature review. Practical Assessment,
Research and Evaluation, 14, 13.
Rousseau, D.M., Manning, J. and Denyer, D. (2008) Evidence in management and organiza-
tional science: assembling the fields full weight of scientific knowledge through syntheses.
Academy of Management Annals, 2, 475515.

Sandelowski, M., Voils, C.I., Leeman, J. and Crandell, J.L. (2012) Mapping the mixed methods
mixed research synthesis terrain. Journal of Mixed Methods Research,6, 4, 31731.
Shea, B., Moher, D., Graham, I., Pham, B. and Tugwell, P. (2002) A comparison of the quality of
Cochrane Reviews and systematic reviews published in paper-based journals. Evaluation and
the Health Professions, 25, 11629.
Song, F., Parekh, S., Hooper, L., Loke, Y.K., Ryder, J., Sutton, A.J., Hing, C., Kwok, C.S., Pang, C.
and Harvey, I. (2010) Dissemination and publication of research findings: an updated review
of related biases. Health Technology Assessment, 14, 8, 1193.
Strike, K. and Posner, G. (1983) Types of synthesis and their criteria. In S. Ward and L. Reed (eds),
Knowledge Structure and Use. Philadelphia: Temple University Press, 34562.
Suri, H. (2011) Purposeful sampling in qualitative research synthesis. Qualitative Research
Journal,11, 2, 6375.
Sweet, M. and Moynihan, R. (2007) Improving Population Health: The Uses of Systematic Reviews.
New York: Milbank Memorial Fund.
Tricco, A.C., Tetzlaff, J. and Moher, D. (2011) The art and science of knowledge synthesis.Journal
of Clinical Epidemiology,64, 1, 1120.
Trinder, L. and Reynolds, S. (eds) (2000) Evidence-based Practice: A Critical Appraisal. Oxford:
Blackwell Science.
Walker, L.O. and Avant, K.C. (2005) Strategies for Theory Construction in Nursing, 4th edition.
Upper Saddle River, NJ: Pearson Prentice Hall.
Walsh, D. and Downe, S. (2005) Meta-synthesis method for qualitative research: a literature
review. Journal of Advanced Nursing, 50, 2, 20411.
Webster, J. and Watson, R.T. (2002) Analyzing the past to prepare for the future: writing a litera-
ture review. MIS Quarterly, 26, 2, 1323.
Weed, M. (2005) Meta interpretation: a method for the interpretive synthesis of qualitative
research. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 6, 1, Art. 37.
Whittemore, R. and Knafl, K. (2005) The integrative review: updated methodology. Journal of
Advanced Nursing, 52, 5, 54653.
Whittemore, R., Chao, A., Jang, M., Minges, K.E. and Park, C. (2014) Methods for knowledge
synthesis: an overview.Heart & Lung: The Journal of Acute and Critical Care, 43, 5, 45361.
Wilson, P.M., Petticrew, M., Calnan, M.W. and Nazareth, I. (2010) Disseminating research find-
ings: what should researchers do? A systematic scoping review of conceptual frameworks.
Implementation Science, 5, 91.
Xyrichis, A. and Ream, E. (2008) Teamwork: a concept analysis. Journal of Advanced Nursing, 61,
2, 23241.

Booth Taking A Systematic Approach To Your Literature Review

Uploaded by

Copyright:

Available Formats

Booth Taking A Systematic Approach To Your Literature Review

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Booth Taking A Systematic Approach To Your Literature Review

Uploaded by

Copyright:

Available Formats

00_Booth_et al_2e_Prelims.

indd 3 19-Apr-16 6:30:38 PM

Editor: Mila Steele

00_Booth_et al_2e_Prelims.indd 4 19-Apr-16 6:30:38 PM

02_Booth_et al_2e_Ch_02.indd 9 19-Apr-16 6:30:34 PM

Some common types of review

Mapping review/systematic map

Mixed studies review/mixed methods review

Qualitative systematic review/qualitative evidence synthesis

Systematic search and review

02_Booth_et al_2e_Ch_02.indd 10 19-Apr-16 6:30:34 PM

WHY IS THE LITERATURE REVIEW SO IMPORTANT?

02_Booth_et al_2e_Ch_02.indd 11 19-Apr-16 6:30:34 PM

WHERE DOES A LITERATURE REVIEW FIT

1. As a major component of a dissertation, thesis or other academic deliverable.

02_Booth_et al_2e_Ch_02.indd 12 19-Apr-16 6:30:34 PM

WHAT TYPES OF RESEARCH QUESTION ARE

WHY REVIEW THE LITERATURE?

02_Booth_et al_2e_Ch_02.indd 13 19-Apr-16 6:30:34 PM

Purposes for a literature review

A BRIEF HISTORY OF RESEARCH SYNTHESIS

02_Booth_et al_2e_Ch_02.indd 14 19-Apr-16 6:30:34 PM

Table 2.1 Milestones in the history of research synthesis

1753 James Lind published first systematic review

02_Booth_et al_2e_Ch_02.indd 15 19-Apr-16 6:30:34 PM

As evidence-based policy and evidence-based practice became popular in the 1990s

02_Booth_et al_2e_Ch_02.indd 16 19-Apr-16 6:30:34 PM

WHAT IS THE PLACE OF THEORY IN LITERATURE REVIEW?

THE TRADITIONAL NARRATIVE REVIEW, THE SYSTEMATIC

02_Booth_et al_2e_Ch_02.indd 17 19-Apr-16 6:30:34 PM

Type of review Reference Interplay of literature review and theory

02_Booth_et al_2e_Ch_02.indd 19 19-Apr-16 6:30:35 PM

How systematic is this review?

Table 2.3 Words associated with being systematic or unsystematic

Systematic Either Unsystematic

02_Booth_et al_2e_Ch_02.indd 20 19-Apr-16 6:30:35 PM

1) Exhaustive coverage, citing all relevant literature.

Systematic means comprehensive accumulation, transparent analysis, and reflective inter-

Compare a systematic and a traditional review

02_Booth_et al_2e_Ch_02.indd 21 19-Apr-16 6:30:35 PM

02_Booth_et al_2e_Ch_02.indd 22 19-Apr-16 6:30:35 PM

Table 2.4 Configurative/interpretive or aggregative?

Will my question develop and Is my question fixed and focused, allowing me

INTRODUCING THE SALSA FRAMEWORK

02_Booth_et al_2e_Ch_02.indd 23 19-Apr-16 6:30:35 PM

(Adapted from Grant and Booth, 2009)

What does systematic look like?

02_Booth_et al_2e_Ch_02.indd 27 19-Apr-16 6:30:35 PM

How systematic is that review?

A structured review of the literature

02_Booth_et al_2e_Ch_02.indd 28 19-Apr-16 6:30:35 PM

key learning points

frequently asked questions (FAQs)

FAQ 2.1 Will it always be possible to be systematic in conducting

FAQ 2.2 If all reviews are systematic should I call my review a

02_Booth_et al_2e_Ch_02.indd 29 19-Apr-16 6:30:35 PM

FAQ 2.3 Must a minimum amount of literature exist if I am planning

FAQ 2.4 Must the literature be of a particular quality or study

FAQ 2.5 Do I need to be skilled in statistics if I am planning to