Gill Westhorp

The Saskatchewan Ministry of Health has committed to a multi-million dollar investment toward the implementation of Lean methodology across the province's healthcare system. Originating as a production line discipline (the Toyota Production System), Lean has evolved to encompass process improvements including inventory management, waste reduction and quality improvement techniques. With an initial focus on leadership, strategic alignment, training and the creation of a supportive infrastructure (Lean promotion offices), the goal in Saskatchewan is a whole health system transformation that produces "better health, better value, better care, and better teams." Given the scope and scale of the initiative and the commitment of resources, it is vital that a comprehensive, longitudinal evaluation plan be implemented to support ongoing decision-making and program design. The nature of the initiative also offers a unique opportunity to contribute to health quality improvement science by advancing our understanding of the implementation and evaluation of complex, large-scale healthcare interventions. The purpose of this article is to summarize the background to Lean in Saskatchewan and the proposed evaluation methods.

Lean as a management system has been increasingly adopted in health care settings in an effort to enhance quality, capacity and safety, while simultaneously containing or reducing costs. The Ministry of Health in the province of Saskatchewan, Canada has made a multi-million dollar investment in Lean initiatives to create "better health, better value, better care, and better teams", affording a unique opportunity to advance our understanding of the way in which Lean philosophy, principles and tools work in health care. In order to address the questions, "What changes in leadership practices are associated with the implementation of Lean?" and "When leadership practices change, how do the changed practices contribute to subsequent outcomes?", we used a qualitative, multi-stage approach to work towards developing an initial realist program theory. We describe the implications of realist assumptions for evaluation of this Lean initiative. Formal theories including Normalization Process Theory, Theories of Double Loop and Organization Leaning and the Theory of Cognitive Dissonance help understand this initial rough program theory. Data collection included: key informant consultation; a stakeholder workshop; documentary review; 26 audiotaped and transcribed interviews with health region personnel; and team discussions. A set of seven initial hypotheses regarding the manner in which Lean changes leadership practices were developed from our data. We hypothesized that Lean, as implemented in this particular setting, changes leadership practices in the following ways. Lean: a) aligns the aims and objectives of health regions; b) authorizes attention and resources to quality improvement and change management c) provides an integrated set of tools for particular tasks; d) changes leaders' attitudes or beliefs about appropriate leadership and management styles and behaviors; e) demands increased levels of expertise, accountability and commitment from leaders; f) measures and uses data effectively to identify actual and relevant local problems and the root causes of those problems; and g) creates or supports a 'learning organization' culture. This study has generated initial hypotheses and realist program theory that can form the basis for future evaluation of Lean initiatives. Developing leadership capacity and culture is theorized to be a necessary precursor to other systemic and observable changes arising from Lean initiatives.

Background: There is growing interest in theory-driven, qualitative and mixed-method approaches to systematic review as an alternative to (or to extend and supplement) conventional Cochrane-style reviews. These approaches offer the potential to expand the knowledge base in policy-relevant areas-for example by explaining the success, failure or mixed fortunes of complex interventions. However, the quality of such reviews can be difficult to assess. This study aims to produce methodological guidance, publication standards and training resources for those seeking to use the realist and/or meta-narrative approach to systematic review. Methods/design: We will: [a] collate and summarise existing literature on the principles of good practice in realist and meta-narrative systematic review; [b] consider the extent to which these principles have been followed by published and in-progress reviews, thereby identifying how rigour may be lost and how existing methods could be improved; [c] using an online Delphi method with an interdisciplinary panel of experts from academia and policy, produce a draft set of methodological steps and publication standards; [d] produce training materials with learning outcomes linked to these steps; [e] pilot these standards and training materials prospectively on real reviews-in-progress, capturing methodological and other challenges as they arise; [f] synthesise expert input, evidence review and real-time problem analysis into more definitive guidance and standards; [g] disseminate outputs to audiences in academia and policy. The outputs of the study will be threefold: 1. Quality standards and methodological guidance for realist and meta-narrative reviews for use by researchers, research sponsors, students and supervisors 2. A 'RAMESES' (Realist and Meta-review Evidence Synthesis: Evolving Standards) statement (comparable to CONSORT or PRISMA) of publication standards for such reviews, published in an open-access academic journal. 3. A training module for researchers, including learning outcomes, outline course materials and assessment criteria. Discussion: Realist and meta-narrative review are relatively new approaches to systematic review whose overall place in the secondary research toolkit is not yet fully established. As with all secondary research methods, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency of studies.

Bonell et al. discuss the challenges of carrying out randomised controlled trials (RCTs) to evaluate complex interventions in public health, and consider the role of realist evaluation in enhancing this design (Bonell, Fletcher, Morton, Lorenc, & Moore, 2012). They argue for a " synergistic, rather than oppositional relationship between realist and randomised evaluation " and that " it is possible to benefit from the insights provided by realist evaluation without relinquishing the RCT as the best means of examining intervention causality. " We present counter-arguments to their analysis of realist evaluation and their recommendations for realist RCTs. Bonell et al. are right to question whether and how (quasi-)experimental designs can be improved to better evaluate complex public health interventions. However, the paper does not explain how a research design that is fundamentally built upon a positivist ontological and epistemological position can be meaningfully adapted to allow it to be used from within a realist paradigm. The recommendations for " realist RCTs " do not sufficiently take into account important elements of complexity that pose major challenges for the RCT design. They also ignore key tenets of the realist evaluation approach. We propose that the adjective 'realist' should continue to be used only for studies based on a realist philosophy and whose analytic approach follows the established principles of realist analysis. It seems more correct to call the approach proposed by Bonell and colleagues 'theory informed RCT', which indeed can help in enhancing RCTs.

Background: Meta-narrative review is one of an emerging menu of new approaches to qualitative and mixed-method systematic review. A meta-narrative review seeks to illuminate a heterogeneous topic area by highlighting the contrasting and complementary ways in which researchers have studied the same or a similar topic. No previous publication standards exist for the reporting of meta-narrative reviews. This publication standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The project's aim is to produce preliminary publication standards for meta-narrative reviews. Methods: We (a) collated and summarized existing literature on the principles of good practice in meta-narrative reviews; (b) considered the extent to which these principles had been followed by published reviews, thereby identifying how rigor may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, meta-narrative reviews, policy and/or publishing to produce and iteratively refine a draft set of methodological steps and publication standards; (d) provided real-time support to ongoing meta-narrative reviews and the open-access RAMESES online discussion list so as to capture problems and questions as they arose; and (e) synthesized expert input, evidence review and real-time problem analysis into a definitive set of standards.

The Saskatchewan Ministry of Health has committed to a multi-million dollar investment toward the implementation of Lean methodology across the province's healthcare system. Originating as a production line discipline (the Toyota Production System), Lean has evolved to encompass process improvements including inventory management, waste reduction and quality improvement techniques. With an initial focus on leadership, strategic alignment, training and the creation of a supportive infrastructure (Lean promotion offices), the goal in Saskatchewan is a whole health system transformation that produces "better health, better value, better care, and better teams." Given the scope and scale of the initiative and the commitment of resources, it is vital that a comprehensive, longitudinal evaluation plan be implemented to support ongoing decision-making and program design. The nature of the initiative also offers a unique opportunity to contribute to health quality improvement science by advancing our understanding of the implementation and evaluation of complex, large-scale healthcare interventions. The purpose of this article is to summarize the background to Lean in Saskatchewan and the proposed evaluation methods.

Background: There is growing interest in realist synthesis as an alternative systematic review method. This approach offers the potential to expand the knowledge base in policy-relevant areas-for example, by explaining the success, failure or mixed fortunes of complex interventions. No previous publication standards exist for reporting realist syntheses. This standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The project's aim is to produce preliminary publication standards for realist systematic reviews. Methods: We (a) collated and summarized existing literature on the principles of good practice in realist syntheses; (b) considered the extent to which these principles had been followed by published syntheses, thereby identifying how rigor may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, realist research, policy and/or publishing to produce and iteratively refine a draft set of methodological steps and publication standards; (d) provided real-time support to ongoing realist syntheses and the open-access RAMESES online discussion list so as to capture problems and questions as they arose; and (e) synthesized expert input, evidence syntheses and real-time problem analysis into a definitive set of standards. Results: We identified 35 published realist syntheses, provided real-time support to 9 ongoing syntheses and captured questions raised in the RAMESES discussion list. Through analysis and discussion within the project team, we summarized the published literature and common questions and challenges into briefing materials for the Delphi panel, comprising 37 members. Within three rounds this panel had reached consensus on 19 key publication standards, with an overall response rate of 91%. Conclusion: This project used multiple sources to develop and draw together evidence and expertise in realist synthesis. For each item we have included an explanation for why it is important and guidance on how it might be reported. Realist synthesis is a relatively new method for evidence synthesis and as experience and methodological developments occur, we anticipate that these standards will evolve to reflect further methodological developments. We hope that these standards will act as a resource that will contribute to improving the reporting of realist syntheses. To encourage dissemination of the RAMESES publication standards, this article is co-published in the Journal of Advanced Nursing and is freely accessible on Wiley Online Library (http://www.wileyonlinelibrary.com/journal/jan). Please see related article

BackgroundMeta-narrative review is one of an emerging menu of new approaches to qualitative and mixed-method systematic review. A meta-narrative review seeks to illuminate a heterogeneous topic area by highlighting the contrasting and complementary ways researchers have studied the same or a similar topic. No previous publication standards exist for the reporting of meta-narrative reviews. This publication standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The project's aim is to produce preliminary publication standards for meta-narrative reviews.Meta-narrative review is one of an emerging menu of new approaches to qualitative and mixed-method systematic review. A meta-narrative review seeks to illuminate a heterogeneous topic area by highlighting the contrasting and complementary ways researchers have studied the same or a similar topic. No previous publication standards exist for the reporting of meta-narrative reviews. This publication standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The project's aim is to produce preliminary publication standards for meta-narrative reviews.DesignA mixed method study synthesising data between 2011 to 2012 from a literature review, online Delphi panel and feedback from training, workshops and email list.A mixed method study synthesising data between 2011 to 2012 from a literature review, online Delphi panel and feedback from training, workshops and email list.MethodsWe: (a) collated and summarized existing literature on the principles of good practice in meta-narrative reviews; (b) considered the extent to which these principles had been followed by published reviews, thereby identifying how rigor may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, meta-narrative reviews, policy, and/or publishing to produce and iteratively refine a draft set of methodological steps, and publication standards; (d) provided real-time support to ongoing meta-narrative reviews and the open-access RAMESES online discussion list so as to capture problems and questions as they arose; and (e) synthesized expert input, evidence review, and real-time problem analysis into a definitive set of standards.We: (a) collated and summarized existing literature on the principles of good practice in meta-narrative reviews; (b) considered the extent to which these principles had been followed by published reviews, thereby identifying how rigor may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, meta-narrative reviews, policy, and/or publishing to produce and iteratively refine a draft set of methodological steps, and publication standards; (d) provided real-time support to ongoing meta-narrative reviews and the open-access RAMESES online discussion list so as to capture problems and questions as they arose; and (e) synthesized expert input, evidence review, and real-time problem analysis into a definitive set of standards.ResultsWe identified nine published meta-narrative reviews, provided real-time support to four ongoing reviews, and captured questions raised in the RAMESES discussion list. Through analysis and discussion within the project team, we summarized the published literature, and common questions and challenges into briefing materials for the Delphi panel, comprising 33 members. Within three rounds this panel had reached consensus on 20 key publication standards, with an overall response rate of 90%.We identified nine published meta-narrative reviews, provided real-time support to four ongoing reviews, and captured questions raised in the RAMESES discussion list. Through analysis and discussion within the project team, we summarized the published literature, and common questions and challenges into briefing materials for the Delphi panel, comprising 33 members. Within three rounds this panel had reached consensus on 20 key publication standards, with an overall response rate of 90%.ConclusionsThis project used multiple sources to draw together evidence and expertise in meta-narrative reviews. For each item we have included an explanation for why it is important and guidance on how it might be reported. Meta-narrative review is a relatively new method for evidence synthesis and as experience and methodological developments occur, we anticipate that these standards will evolve to reflect further theoretical and methodological developments. We hope that these standards will act as a resource that will contribute to improving the reporting of meta-narrative reviews.This project used multiple sources to draw together evidence and expertise in meta-narrative reviews. For each item we have included an explanation for why it is important and guidance on how it might be reported. Meta-narrative review is a relatively new method for evidence synthesis and as experience and methodological developments occur, we anticipate that these standards will evolve to reflect further theoretical and methodological developments. We hope that these standards will act as a resource that will contribute to improving the reporting of meta-narrative reviews.

BackgroundThere is growing interest in realist synthesis as an alternative systematic review method. This approach offers the potential to expand the knowledge base in policy-relevant areas – for example, by explaining the success, failure or mixed fortunes of complex interventions. No previous publication standards exist for reporting realist syntheses. This standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The project's aim is to produce preliminary publication standards for realist systematic reviews.There is growing interest in realist synthesis as an alternative systematic review method. This approach offers the potential to expand the knowledge base in policy-relevant areas – for example, by explaining the success, failure or mixed fortunes of complex interventions. No previous publication standards exist for reporting realist syntheses. This standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The project's aim is to produce preliminary publication standards for realist systematic reviews.DesignA mixed method study synthesising data between 2011–2012 from a literature review, online Delphi panel and feedback from training, workshops and email list.A mixed method study synthesising data between 2011–2012 from a literature review, online Delphi panel and feedback from training, workshops and email list.MethodsWe: (a) collated and summarized existing literature on the principles of good practice in realist syntheses; (b) considered the extent to which these principles had been followed by published syntheses, thereby identifying how rigour may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, realist research, policy and/or publishing to produce and iteratively refine a draft set of methodological steps and publication standards; (d) provided real-time support to ongoing realist syntheses and the open-access RAMESES online discussion list to capture problems and questions as they arose; and (e) synthesized expert input, evidence syntheses and real-time problem analysis into a definitive set of standards.We: (a) collated and summarized existing literature on the principles of good practice in realist syntheses; (b) considered the extent to which these principles had been followed by published syntheses, thereby identifying how rigour may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, realist research, policy and/or publishing to produce and iteratively refine a draft set of methodological steps and publication standards; (d) provided real-time support to ongoing realist syntheses and the open-access RAMESES online discussion list to capture problems and questions as they arose; and (e) synthesized expert input, evidence syntheses and real-time problem analysis into a definitive set of standards.ResultsWe identified 35 published realist syntheses, provided real-time support to 9 ongoing syntheses and captured questions raised in the RAMESES discussion list. Through analysis and discussion within the project team, we summarized the published literature and common questions and challenges into briefing materials for the Delphi panel, comprising 37 members. Within 3 rounds this panel had reached consensus on 19 key publication standards, with an overall response rate of 91%.We identified 35 published realist syntheses, provided real-time support to 9 ongoing syntheses and captured questions raised in the RAMESES discussion list. Through analysis and discussion within the project team, we summarized the published literature and common questions and challenges into briefing materials for the Delphi panel, comprising 37 members. Within 3 rounds this panel had reached consensus on 19 key publication standards, with an overall response rate of 91%.ConclusionsThis project used multiple sources to develop and draw together evidence and expertise in realist synthesis. For each item we have included an explanation for why it is important and guidance on how it might be reported. Realist synthesis is a relatively new method for evidence synthesis and as experience and methodological developments occur, we anticipate that these standards will evolve to reflect further methodological developments. We hope that these standards will act as a resource that will contribute to improving the reporting of realist syntheses.This project used multiple sources to develop and draw together evidence and expertise in realist synthesis. For each item we have included an explanation for why it is important and guidance on how it might be reported. Realist synthesis is a relatively new method for evidence synthesis and as experience and methodological developments occur, we anticipate that these standards will evolve to reflect further methodological developments. We hope that these standards will act as a resource that will contribute to improving the reporting of realist syntheses.

Medical Education 2012: 46: 89–96Context  Education is a complex intervention which produces different outcomes in different circumstances. Education researchers have long recognised the need to supplement experimental studies of efficacy with a broader range of study designs that will help to unpack the ‘how’ and ‘why’ questions and illuminate the many, varied and interdependent mechanisms by which interventions may work (or fail to work) in different contexts.Methods  One promising approach is realist evaluation, which seeks to establish what works, for whom, in what circumstances, in what respects, to what extent, and why. This paper introduces the realist approach and explains why it is particularly suited to education research. It gives a brief introduction to the philosophical assumptions underlying realist methods and outlines key principles of realist evaluation (designed for empirical studies) and realist review (the application of realist methods to secondary research).Discussion  The paper warns that realist approaches are not a panacea and lists the circumstances in which they are likely to be particularly useful.

Introduction: Realist evaluation is an increasingly
popular methodology in health services research. For
realist evaluations (RE) this project aims to: develop
quality and reporting standards and training materials;
build capacity for undertaking and critically evaluating
them; produce resources and training materials for lay
participants, and those seeking to involve them.
Methods: To achieve our aims, we will: (1) Establish
management and governance infrastructure; (2) Recruit
an interdisciplinary Delphi panel of 35 participants with
diverse relevant experience of RE; (3) Summarise
current literature and expert opinion on best practice in
RE; (4) Run an online Delphi panel to generate and
refine items for quality and reporting standards;
(5) Capture ‘real world’ experiences and challenges of
RE—for example, by providing ongoing support to
realist evaluations, hosting the RAMESES JISCmail list
on realist research, and feeding problems and insights
from these into the deliberations of the Delphi panel;
(6) Produce quality and reporting standards; (7) Collate
examples of the learning and training needs of
researchers, students, reviewers and lay members in
relation to RE; (8) Develop, deliver and evaluate
training materials for RE and deliver training
workshops; and (9) Develop and evaluate information
and resources for patients and other lay participants in
RE (eg, draft template information sheets and model
consent forms) and; (10) Disseminate training
materials and other resources.
Planned outputs: (1) Quality and reporting standards
and training materials for RE. (2) Methodological
support for RE. (3) Increase in capacity to support and
evaluate RE. (4) Accessible, plain-English resources for
patients and the public participating in RE.
Discussion: The realist evaluation is a relatively new
approach to evaluation and its overall place in the is
not yet fully established. As with all primary research
approaches, guidance on quality assurance and
uniform reporting is an important step towards
improving quality and consistency.