2010

Program in Evidence-Based Care Handbook

The Program in Evidence-based Care
The Program in Evidence-based Care (PEBC) is an initiative of the Ontario provincial cancer system, Cancer Care Ontario (CCO), supported by the Ontario Ministry of Health and Long Term Care. The PEBC mandate is to improve the quality of cancer care and the lives of Ontarians affected by cancer, through the development, dissemination and evaluation of evidence-based guidelines and other advice documents that are designed to facilitate clinical, planning, and policy decisions about cancer control. The PEBC is based at McMaster University in Hamilton, Ontario, and is academically linked with the Department of Clinical Epidemiology and Biostatistics and the Department of Oncology in the Faculty of Health Sciences.

The PEBC Context and Staff

The PEBC supports a network of Guideline Development Groups (GDGs), including standing committees for specific cancers, called Disease Site Groups (DSGs), and Expert Panels that are convened to address a specific topic. All GDGs are comprised of clinicians, other health care providers and decision makers, and methodologists from across the province. The PEBC also works with the various Clinical Programs and other divisions and initiatives of CCO to identify areas where evidence-based guidance is needed. Figure 1 illustrates the core groups with whom the PEBC is currently partnering. The PEBCs staff includes a core team of trained methodologists, the Research Coordinators, who work in partnership with the GDGs to develop evidence-based guidance on priority topics. These PEBC methodologists develop the evidence base on which guidance will be based, assist in the formulation of recommendations, and generally support the development of the PEBC documents. They are supported by the administrative, quality control, and management teams of the PEBC.
Figure 1. Core Guideline Development Partners of the PEBC.

The Evidence-Based Series

The majority of PEBC documents use a common structural template, the Evidence-based Series (EBS). The EBS has a three-part modular structure: Section 1 Guideline Recommendations: This section profiles the guideline recommendations and might include clinical guidance, organizational guidance, or a combination of the two. It is written in a style that allows it to be read independently from the other sections of the document, drawing attention to the recommendations themselves. This section includes the guideline question(s) and a summary of the key evidence underpinning the recommendations. Section 2 Evidentiary Base: This section consists of a detailed summary and analysis of the available evidence upon which the recommendations are based. It includes a description of the methods employed to develop the evidentiary base, the results, a discussion, and a conclusion. As described in more detail below, the core methods used by the PEBC are systematic review, adaptation, formal consensus, and environmental scan. In some cases, there may be several sections to Section 2 (Section 2A, Section 2B, etc.) that provide different aspects of the available evidence or are a series of evidence updates over time Section 3 EBS Development Methods and External Review Process: This section describes not only the general methods used by the PEBC to create the document and develop its recommendations, but also the internal and external review processes that the document has undergone and the results of those reviews. A typical example of an EBS is EBS #2-29: Adjuvant Systemic Chemotherapy for Stage II and III Colon Cancer Following Complete Resection. The PEBC has found three main advantages to this modular document structure. First, it allows more flexibility in document development and maintenancenew sections can be easily added or altered without making major changes to the rest of the document. Second, it eases the process of publication in the peer-reviewed literature, as described below under Document Publication. Finally, it ensures that different components are available that will appeal to different stakeholders.

PEBC Development Cycle

The PEBC uses a core framework called the Guideline Development Cycle, based on the original work of Browman et al (1) and modified since that time. Figure 2 illustrates the key steps of the Development Cycle. Each stage of the development cycle will be detailed below, beginning with Project Planning.

Project Planning
The impetus to begin a new PEBC guideline project may come from an existing DSG/GDG, CCO Clinical Programs or Executive Team, the Ontario Ministry of Health and Long Term Care or other stakeholders in the Ontario cancer system. Some common criteria used to set priorities include the burden of disease, emergence of new care options, unwanted variation in clinical practice, opportunity to improve quality of care, safety or system performance, and new evidence. Every PEBC project begins with a planning

Page 2 of 15

process, involving the project leaders and a project Working Group, and with a formal project plan document as the outcome.

Putting together a new Working Group

Regardless of the origin of the project, before any work can be done a Working Group needs to be established. The working group is the small group of individuals that draft the guidance document. They define the topic to be addressed, determine the research questions, decide the scope of the report, provide input into the systematic review, and draft the guidance statements. This group consists of a lead author, one to four additional authors, and a PEBC Research Coordinator. With the exception of the PEBC Research Coordinator, all these individuals are usually volunteers. The level of familiarity with the methods of guideline development is often quite variable among the members of the Working Group, and so an important role of the PEBC Research Coordinator is to explain the process and guide the Working Group from a methodological standpoint.

Figure 2. The PEBC Guideline Development Cycle.

The process of determining the Working Group membership varies somewhat, depending on whether the project will be conducted within an existing DSG/GDG or not. Within an existing DSG/GDG, the Working Group members are indentified within the group, or are recruited by the group. Sometimes, depending on the topic, clinicians from outside the group will be recruited to provide specific subject matter expertise. These extra members may be recruited from outside Ontario, or even internationally; the goal is to create a group that has intimate and in-depth knowledge of the topic of the guideline project. For topics that will be conducted outside an existing GDG, a project Lead Author/Chair for the Working Group is usually established at the outset. This individual might be an appropriate CCO Clinical Program Head, or someone designated by the Clinical Program Head, or it might be the person who is most associated with bringing the topic forward for guideline development. Then, specific individuals are nominated by the LEAD AUTHOR and by other key partners (DSG Chairs, Clinical Program Heads, etc.), and these individuals are contacted in order to put together a balanced and complete Working Group. These individuals might be recognized as provincial leaders in their discipline or might have previously collaborated on PEBC projects. Ultimately, the core group of two to six clinicians/content experts and a PEBC Research Coordinator will be ready to begin the guideline development work. Conflict of interest (COI) statements are obtained from the members of the Working Group. The COI disclosures focus on financial involvement with industry or manufacturing parties related to the guideline topic, although the PEBC recognizes that many other conflicts might exist. At this time, the PEBC does not have formal policy regarding group participation if there are conflicts declared.

Page 3 of 15

Creating a Project Plan

Every PEBC project begins with a planning process, involving the project leaders and a project working group, with a formal project plan document as the outcome. The PEBC project plan is developed from a generic template that describes all the pertinent aspects of the project: Membership of the overall GDG that will review and take responsibility for the content of the guideline. Working Group members and their roles Topic and purpose of the project Scope of the project (including) o Type of recommendations expected (clinical practice recommendations, organizational recommendations, or both) o Research questions, including the populations, interventions, comparisons, and outcomes to be considered (PICO format). o Planned analyses (meta-analysis, etc.) o Domains that will be covered by organizational recommendations (practice team makeup, training or experience, institutional setting, regional or wider system level, reporting/evaluation/monitoring, etc.), if relevant Methods that will be used to develop the evidence base and the recommendations o Systematic Review Types of evidence to be considered Databases/Sources to be searched Inclusion/Exclusion Criteria o Adaptation (including known existing guidelines for possible adaptation) o Environmental Scan (including relevant organizations/sources to include in the scan) o Formal Consensus Participants in the consensus Methods to achieve consensus (i.e. Modified Delphi) Expected external review process for document Peer-reviewed publication strategy for document Timeline for completion of key development steps, and overall project completion.

The project plan is developed by the Working Group, and is reviewed by, at a minimum, an Assistant Director of the PEBC prior to project initiation. For projects that might be expected to require resourceintensive development or have potentially complex implications for implementation, the project plan would also be reviewed by additional stakeholders and managers (the PEBC Director, CCO Clinical Program Heads, CCO Executive Team, etc.) Although the project plan is completed prior to the initiation of the project, it is considered to be a living document that is subject to change throughout the execution of the project. However, alterations to the planned scope and/or methodologies are reviewed carefully by the PEBC as they are likely to have a large impact on the resources and time required for completion of the project. All changes made to the project plan document throughout the project are documented, dated, and approved by the group members.

Page 4 of 15

Document Creation
Once a project plan has been completed, the process of creating the document can begin. The document creation process will depend on the choices made during project planning, but the process can be generally divided into four stages: identification and review of existing guidelines, systematic review of the evidence, recommendations formulation, and draft completion.

Identification and Review of Existing Guidelines

As there are a number of national and international groups that develop high-quality guidelines on every conceivable topic, the first stage in the creation of PEBC documents is a systematic review of existing guidelines on that subject. A systematic search of the available electronic databases (Medline, EMBASE, National Guidelines Clearinghouse, CMA Infobase, etc.) is conducted. The websites of recognized guideline development groups, for example, NICE, SIGN, NCCN, and ASCO, are also reviewed for relevant guidelines. The intent of this search is to create a comprehensive list of all existing guidelines that are relevant to the project. All relevant guidelines are evaluated using the AGREE instrument (2), and the Working Group then considers the available guidelines and decides whether adaptation of one, or in rare instances more than one, of these guidelines would be sufficient to address the clinical and/or organizational questions outlined in the project plan. The Working Group looks at the currency and quality of the identified guidelines and the relevance of their recommendations to the populations of interest and the Ontario context when considering whether the identified guidelines might be endorsed (the recommendations should be used without modification) or adapted (the recommendations should be modified) for use in Ontario. If one or more existing guidelines are deemed worthy of adaptation, the Working Group proceeds, using the ADAPTE methods (3) where feasible and relevant. The Working Group may be aware during project planning of highly relevant existing guidelines that could be adapted. In fact, the existing guideline(s) might have been the impetus for the development of an Ontario guideline. Based on their knowledge of an existing guideline, the Working Group might choose not to conduct a complete search for other guidelines but instead, move forward with adapting the known guideline. However, the Working Group should have a strong justification for assuming that the guideline they have chosen is the one most worthy of adaptation. In some cases, existing guidelines may address some of the questions of interest but not address all questions completely. In such cases, the Working Group may choose to adapt existing guidelines for those questions and use other methods, as described below, to address the remaining questions. The process of adapting the existing recommendations can be cumbersome and time consuming, especially when those recommendations are out-of-date, inappropriate for the Ontario context, or involve large numbers of recommendations from multiple sources. In the experience of the PEBC, adaptation of recommendation(s) from existing guidelines is a pragmatic method to use when: There are a limited number of existing guidelines, and The existing guidelines address all, or nearly all, of the topics and questions the Working Group wants to address, and The evidence base used in those guidelines is fairly recent, usually not more than three years outof-date.

Page 5 of 15

When these criteria are not met, the PEBC has found that the most efficient way to move forward is to use the evidentiary base of the existing guideline(s) to formulate new recommendations, rather than using the existing recommendations, and to update the evidentiary base as necessary.

Systematic Review of the Evidence

The PEBC develops the evidentiary base for recommendations primarily through a systematic review and conducts such reviews using methods that are well understood and are described in other sources (4). Therefore, this handbook will describe only those areas where the PEBC differs from what other groups might consider customary or standard practice.

Literature Search
A search for all the available and relevant literature is conducted, based on the research question(s) and the study selection criteria set out in the project plan. The search covers, at minimum, peer-reviewed articles indexed in Medline and EMBASE, and a review of both the Central Registry of Clinical Trials and the Database of Systematic Reviews of the Evidence in the Cochrane Library. In addition, the websites of known developers of high-quality systematic reviews, whose work is not always published in the peer reviewed journals, are searched for existing reviews on the same topic. These sites include those of other guideline developer organizations such as the National Institute for Clinical Excellence (NICE) and the Scottish Intercollegiate Guideline Network (SIGN), as well as organizations that create health technology assessments and other systematic reviews for policy purposes, such as the Agency for Research on Healthcare Quality (AHRQ). In order to identify studies that might not have been published in peer-reviewed journals, the search most often covers abstracts and/or presentations of the results of randomized trials, made at appropriate, internationally recognized conferences such as the ASCO annual meeting, the ESMO meetings, etc. Although data presented in meeting abstracts or presentations may not be as reliable and complete as that from papers published in peer-reviewed journals, abstracts can be a source of important evidence from randomized trials and add to the evidence available from fully published studies. Those data often appear first in meeting abstracts and may not be published for several years. This search is usually conducted in two phases, although elements of the phases may be conducted simultaneously in order to save time. First, the existing systematic reviews/meta-analyses relevant to the research question(s) outlined in the project plan are identified. The objective is to make use of existing work to the greatest extent possible to develop the evidence base for the document efficiently. The findings are evaluated by the Working Group to determine if any of them provide a sufficiently comprehensive evidence base to answer one or more of the research questions. The Working Group takes into account the timeliness and quality of the systematic review and whether it summarizes the target populations, interventions, comparisons, and outcomes of interest as outlined in the project plan. After the Working Group has evaluated the existing systematic reviews and determined how to apply them, a literature search is conducted covering the research questions and those areas (time frame, populations, interventions, etc.) that the existing reviews do not address fully or at all. The search employs the standard methods of a literature search that have been described elsewhere (4). In conducting a search for primary study evidence, the intent of the PEBC is not to develop a complete summary of all the available evidence, regardless of its form. The intent is to identify, in a systematic fashion, that evidence which is relevant and useful for the formulation of recommendations. Metaanalyses and randomized trials are the primary evidence sought by the PEBC, but for some questions, other types of comparative studies (cohort studies, case-control studies, diagnostic/screening studies, and

Page 6 of 15

other experimental designs) can be useful in developing recommendations. In contrast, in the PEBCs experience, non-comparative study data (retrospective or prospective case series and phase I and II clinical trials), regardless of the number of studies and subjects, are rarely a sufficient basis for recommendations, while at the same time being more difficult to identify, collect, and summarize than that from comparative studies. Therefore, the assumption when beginning any project is that noncomparative studies will be excluded, even when the a priori belief is that there is limited comparative data available. In cases where the evidence is limited to non-comparative data, the PEBC has found that moving directly to formal consensus methods for the development of recommendations can be a more fruitful and pragmatic way to develop guidelines.

Collection, Assessment, Analysis, and Reporting of Evidence

Due to the limited availability of skilled researchers, it is usually not possible for the literature in a PEBC systematic review to be assessed by multiple reviewers. While there is no doubt that multiple, blinded review of the evidence is the recommended method in the literature (REFS), the PEBC is confident that the extensive internal and external review that each of our documents undergoes is sufficient to ensure that any bias that might exist due to the lack of multiple reviewers is addressed. While the PEBC considers the various assessment tools available in the literature (5) to be useful in outlining the features that should be looked for in good quality studies, we do not routinely use quality grading or rating systems to evaluate the quality of studies, particularly for excluding studies that do not meet a particular threshold. Rather, the important quality-related features of the studies (blinding, randomization methods, sample size calculations, intention to treat analysis, etc.) are reported for each study individually, and the overall quality of the evidence is evaluated in a more narrative fashion to present the reader with the information necessary for judging the quality of the included studies. The PEBC synthesizes the available evidence, when reasonable, using standard meta-analytic methods that have been described elsewhere (6).

Environmental Scan
In some cases, and especially when the topic under consideration deals with organizational questions, for instance, the methods of care delivery or the training requirements for clinicians, a systematic review will not identify important and relevant sources of information. The need then is to determine whether experiences in a different context or jurisdiction might be useful to consider or avoid in the Ontario context. These sources of information might include policy documents and guidance from other organizations or reports from government institutions or professional organizations that are found outside of the indexed medical literature. These sources are often not evidence-based, but they are useful for establishing a general consensus of worldwide opinion regarding topics that do not allow for evidencebased recommendations. In such cases, an environmental scan may be appropriate. The PEBC environmental scan process includes a targeted search of known or suspected relevant sources (other government, institutional, or professional organization websites, for example), and an untargeted search of the Internet using Google (2009) or another search engine, with relevant search terms, to identify these sources of information. The PEBC uses this information to address gaps in the existing evidence base and to provide a basis for initial recommendation formulation, which often leads to a formal consensus process, as described below.

Page 7 of 15

Recommendations Development
Initial Formulation of Recommendations
The Working Group formulates recommendations for practice or the delivery of care on the basis of the evidence synthesized in the process outlined above. Regardless of the quality and quantity of the evidence available to address all the research questions posed, the Working Group always needs to judge, interpret, and reach consensus on the meaning of this evidence for practice in Ontario. Expert opinion always plays a role in the development of recommendations, and frequently, the interpretation of the available evidence and the recommendations that the evidence can support is ambiguous--reasonable people may differ on the meaning and importance of the evidence. Even in the absence of any relevant evidence from the systematic review, the Working Group may still choose to make recommendations justified on other grounds such as logical argument, related evidence from other sources, and/or clinical experience. However, the justification for each recommendation and the degree to which it is based on the evidence directly versus the opinion and consensus of the Working Group must be explicitly stated in the recommendation itself.

Formal Consensus
Usually the recommendations are the result of informal consensus among the Working Group members. There are situations, however, where a formal process of ensuring consensus is necessary and useful. These situations include: When the available literature is very limited in terms of quantity and/or quality, but recommendations are still necessary. When it is desirable that a broader array of individuals, not just the Working Group, be involved in the recommendations development process, and not just in the review and approval of the recommendations. Formal consensus processes are used by a group of content experts to allow explicit reporting of the consensus opinion. The goal is to produce a set of recommendations based on the consensus of a wide array of appropriate experts in a transparent and systematic fashion, and to describe fully any controversial areas where consensus cannot be reached. As noted by Murphy et al (7): [Consensus development]is a process for making policy decisions, not a scientific method for creating new knowledge. At its best, consensus development merely makes the best use of available information, be that scientific data or the collective wisdom of the participants. Thus, although it may capture collective knowledge, it is inevitably vulnerable to the possibility of capturing collective ignorance. The decision to employ a formal consensus process may occur when: There is no evidence or when the evidence is very poor and it is unlikely that good quality studies will be conducted in the future (as in the case of a rare disease such as thymoma). As part of an Adaptation process: When recommendations are derived by a small working Group from an existing guideline, a formal consensus process with a larger group of clinical/content experts provides a method for refining the recommendations and allows for a transparent reporting of the level of consensus for each recommendation within the practice community.

Page 8 of 15

There are several methods for obtaining a formal consensus, and their use in clinical guideline development is outlined by Murphy et al (7). The Delphi technique is described by Linstone and Turoff (8). The PEBC has used a modified Delphi process with two rounds of feedback, as outlined in Figure 3. First, the Working Group members formulate draft recommendations on the basis of existing evidence and/or their own clinical experience and/or through the adaptation of recommendations from a relevant existing guideline. Then, in each round of feedback, the draft recommendations are submitted to the consensus body of experts, who are asked to rate their level of agreement with each recommendation using a Likert

Figure 3. Steps in a generic formal consensus process (a modified Delphi approach) used to develop recommendations for the PEBC guidelines.

scale, and to provide feedback on each recommendation. The Working Group makes an a priori decision on the interpretation of the responses from the consensus group regarding what constitutes a consensus agreement. For instance, if 75% or more of the consensus group agree or strongly agree with the recommendation then the recommendation is accepted. Defining thresholds for both consensus in support of a recommendation as well as consensus against a recommendation is important. After the first round, the responses are evaluated and summarized, to identify the recommendations with a high level of agreement and those for which there is no agreement or frank disagreement. Controversial recommendations are then rewritten or modified, based on the feedback, in an attempt to improve the level of agreement. In the second feedback round, the recommendations are again presented to the consensus body, along with the results of the previous round, in the form of a table or chart that illustrates or describes the level

Page 9 of 15

of agreement on each recommendation. For controversial recommendations, both the old and the newer, modified versions of the recommendation are presented. The consensus body is again asked to rate their level of agreement with all the recommendations, taking into account their knowledge of how their peers responded in the previous round. After two rounds, a final set of recommendations is drafted. Recommendations that complete the last stage with high levels of agreement are considered consensual. Draft recommendations where controversy still exists are presented in the final document as points for discussion, not as recommendations, with a description of the controversy surrounding them. The first example of PEBC guideline development using formal consensus methods is EBS 7-11 The Management of Thymoma, published online on the CCO website in September 2008.

Completion of Review Draft

After the recommendations are drafted and have undergone any necessary consensus process, a draft guideline for review is completed. As noted above, the justification for each recommendation is provided in the document (for example, direct evidence, evidence plus expert opinion, formal consensus, and so on).

Review: Internal and External

Internal Review
To begin the review process, a new draft of the document, incorporating the draft recommendations in Section 1 and the evidentiary base in Section 2, is prepared by the Working Group and is reviewed internally to ensure its quality. All documents are reviewed by the Report Approval Panel (RAP), whose membership includes the PEBC Director and Assistant Directors, an expert methodological consultant contracted by the PEBC for this purpose, and occasionally other methodological experts. The intent of this review is to ensure that the guideline development was methodologically rigourous and that the evidence-based recommendations are indeed supported by the evidence in a transparent way. Concurrent with the RAP review, the membership of the GDG responsible for the guideline reviews the draft, with the intent of ensuring the clinical relevance and utility of the guideline, the absence of obvious defects in the evidence base, and the reasonableness of recommendations derived through expert opinion. Feedback from the RAP and the GDG is considered, and responses to the review are prepared through discussion and informal consensus by the Working Group. The process of and details arising from the internal review are documented in Section 3 of the EBS, including any changes that were made to the guideline as a response.

External Review
Following the approval by the RAP and the GDG and the integration of the internal review process into Section 3, the new draft is prepared for external review. All but the following PEBC documents undergo complete external review: 1. Special advice reports submitted by the PEBC to the Committee to Evaluate Drugs (described below) are requested by and provide advice to that group for funding decisions and are therefore not submitted to external review. 2. If a formal consensus process has been undertaken to develop the recommendations, and the document has already been thoroughly reviewed by appropriate experts during that process, then the recommendations cannot be changed and still be considered the result of consensus. In this

Page 10 of 15

case, external review would be both redundant and in violation of the method used to develop the recommendations. The PEBC submits its documents to two forms of external review: targeted peer review and professional consultation.

Targeted Peer Review

The PEBCs Targeted Peer Review process is modeled on that used by peer-reviewed journals. The intent of this review is to ensure both the clinical and methodological quality and the relevance of the evidentiary base and recommendations. During project planning, a small number of individuals (3-10), who are not members of the GDG involved in the project, are identified as possible peer reviewers for the document. These potential reviewers are selected on the basis of their expertise, both clinical and methodological, and are invited early in the guideline development process to commit to review the final draft document. The process goal is to obtain comprehensive feedback and criticism from all the committed reviewers. When the draft document is ready for external review, the reviewers are provided with the draft document and a questionnaire to structure their feedback, which can be returned by mail, email, or the Internet. Reviewers are asked to comment in detail on all aspects of the document. The responses are summarized by the Research Coordinator, and presented to the Working Group members, who are to respond to each point, just as authors must respond to peer feedback during the journal publication process.

Professional Consultation
The PEBCs Professional Consultation process is intended to disseminate the draft guideline as widely as possible to its intended readership, to provide a forum for recipients to explain any disagreement with the conclusions and/or recommendations, and to further ensure the quality and relevance of the document. Initially, the PEBC identifies individuals in the PEBC internal contact database who are likely to be interested in the guideline, organizations whose membership might be interested in the subject matter, and other important relevant stakeholders suggested by the Working Group. These individuals are then contacted by email and mail and are invited to review the document and provide feedback either online, by mail, fax or by email. Each consultee is asked to provide general feedback about the document and answer a very short questionnaire (available upon request). The feedback is taken into consideration by the Working Group and used, as necessary, to correct and revise the document. The intent of the Professional Consultation process is to alert a large number of applicable individuals and groups to the existence of the guideline and to obtain general feedback and criticism from interested individuals. Achieving a specified proportion of responses is not the intent. The Working Group considers the critical review from the external review process, and responses to the review are prepared through discussion and informal consensus. The process of and details arising from the external review are documented in Section 3 of the EBS, as are any changes made to the rest of the document as a result of the feedback.

Document Publication
Once the results of the external review are incorporated, the document undergoes approval by the GDG. It then receives a final copy editing and is considered completed. The guideline can now be published and disseminated not only to its intended users but also to other interested parties.

Page 11 of 15

Web Publication
All PEBC documents are released publicly on the CCO website (www.cancercare.on.ca) to make the document freely available to Ontario health care providers and the general public, and to anyone else who wishes to access them. The PEBC also works with the National Guideline Clearinghouse (6) and the CMA Infobase (7) to ensure that our documents are properly indexed in those two databases so that individuals seeking guidance on topics that our documents address can locate them.

Journal Publication
In addition to web publication, the PEBC typically seeks to have each document, or certain portions, published in the peer-reviewed medical literature. In many cases, this involves producing two separate manuscripts for submission to relevant journals: an evidentiary-base manuscript and a practice guideline manuscript. In the experience of the PEBC, the evidentiary-base of our documents is often more easily published than the guideline recommendations, because it is relevant to a wider audience. The guideline recommendations, on the other hand, are written to be specifically relevant to the Ontario context. The basic structure of our documents, the three-part EBS, as described above, makes this submission process easier, because the document is already divided into a separate guideline (Section 1), evidence base (Section 2), and methods (Section 3).

Further Dissemination
The PEBC has found that the formal consensus and professional consultation processes described above are often sufficient to ensure that the intended Ontario users of our documents are made aware of them. In some cases, however, the PEBC document may be part of a larger initiative of CCO or the Ontario Ministry of Health and Long Term Care to affect practice in Ontario. In such cases, there may be additional dissemination efforts, including public meetings, news releases, implementation projects, and public policy development.

Document Maintenance
Once a document has been completed, it remains available on the PEBC website until it is updated or identified as being irrelevant to practice or incorrect. Ideally, the PEBC would choose to continually update the evidence base and recommendations of all of its documents, but the reality is that there are insufficient resources to do so. The PEBC has instead implemented a process of document maintenance that prioritizes those documents most in need of review and that makes the current status, relevance, and currency of the its documents clear to the users.

Periodic Review
The aim of the PEBC is that all our publicly available documents undergo a review within three years of their initial release, and then at least every two years thereafter. The goal of this review is to determine whether the document is still relevant to current practice or whether there are known or suspected problems with its current recommendations and to determine the priority of the document for update or rewrite. The PEBC has implemented the Document Assessment and Review Tool (DART) as a formal method of conducting these reviews. There are several outcomes for a document as a result of the DART review: Archiving: A document is archived when it is no longer considered relevant to current practice--it has become out-of-date and is unreliable to guide practice. The document is still available on the

Page 12 of 15

CCO website but has text and watermarking added to it to make clear that it should not be used for clinical decision making. Removal: A document is removed from public availability when there are known or suspected problems with its recommendations such that its implementation might cause harm or result in unnecessary care or treatment. The document is available only by contacting the PEBC. Endorsement: A document is endorsed when no update to the recommendations is considered necessary, and the recommendations are still considered valid and relevant for clinical practice. Endorsement can occur in the following three cases. o First, the document may be a definitive statement regarding the questions it addresses. That is, its recommendations are based on a large, consistent, and unambiguous evidence base, and there are no new questions that have arisen. Second, the recommendations may be derived primarily from consensus and expert opinion regarding a topic area where limited evidence is expected to be available such as for very rare diseases or conditions. Third, a new evidence search has been completed, and either no new evidence has been identified, or the evidence wholly supports the existing recommendations. The document is modified to indicate its endorsement and remains available on the website.

Deferral: A document may be deferred when there are insufficient resources to fully review the document, or when there are good reasons to postpone the review (for instance, a known relevant study will be published in the near future). The document remains available on the website, modified so that the reader can see the reasons for deferral. Prioritization for Update: The document remains available on the website, but is prioritized to be updated as soon as possible. In each case, the DART and any accompanying literature search results are incorporated into the document so that the user can see the documents current status and the reasons for that status.

Updating and Rewriting

As noted above, one of the possible outcomes of review with the DART is prioritization for update or rewrite.

Updating
A document is updated in the following circumstances: The clinical questions in the existing document are still relevant, and no new questions are required. There is believed or known to be a limited number of new sources of evidence regarding those questions. There is a low likelihood that major revisions and additions to the recommendations will be necessary. In an update, the original evidence base of the document (Section 2 of the EBS) is left unchanged, becoming Section 2B or similar. An updated evidence base is created, using the methods described above

Page 13 of 15

and incorporating all evidence that has been released since the completion of the original document. Only the guidance portion of the document (Section 1 of the EBS) is rewritten. Additional information is added to the methods and review section (Section 3 of the EBS) to reflect and describe the changes that were made. The updated document then undergoes a new round of internal and external review, as described above, although in some cases the changes made to the recommendations and the volume of new evidence may be considered small enough to not warrant review.

Rewriting
If any of the criteria for updating mentioned above are not met, the document may require rewriting. In this case, the document will undergo the entire PEBC development cycle again, including the creation of a new project plan. The existing document will be used to whatever extent possible to speed the process of development. However, a rewrite is essentially a new project, as the research questions, the scope of the evidence search, and the scope of the recommendations may all change radically from the earlier version.

Implementing the PEBC Development Cycle in other contexts

The PEBC is committed to furthering the science and practice of evidence-based guidance development both in Canada and around the world. The PEBC has a number of other tools, templates, examples, and protocols that are used by PEBC staff for training and to conduct the processes described above. While these materials are customized for our use, they may be useful to other organizations seeking to develop evidence-based guidance or implement an evidence-based guidance program, or for other purposes. We are happy to share these materials with interested parties. Please contact the PEBC, by phone at 1-905527-4233 ext. 42822 or by email at ccopgi.mcmaster.ca, to obtain these materials, or to ask any questions you may have regarding this handbook, the PEBC methods, or any of our documents.

Page 14 of 15

References
1. 2. 3. 4. 5. 6. 7. 8. Browman G, Levine M, Mohide E, Hayward R, Prtichard K, Gafni A, et al. The practice guidelines development cycle: a conceptual tool for practice guidelines development and implementation. J Clin Oncol.1995;13:502-12. The AGREE Research Trust. The Appraisal of Guidelines for Research & Evaluation (AGREE) instrument. London: The AGREE Research Trust; 2001 [cited 2009 Jul 29]. Available from: http://www.agreetrust.org/instrument.htm. The ADAPTE Collaboration. The ADAPTE Manual and Resource Toolkit. The ADAPTE Collaboration; 2007 [cited 2009 Jul 29]. Available from: http://www.adapte.org/rubrique/manual-and-tools.php. Scottish Intercollegiate Guidelines Network. SIGN 50: A guideline developer's handbook. Edinburgh: Scottish Intercollegiate Guidelines Network; 2008 [cited 2009 Jul 29]. Available from: http://www.sign.ac.uk/guidelines/fulltext/50/index.html. Moher D, Jadad A, Tugwell P. Assessing the quality of randomized controlled trials. Int J Technol Assess Health Care. 1996;12(2):195-208. Lipsey M, Wilson D. Practical meta-analysis. Bickman L, Rog D, editors. Thousand Oaks (CA): Sage Publications; 2001. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CFB, Askham J, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assess. 1998;2(3). Linstone HA, Turnoff M, editors. The Delphi method: techniques and applications. Don Mills (ON): Addison-Wesley Publishing; 2002. Available from: http://is.njit.edu/pubs/delphibook/delphibook.pdf.

Page 15 of 15