Journal of Clinical Epidemiology 129 (2021) 1e11

ORIGINAL ARTICLE

Developing trustworthy recommendations as part of an urgent response

(1e2 weeks): a GRADE concept paper
Elie A. Akla,*, Rebecca L. Morgana, Andrew A. Rooneyb, Brandiese Beverlyb,
Srinivasa Vittal Katikireddic, Arnav Agarwald, Brian S. Alpere, Carlos Alva-Diazf,
Laura Amatog, Mohammed T. Ansarih, Jan Brozeka, Derek K. Chui,j,k, Philipp Dahml,
Andrea J. Darzia, Maicon Falavignam, Gerald Gartlehnern, Hector Pardo-Hernandezo,
Valerie Kingp, Jitka Klugarov
aq, M.W. Miranda Langendamr, Craig Lockwoods,
Manoj Mamment, Alexander G. Mathioudakisu,v, Michael McCaulw, Joerg J. Meerpohlx,y,
Silvia Minozziz, Reem A. Mustafaaa,ab, Francesco Noninoac, Thomas Piggottad, Amir Qaseemae,
John Rivaaf, Rachel Rodinag, Nigar Sekerciogluah, Nicole Skoetzai, Gregory Traversyaj,
Kris Thayerak, Holger Sch€
unemanna
a
Department of Health Research Methods, Evidence, and Impact, McMaster University, McMaster University Medical Centre, 1280 Main Street West,
Hamilton, Ontario L8S 4K1, Canada
b
National Institute of Environmental Health Science, Durham, NC, USA
c
MRC/CSO Social & Public Health Sciences Unit, University of Glasgow, 200 Renfield Street, Glasgow G2 3AX, UK
d
Department of Medicine, University of Toronto, C. David Naylor Building, 6 Queen’s Park Crescent West, Third Floor, Toronto, ON M5S 3H2, Canada
e
Department of Family and Community Medicine, Medical Knowledge Office, EBSCO Information Services, University of Missouri-Columbia School of
Medicine, 10 Estes Street, Ipswich, MA 01938, USA
f
Universidad Cientifica del Sur, Facultad de Ciencias de la Salud, Panamericana, Sur km 19, Villa, Lima, Peru
g
Department of Epidemiology, Lazio Regional Health Services Rome, via Cristoforo Colombo 112, 00147 Rome, Italy
h
School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Room 101, 600 Peter Morand Crescent, Ottawa, Ontario K1G 5Z3,
Canada
i
Department of Health Research Methods, Evidence, and Impact, McMaster University, Rm 3V49 McMaster University Medical Centre, Hamilton, ON L8S
4K1, Canada
j
Department of Medicine, Rm 3V49 McMaster University Medical Centre, Hamilton, ON L8S 4K1, Canada
k
The Research Institute of St. Joe’s Hamilton, Rm 3V49 McMaster University Medical Centre, Hamilton, ON L8S 4K1, Canada
l
Department of Urology, Minneapolis VA Healthcare System, Urology Section, University of Minnesota, One Veterans Drive, Minneapolis, MN 55417, USA
m
Hospital Moinhos de Vento, Porto Alegre, Brazil
n
Department for Evidence-based Medicine and Evaluation, RTI International, Danube University, Dr. Karl Dorrekstrasse, Krems Austria, 3040 East
Cornwallis Rd, Research Triangle Park, NC, USA
o
Iberoamerican Cochrane Centre, Sant Pau Biomedical Research Institute (IIB Sant Pau), CIBER de Epidemiolog
ıa y Salud P
ublica (CIBERESP), Carrer
Sant Antoni Maria Claret, 167, Pavell
o 18, Planta Baja (Ground Floor), 08025 Barcelona, Spain
p
Center for Evidence-based Policy, Oregon Health & Science University, 3030 South Moody Ave., Suite 250, Portland, OR 97201, USA

Funding: None.
Contributions: Elie Akl, Rebecca L. Morgan, Andrew Rooney, Kris Thayer, and Holger Sch€unemann conceived and designed the study. All authors
contributed to the discussion of the concepts included in the paper. Elie Akl drafted the first version of the paper with initial input from Rebecca L. Morgan,
Andrew Rooney, Brandiese Beverly, Srinivasa V. Katikireddi, Kris Thayer, and Holger Sch€unemann. All authors reviewed and approved the final version of
the paper.
Conflicts of interest: None of the authors reported a relevant conflict of interest.
Author statement: The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all authors, a worldwide license to
the Publishers and its licensees in perpetuity, in all forms, formats, and media (whether known now or created in the future), to (1) publish, reproduce,
distribute, display, and store the Contribution; (2) translate the Contribution into other languages, create adaptations, reprints, include within collections
and create summaries, extracts and/or, abstracts of the Contribution and convert or allow conversion into any format including without limitation audio;
(3) create any other derivative work(s) based in whole or part on the Contribution; (4) to exploit all subsidiary rights to exploit all subsidiary rights that
currently exist or as may exist in the future in the Contribution; (5) the inclusion of electronic links from the Contribution to third party material
where-ever it may be located; and, (6) license any third party to do any or all of the above.’’
* Corresponding author. Department of Internal Medicine, American University of Beirut Medical Center, P.O. Box 11-0236/CRI (E15), Riad-El-Solh
Beirut 1107 2020, Beirut, Lebanon. Tel.: 961 1 374374..
E-mail address: ea32@aub.edu.lb (E.A. Akl).

https://doi.org/10.1016/j.jclinepi.2020.09.037
0895-4356/Ó 2020 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/
licenses/by-nc-nd/4.0/).
2 E.A. Akl et al. / Journal of Clinical Epidemiology 129 (2021) 1e11
q
Czech National Centre for Evidence-Based Healthcare and Knowledge Translation, Institute of Biostatistics and Analyses, Faculty of Medicine, Masaryk
University, Brno, Czech Republic
r
Amsterdam UMC, University of Amsterdam, Epidemiology and Data Science, Amsterdam Institute of Public Health, Meibergdreef 9, Amsterdam, the
Netherlands
s
JBI, School of Public Health, University of Adelaide, Level 3/55 King William Road, North Adelaide 5005, Australia
t
Department of Medicine, State University of New York at Buffalo, 100 High Street, Buffalo, NY, USA
u
Division of Infection, Immunity and Respiratory Medicine, School of Biological Sciences, The University of Manchester, Manchester Academic Health
Science Centre, 2nd Floor, Education and Research Centre, Wythenshawe Hospital, Manchester University NHS Foundation Trust, Southmoor Road,
Manchester M23 9LT, UK
v
The North West Lung Centre, Wythenshawe Hospital, Manchester University NHS Foundation Trust, 2nd Floor, Education and Research Centre,
Wythenshawe Hospital, Manchester University NHS Foundation Trust, Southmoor Road, Manchester M23 9LT, UK
w
Division of Epidemiology and Biostatistics, Department of Global Health, Stellenbosch University, Teaching Building, Faculty of Medicine and Health
Sciences, Francie van Zijl Drive, Bellville, Western Cape, South Africa
x
Institute for Evidence in Medicine, Medical Center & Faculty of Medicine, University of Freiburg, Breisacher Strasse 153, 79110 Freiburg, Germany
y
Cochrane Germany, Cochrane Germany Foundation, Breisacher Strasse 153, 79110 Freiburg, Germany
z
Department of Biomedical Sciences for Health, University of Milan, Milan, Italy
aa
Department of Internal Medicine, University of Kansas Medical Center, 3901 Rainbow Blvd, MS3002, Kansas City, KS 66160, USA
ab
Department of Health Research Methods, Evidence, and Impact, McMaster University, 3901 Rainbow Blvd, MS3002, Kansas City, KS 66160, USA
ac
IRCCS Istituto delle Scienze Neurologiche di Bologna, UOSI Epidemiologia e Statistica, IRCCS-ISNB, Ospedale Bellaria, Via Altura, 3, 40139 Bologna,
Italy
ad
Department of Health Research Methods, Evidence, and Impact, McMaster University, McMaster University Medical Centre, 1280 Main Street West 2C
Area, Hamilton, Ontario L8S 4K1, Canada
ae
American College of Physicians, 190 N Independence Mall West, Philadelphia, PA 19087, USA
af
Department of Family Medicine, Department of Health Research Methods, Evidence, and Impact, McMaster University, McMaster University Medical
Centre, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada
ag
Public Health Agency of Canada, 785 Carling Avenue, Ottawa, Ontario, Canada
ah
Department of Health Research Methods, Evidence, and Impact, McMaster University, McMaster University Medical Centre, 1280 Main Street West,
Hamilton, Ontario L8S 4K1, Canada
ai
Department I of Internal Medicine and Center for Integrated Oncology Aachen Bonn Cologne Duesseldorf, Evidence-based Oncology, Faculty of Medicine
and University Hospital Cologne, University of Cologne, Kerpener Str. 15, 50935 Cologne, Germany
aj
Public Health Agency of Canada, Ottawa, Ontario, Canada
ak
U.S. Environmental Protection Agency, Office of Research and Development (ORD, Center for Public Health and Environmental Assessment (CPHEA),
Mail Code: B243-01, Building: Bldg B (Room B210I), Research Triangle Park, NC 27711, USA
Accepted 1 September 2020; Published online 30 September 2020

Abstract
Objectives: The aim of this study is to propose an approach for developing trustworthy recommendations as part of urgent responses
(1e2 week) in the clinical, public health, and health systems fields.
Study Design and Setting: We conducted a review of the literature, outlined a draft approach, refined the concept through iterative
discussions, a workshop by the Grading of Recommendations Assessment, Development and Evaluation Rapid Guidelines project group,
and obtained feedback from the larger Grading of Recommendations Assessment, Development and Evaluation working group.
Results: A request for developing recommendations within 2 week is the usual trigger for an urgent response. Although the approach
builds on the general principles of trustworthy guideline development, we highlight the following steps: (1) assess the level of urgency; (2)
assess feasibility; (3) set up the organizational logistics; (4) specify the question(s); (5) collect the information needed; (6) assess the ad-
equacy of identified information; (7) develop the recommendations using one of the 4 potential approaches: adopt existing recommenda-
tions, adapt existing recommendations, develop new recommendations using existing adequate systematic review, or develop new
recommendations using expert panel input; and (8) consider an updating plan.
Conclusion: An urgent response for developing recommendations requires building a cohesive, skilled, and highly motivated multidis-
ciplinary team with the necessary clinical, scientific, and methodological expertise; adapting to shifting needs; complying with the princi-
ples of transparency; and properly managing conflicts of interest. Ó 2020 The Author(s). Published by Elsevier Inc. This is an open access
article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Keywords: GRADE; Urgent recommendation; Pandemic; Trustworthy guideline

1. Background scoping the remit of the guideline, identifying an expert

panel group, and managing conflicts of interest [1,2]. The
The development of practice guideline recommenda-
most time-consuming tasks are the identification, synthesis,
tions requires a complex interplay of many participants
assessment, and presentation of the evidence in ways that
and involves various tasks including setting priorities,
allow the guideline groups to formulate recommendations.
 E.A. Akl et al. / Journal of Clinical Epidemiology 129 (2021) 1e11 3

2. Methods
What is new?
The target audience for the proposed approach are
guideline developers. We consider the timeframe from the
Key findings
perspective of the guideline developer, that is, the starting
 We propose an approach for developing trust-
point for the approach is the receipt of a request for devel-
worthy recommendations as part of an urgent
oping recommendations (if applicable), while the finishing
response (1e2 weeks).
point is the submission of guidance to the requesting orga-
nization. We use the WHO definition of guidelines as ‘‘sys-
What this adds to what was known?
tematically developed evidence-based statements which
 The proposed alternatives are: adopting or adapting
assist providers, recipients and other stakeholders to make
existing recommendations; using existent system-
informed decisions about appropriate health interventions’’
atic reviews to develop new recommendations;
[10].
and, relying on expert panel input to develop new
We developed this approach as members of the GRADE
recommendations.
Rapid Guidelines project group. The GRADE Rapid Guide-
lines project group includes 31 members of the GRADE
What is the implication and what should change
working group (gradeworkinggroup.org). These members
now?
collectively have expertise in preclinical, clinical, and
 It is critical to leverage collaborations, capabilities,
epidemiological primary research, in modeling, in evidence
and resources that allow for rapid assessment of
synthesis (including systematic reviews and rapid reviews),
adequate information to support recommendation
and in trustworthy practice guideline development. This
development.
expertise spans the breadth fields of clinical, public health,
and health policy. The development of the approach started
in May 2016, and was based on reviewing selected relevant
research (some of which was conducted by members of the
Over the last 2 decades, the Grading of Recommendations
GRADE Rapid Guidelines project group) [6,11e13], itera-
Assessment, Development and Evaluation (GRADE) work-
tive discussions within the project group reflecting on the
ing group has developed a systematic and transparent
members’ experiences, informal consensus process, and
approach to creating evidence-informed practice guidelines
feedback from the larger GRADE working group.
that facilitate this. The method has been applied across the
In terms of the review of selected relevant research, we
clinical, public health, and health systems fields [3e5].
first considered the findings of a 2018 systematic survey of
Some situations raise the need for urgent recommenda-
methods manuals and published guidelines that were devel-
tions to support the interventions of clinicians, public
oped in a shortened timeframe [12]. The review found that
health practitioners, and policymakers, for example, to
rapid guidelines were usually commissioned to address
address the COVID-19 pandemic [6,7]. Under such circum-
emergencies, rapid increases in the incidence or severity
stances, developing trustworthy recommendations in a suf-
of a condition emerging new evidence for a specific treat-
ficiently short timeframe is essential but can be challenging
ment. Overall, there was no consensus on the methods for
in rapidly changing contexts. Groups producing guidelines
developing rapid guidance or on the timeframe for develop-
specifically need to balance the need for developing a
ment, ranging from 1 to 13 months, when reported. Second,
timely response with the need to ensure the trustworthiness
we considered the findings of an interview of guideline de-
of their advice.
velopers about key aspects for rapid guideline development
The World Health Organization (WHO) defined two
[11]. Third, we discussed published and unpublished exam-
types of guidelines developed in response to an emergency
ples of recommendations developed in response to rapid or
or urgent need: emergency (rapid response) guidelines (pro-
urgent needs, with a focus on understanding how standard
duced within hours to days) and rapid advice guidelines [8].
processes were modified [7,13e16].
Thayer and Schunemann [6] defined four levels of urgency
for developing recommendations: ultra-short emergency
response (1e2 hours), urgent response (1e2 weeks), rapid
response (1e3 months), and routine response (more than
Box 1 Levels of urgency for developing
3 months) (Box 1). Although more detailed advice exists
recommendations
for routine [1] and rapid responses, so far there is no formal
guidance on how to apply GRADE in situations requiring Ultra-short emergency response: 1e2 h
urgent responses [6,9]. Urgent response: 1e2 wk
The objective of this paper is to propose an approach for Rapid response: up to 3 mo
developing trustworthy recommendations as part of urgent Routine response: more than 3 mo
responses (1e2 weeks) in the clinical, public health, and
health systems fields.
4 E.A. Akl et al. / Journal of Clinical Epidemiology 129 (2021) 1e11

We debated, revised, and refined the concepts to be The following paragraphs discuss the steps of the urgent
included in the approach based on iterative discussions dur- response approach and assume that existing and adequate
ing conference calls and a workshop by the project group guidelines and systematic reviews, but not new rapid re-
(Rome, Italy, April 27, 2017). Members of the project views, are being considered. However, if a rapid review is
group also drew on their experiences with developing rec- performed it may be added at step 5 in this process. We pro-
ommendations in response to rapid or urgent needs vide illustrative examples of urgent situations related to
[17e19]. Finally, the GRADE working group approved this environmental exposures and to the COVID-19 pandemic.
paper using its expedited publication approval process,
developed in light of the COVID-19 pandemic, and the 3.1. Assess the level of urgency
need for development of trustworthy recommendations
for its management internationally as part of urgent The purpose of this first step is to confirm that the
responses. response in developing the recommendation should be an
‘‘urgent’’ one, as opposed to ‘‘emergent’’ or ‘‘rapid’’ (see
Box 1). Although there is no tool specifically developed
for that purpose, one can use tools such as the WHO rapid
risk assessment of acute public health events [24]. Risk
3. Results assessment includes the assessment of the likelihood of
The project group agreed to build the urgent response the occurrence of a hazard causing an adverse event, as well
approach on the commonly accepted principles of standard as the likely magnitude of the consequences of the event
guideline development (e.g., conflict of interest manage- over a specified period.
ment) [1,2]. The group also agreed that the urgent response
approach can also build on the GIN-McMaster guideline 3.2. Assess feasibility
development checklist extension for rapid recommenda- It is important for the guideline developers to assess the
tions (see table here: https://bit.ly/rapidGDC) [11]. Ideally, feasibility of meeting the ‘‘urgent’’ timeframe (i.e.,
the process should include the conduct of a rapid review, 1e2 weeks). Aspects to be considered include the
whenever feasible. However, and given the major chal- following: (1) the number and complexity of questions
lenges with feasibility, in the subsequent sections we did and comparisons; (2) the expected amount of work needed
not consider a rapid review as an option. for collecting the literature; (3) the number of team mem-
The proposed approach assumes adherence to the bers available to contribute to the different tasks; (4) sour-
criteria for applying or using GRADE [20]. These criteria ces of funding that can be promptly tapped to support the
include giving explicit consideration to the GRADE factors urgent response; (5) the availability of adequate coordina-
for determining the direction and strength of a recommen- tion capacity; (6) the panelists who can be summoned to
dation and, ideally, using GRADE Evidence to Decision contribute to collecting the needed information, panel dis-
(EtD) frameworks and tables [21]. Under standard guide- cussion, and peer review in a very intensive manner; and
line development conditions, the EtD table is populated (7) the political and institutional support to expedite the
by evidence (typically from a systematic review) for a num- administrative processdcollaboratively with the relevant
ber of factors (the health effects of the interventions of in- endorsing bodies (particularly the approval of the final
terest, values and preferences, certainty of evidence, product and if needed, of the dissemination and implemen-
resource use, impact on equity, acceptability, and feasi- tation strategies), if applicable.
bility). The panelists review the evidence for each factor,
then make judgments for those factors in relation to the in-
3.3. Set up the organizational logistics
terventions of interest, before deciding on the strength and
direction for the recommendation [21]. As the guideline developers start working with the re-
In terms of sources of evidence, systematic reviews may questors on specifying the questions (see next section), it
not be feasible given the timeframe of 1e2 weeks for the is important to set up the organizational logistics for the
complete recommendation development process. If, in spite project. Forming a steering group with executive power that
of this tight timeframe, the guideline developers find it would meet frequently (e.g., at least on a daily basis) to
feasible to conduct a rapid review, the reviewers could refer ensure the progress of the project and the compliance with
to the Interim Guidance from the Cochrane Rapid Reviews the timeline is key. Similarly, the steering group should
Methods Group [22]. In addition, the guideline developers involve methodologists with prior experience in developing
may consider a timely publication of the systematic review, recommendations as part of urgent or rapid responses. Also,
including coordination with the coordinating organization/ in the context of international collaboration, having mem-
funder, reviewers, and journal [23]. The proposed approach bers from different time zones can help ensure the ability
offers the alternatives listed at the bottom of Fig. 1, which to sustain project workflow.
are based on the use of existing guidelines and existent sys- Starting on day 1, the steering group should identify and
tematic reviews (discussed in more detail below). quickly recruit members of the different groups (including
 E.A. Akl et al. / Journal of Clinical Epidemiology 129 (2021) 1e11 5

Fig. 1. Steps involved in urgent guideline development.

methodologists, panelists, reviewers) and secure their solid Intervention, Comparator, Outcome) or PECO (Population,
commitment to the expected tasks with a clear timeline and Exposure, Comparator, Outcome) frameworks for interven-
schedule. However, it might be challenging to recruit stake- tions and exposures respectively [26,27]. Even within the
holders (particularly patients’ representatives) in the PICO or PECO frameworks, additional specificity (narrow-
context of a rapid response (e.g., in the middle of a ing the scope) when defining the population, intervention,
pandemic). Building on existing networks of stakeholders or exposure may facilitate the urgent development process.
or having a roster of stakeholders willing to serve as pan- Annex 1 provides a brief description and examples of the
elists when needed can be helpful. The aim should be elements of the PICO and PECO questions.
meaningful and equitable multi-stakeholder engagement Prioritizing outcomes is an essential step of the guide-
[25]. line development process. However, it can be time
Similarly, when applicable, the steering group needs to consuming and not feasible under urgent conditions.
prospectively liaise closely with the organization’s over- Ideally, there would be a ‘‘ready to use’’ standardized set
sight and quality committee to ensure compliance with of outcomes for the condition under consideration (e.g.,
quality requirements. The steering group should start as COVID-19 Core Outcomes initiative) [28]. When such a
early as possible drafting the final report and fill in its set is not available, the guideline developers could rely
different sections as the information becomes available, to on previous prioritization efforts. For example, the Surviv-
expedite its submission and later publication. Other impor- ing Sepsis Campaign (SSC) guidelines on the management
tant logistics include a streamlined communication plan of critically ill adults with COVID-19, instead of perform-
with the different groups involved, with the goal of mini- ing a new prioritization of outcomes, used the outcome pri-
mizing email burden; and reliable and easy to use document oritization informed by the ongoing SSC guideline 2020
sharing and online meeting platforms. Ideally, the organiza- work and expert panel input [29]. The group could also rely
tion should have in place a ‘‘preparedness plan’’ to mount on outcomes prioritized for a similar disease (e.g., influenza
an urgent response for developing recommendations (with or severe acute respiratory syndrome [SARS]). In some ur-
the appropriate processes and tools). gent situations, the group might opt to prioritize only one
critical outcome. For example, slowing the spread of the
pandemic (‘‘flattening the curve’’) was initially considered
3.4. Specify the question(s)
as the sole critical outcome when considering public health
To make the process feasible, the guideline developers interventions for the COVID-19 pandemic.
should work with the requestor to ensure the guideline’s Urgent situations are typically associated with scarcity
scope is reasonably narrow. In addition, the guideline de- of data, particularly for clinical outcomes (e.g., clinical re-
velopers should ensure the questions are specific and well covery). In such cases, the guideline developers might need
defined, for example, using the PICO (Population, to rely on surrogate outcomes (e.g., viral load) for
6 E.A. Akl et al. / Journal of Clinical Epidemiology 129 (2021) 1e11

informing recommendations. Indirect evidence can be used may be important to evaluate or model the evidence for
to inform other components of the PICO or PECO question. health effects when little is known about the exposure
For example, the National Institute for Health and Care (e.g., a chemical with uncertain effects).
Excellence (NICE)’s ‘‘interim process and methods for
developing rapid guidelines on COVID-19’’ calls for using
3.6. Assess the adequacy of identified information
information related to SARS, Middle East respiratory syn-
drome (MERS), and pandemic influenza [30]. Once identified, the published literature (guidelines, sys-
It is important to determine a priori what indirect evi- tematic reviews, and if applicable, primary studies of any
dence would be considered and used, if needed. Similar design) needs to be assessed for ‘‘adequacy’’ based on
key decisions in the design and conduct of the project the three following criteria: relevance, credibility, and cur-
(e.g., eligibility criteria) should ideally be done a priori. rency [32].
However, and given the limited time to fully consider the
 Relevance or directness, according to GRADE termi-
different aspects of the project, post hoc decisions are more
nology [33], refers to the extent to which the popula-
likely in urgent compared with routine response when
tion, exposure/intervention, and outcomes identified
developing guidelines.
in the literature (whether for the original recommen-
dation or for the systematic reviews) reflect the ones
3.5. Collect the information needed in the question being addressed by the recommenda-
tion. If any items are different enough that we would
As mentioned earlier, the proposed approach considers
expect different relative effects, then the evidence is
using existing guidelines and systematic reviews and does
considered indirect.
not necessarily require the conduct of new rapid reviews.
 Credibility reflects the quality of conduct of the
NICE has adopted such an approach in its ‘‘interim process
guideline or of the systematic review being consid-
and methods for developing rapid guidelines on COVID-
ered for use in the process of the urgent response.
19’’ [29]. They aimed to reuse existing guidance as much
There are a number of tools that could be used for
as possible, without conducting systematic literature
that purpose: the advancing guideline
searches. They first searched for published guidance,
development, reporting and evaluation in health care
including WHO COVID-19 guidance, NICE guidance,
II to assess the quality of conduct of guidelines
and guidance from professional organizations, both inside
[34]; the assessment of multiple systematic reviews-
and outside the UK [29]. If no guidance was identified, they
2 to assess the methodological quality of systematic
searched for and prioritized the following types of publica-
reviews [35], and ROBIS to assess the risk of bias
tions, in the following order: systematic reviews, random-
of systematic reviews [36].
ized controlled trials, and published expert opinions.
 Currency or recency: although the currency of a sys-
Many organizations use a similar approach. For example,
tematic review depends on its search date, the cur-
in 2008, the European Food Safety Authority relied on a
rency of a guideline depends similarly to a large
previously published systematic review to develop its ur-
extent on the search dates of the systematic reviews
gent guidance related to the presence of melamine in com-
used to inform recommendations. In some cases, even
posite food products [31].
if the search date is not current, a systematic review
In addition to information about the health effects, the
may still be up to date if no new evidence was pub-
guideline developers may need to identify contextual infor-
lished after the date of the search; however, a search
mation. In the example of mass evacuation in response to a
would need to be conducted to verify this assumption.
nuclear incident, there is a need to collect information
Rapidly updating a published systematic review may
about the cost and feasibility of such a major undertaking,
be efficient, if that review is credible and new evi-
as well as the acceptability by relevant stakeholders (e.g.,
dence is known to have been published since the date
governmental agencies, citizens). This information may
of the original search [37].
be available from administrative databases (e.g., for cost),
and through consulting with community leaders (e.g., to The guideline developers do not need to conduct the
assess the acceptability by stakeholders). above assessment for all identified relevant literature
In terms of information sources, the published literature (guidelines, systematic reviews, and primary studies). For
may not be the only optimal source in urgent and rapidly example, if they end up adopting an existing recommenda-
developing situations, like the COVID-19 pandemic. Infor- tion (addressed in step 6), although the recommendation
mation sources may include preprint servers, open research under consideration needs to be assessed, this would not
databases, and research data repositories. However, such apply to all other identified literature (e.g., systematic re-
sources may not include conventional processes of peer re- views and primary studies).
view and may require additional critical appraisal or The adequacy of identified literature is not an ‘‘all or
caution in the interpretation of the accuracy or stability of none’’ assessment. For example, the guideline developer
the data. Specific to environmental exposure scenarios, it might identify three potentially relevant systematic reviews.
 E.A. Akl et al. / Journal of Clinical Epidemiology 129 (2021) 1e11 7

Each one of the three may be deficient in relation to one of the direction or the strength of recommendation may be
the three above criteria, but to different extents. We suggest affected by any difference in the following factors (ideally
considering the criteria in the following order: relevance, using a decision-making framework, e.g., EtD table) [21]:
credibility, and currency. This means that the guideline de-
 Rating of the importance of outcomes: a change in the
velopers would select a relevant but outdated systematic re-
rating of the importance of outcomes might lead to a
view over an updated, but less relevant review. The idea is
change in the judgment of the balance of benefits and
that it is more efficient to ‘‘fix’’ the former review with a
harms. For example, an urgent response recommen-
quick update than to use the latter less relevant review.
dation addressing the use of invasive ventilation in
During the 2014 Elk River chemical spill, the most rele-
patients with COVID-19 might reflect a different pri-
vant information on the toxicology of the crude 4-
oritization of the importance of ‘‘patient survival’’
methylcyclohexanemethanol (MCHM) was obtained
outcome relative to ‘‘transmission of infection to
directly from the manufacturer [38]. Within 6 days, the Sci-
healthcare workers’’ outcome (in comparison to the
entific Review Panel for the National Library of Medicine’s
original recommendation) [41].
Hazardous Substance Data Bank was able to expedite the
 Indirectness: what could have been judged as direct
evaluation of the toxicology, chemical composition, and
evidence in the context of the original recommenda-
chemical purpose to assess the credibility of the informa-
tion might be judged as indirect evidence in the ur-
tion [38]. This early evaluation was a critical step in the
gent response context (e.g., effectiveness of
process of understanding the risks of exposure and the best
treatment with corticosteroids among patients with
course of action for cleanup. However, additional research
MERS-CoV vs. COVID-19) population. This might
was needed to better understand the short-term adverse
lead to rating down the certainty of evidence. One
health effects associated with MCHM, and no additional in-
could use the GRADE framework to judge whether
formation was available at the time. To expand the knowl-
that situation warrants rating down the certainty of
edge base for MCHM and other chemicals associated with
evidence for indirectness [33].
the spill with the most current data, the Centers for Disease
 Rating of evidence: the expert panel group may rate
Control and Prevention and the Agency for Toxic Sub-
the certainty of evidence differently than the original
stances Disease Registry enlisted the help of the National
recommendation for reasons other than indirectness
Toxicology Program to conduct a series of short duration
judgment (see above). Examples include different
studies [39]. The results obtained from these studies assis-
judgment of the risk of bias or imprecision, and
ted the West Virginia Bureau of Public Health in respond-
consideration of new or emerging evidence.
ing to public health concerns related to the MCHM and
 Baseline risks: the absolute reduction of an outcome
other chemical exposures that occurred during the incident.
(e.g., mortality in laboratorians or other staff with a
potential occupational risk for Ebola receiving a
3.7. Develop the recommendations pre-exposure Ebola vaccination) might be higher or
Depending on the availability and adequacy of identified lower in the context of the urgent response (e.g.,
literature (guidelines or systematic reviews), the guideline Ebola outbreak responders) compared to context of
developers may develop recommendations using one of the original recommendation (e.g., due to a difference
the four potential approaches: (1) adopt existing recom- in baseline risk of mortality between the two con-
mendations; (2) adapt existing recommendations; (3) texts). This could lead to a change in balance of ben-
develop new recommendations using existing systematic efits and harms.
and/or rapid reviews [15]; or (4) develop new recommenda-  Perspective: the original recommendation might have
tions using evidence provided by panelists (also referred to been developed from an individual or clinical
as expert input or expert evidence) [40]. perspective, while the urgent response has a popula-
tion or public health perspective. The new perspective
may broaden the relevant stakeholders (e.g., beyond
3.7.1. Adopt vs. adapt existing recommendations
the patients, their caregivers, and healthcare pro-
The literature search may find an original recommenda-
viders), and increase the weight of EtD criteria such
tion that is ‘adequate’ for the urgent response, i.e., is rele-
as cost effectiveness and impact on equity.
vant, credible and current. At that point, the expert panel
 Contextual factors: the judgment on contextual fac-
group should decide whether to adopt the original recom-
tors (such as resource use, impact on equity, accept-
mendation (i.e., use it as is) or adapt it. Practically, the
ability, and feasibility) might change when
expert panel group ‘adopts’ a recommendation when it does
considering the context of the rapid response (in com-
not change either the direction or the strength of that
parison to the original context).
recommendation. Otherwise, the recommendation is
‘adapted’. For each of the above factors, the judgment (between the
To decide on adopting vs. adapting an original recom- urgent guidance setting and the original one) needs to be
mendation, the expert panel group needs to assess whether sufficiently different for panel members to adapt the
8 E.A. Akl et al. / Journal of Clinical Epidemiology 129 (2021) 1e11

recommendation (i.e., modify its direction and/or strength). and then fill out a ‘‘panelist EtD’’ table [21]. In that table,
The recommendation statement wording may be edited to and in lieu of the systematically collected evidence, the
enhance the usability for the intended target group. panelists provide a description of their ‘‘expert evidence’’
WHO developed interim guidance for the ‘‘clinical man- consisting of their observations and experiences (equivalent
agement of severe acute respiratory infection when to case reports and case series) [40]; these are expected to
COVID-19 infection is suspected’’ in consultation with reflect ‘‘facts’’ (as opposed to opinions). In the next step,
the International Forum for Acute Care Trialists, Interna- the steering group collates the input from all the panelists
tional Severe Acute Respiratory and Emerging Infection and populates the EtD that will be used as the basis for
Consortium, and SSC [14]. Guideline developers originally panel discussion and consensus building. Typically, the
adapted this interim guidance from ‘‘Clinical management panel chair builds consensus with the panelists through dis-
of severe acute respiratory infection when Middle East res- cussion, and if needed through (iterative) voting. Alterna-
piratory syndrome coronavirus (MERS-CoV) infection is tively, and for efficiency purposes, the chair can make
suspected: interim guidance’’ [42]. In the first edition of suggestions and have panelists agree or disagree with them;
COVID-19 guidance, developers adopted many document however, caution is needed to ensure a broad range of per-
sections, research questions, and applicable guidance spectives is considered.
verbatim, adapting some directives to reflect underlying un-
certainty about the microbiological profile of COVID-19 3.8. Consider an updating plan
when informed from the indirect information from SARS
(caused by SARS-CoV1) and MERS (caused by MERS- Urgent situations are typically associated not only with
CoV) cases [14]. scarcity of data but also with rapidly developing evidence
base and contextual information. This raises the consider-
3.7.2. Develop new recommendation using existing ation of establishing an updating plan, ideally through a
adequate systematic review living process [8,44]. This is particularly relevant when
One other reasonable scenario would be to find an emerging data can potentially lead to a change in the
adequate systematic review that would jump-start the recommendation. The updating plan would need to define
development of de novo recommendation(s). The develop- the frequency of reassessment of the recommendation
ment would follow the standard GRADE evidence assess- (e.g., weekly, monthly), and to be adjusted to the speed
ment and recommendation development process with of development of the urgent situation and of the emer-
some potential shortcuts to ensure the process is completed gence of the evidence.
within the desired timeframe [1]. In the case of an urgency related to an environmental
The Infectious Diseases Society of America developed exposure, there might be a need to continuously monitor
rapid recommendations on the treatment and management
of hospitalized patients with COVID-19, including a
recommendation on corticosteroid treatment for hospital-
ized patients with acute COVID-19 [17]. At the time of
the first iteration of the guideline, the review team did not
identify any direct evidence to inform this recommenda-
tion; however, they identified a systematic review reporting
on corticosteroid use among patients with SARS-CoV-1 or
MERS-CoV [43]. The guideline panel determined this ex-
isting review to be direct enough to inform their
recommendation.

3.7.3. Develop new recommendation using expert panel

input
When no adequate recommendations are available for
adopting or adapting, and no adequate systematic reviews
are available to inform a de novo recommendation, using
expert panel input as the sole source of evidence is a
feasible alternative for an urgent response. It is important
to clarify that panel input is still important when developing
the recommendation based on existing recommendations or
systematic reviews.
Fig. 2 shows the steps involved in developing a new
recommendation using expert panel input. The panelists Fig. 2. Steps involved in developing a new recommendation using
are asked to review the literature they are provided with panel input.
 E.A. Akl et al. / Journal of Clinical Epidemiology 129 (2021) 1e11 9

the level of exposure within the population of interest for material is available in advance of catastrophic events
the purpose of triggering or updating the recommendations [47]. Nevertheless, there will continue to be new unex-
[44]. In the example of the nuclear incident, regular collec- pected incidents that require decisions to be made for
tion of environmental radioactive iodine levels from the which no relevant recommendations are already available
field would be needed to reverse the recommendation for or could have been foreseen. In these instances, it is crit-
mass evacuation. ical to leverage collaborations, capabilities, and resources
that allow for rapid assessment of adequate information to
support recommendation development. Moreover, it would
be important for coordinated efforts by guideline devel-
4. Discussion
opers to avoid duplication of efforts and sometimes incon-
Developing trustworthy guidelines in a relatively short sistent or even contradictory recommendations.
timeframe are of utmost importance but can be extraordi-
narily challenging. In this paper, we propose an approach
for developing trustworthy recommendations as part of an
urgent response (1e2 weeks). The approach offers the al- Acknowledgments
ternatives of using existing guidelines to adopt or adapt rec- Alexander G. Mathioudakis is supported by the NIHR
ommendations; using existent systematic reviews to Manchester Biomedical Research Centre (BRC), United
develop new recommendations; and, when the previous op- Kindgom. Srinivasa V. Katikireddi acknowledges funding
tions are not possible, relying on expert panel input to from an NRS Senior Clinical Fellowship (SCAF/15/02),
develop new recommendations. the Medical Research Council (MC_UU_12017/13),
The alternatives of adopting, adapting, and using exis- United Kindgom, and the Scottish Government Chief Sci-
tent reviews to developing new recommendations are entist Office (SPHSU13), United Kindgom. Derek K. Chu
similar to the ones proposed by the GRADE- is a CAAIF-CSACI-AllerGen Emerging Clinician-
ADOLOPMENT methodology [45]. Although that method- Scientist Research Fellow, supported by the Canadian Al-
ology was designed to address guideline development pri- lergy, Asthma and Immunology Foundation (CAAIF), Can-
marily to save and share resources, the approach ada, the Canadian Society of Allergy and Clinical
proposed in this paper is intended to address limited avail- Immunology (CSACI), Canada, and AllerGen NCE Inc.
able time (and also potentially limited resources, recog- (the Allergy, Genes and Environment Network supported
nizing that the absence of guideline recommendations by the Networks of Centres of Excellence), Canada.
may pose more of a threat than rapidly developed ones).
The alternative relying on expert panel input to
develop new recommendations might be perceived as References
going against the principles of evidence-based medicine.
[1] Schunemann HJ, Wiercioch W, Etxeandia I, Falavigna M,
However, Schunemann et al. [40] argued that expert Santesso N, Mustafa R, et al. Guidelines 2.0: systematic development
opinion can be considered as ‘‘expert evidence.’’ They of a comprehensive checklist for a successful guideline enterprise.
did caution however against confusing expert evidence CMAJ 2014;186(3):E123e42.
with expert’s views and judgments. Similarly, Djulbe- [2] Qaseem A, Forland F, Macbeth F, Ollenschlager G, Phillips S, van
govic and Guyatt [46] refer to the ‘‘experience of individ- der Wees P, et al. Guidelines International Network: toward interna-
tional standards for clinical practice guidelines. Ann Intern Med
ual clinicians’’ as a type of evidence. They also make the 2012;156:525e31.
point that evidence requires interpretation and ‘‘a [3] Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R,
consensus process must determine that interpretation’’ Akl EA, Davoli M, et al. GRADE Evidence to Decision (etd) frame-
in the context of guidelines. works: a systematic and transparent approach to making well
Although we may not have captured all situations unique informed healthcare choices. 2: Clinical practice guidelines. BMJ
2016;353:i2089.
to creating urgent responses, we have prioritized providing [4] Moberg J, Oxman AD, Rosenbaum S, Sch€unemann HJ, Guyatt G,
a broad, structured, and pragmatic approach. The illustra- Flottorp S, et al. The GRADE Evidence to Decision (EtD) framework
tive examples exemplify how a number of guideline organi- for health system and public health decisions. Health Res Pol Syst
zations had to rapidly react to urgent situations related to 2018;16(1):45.
[5] Neumann I, Brignardello-Petersen R, Wiercioch W, Carrasco-
environmental exposures or to the COVID-19 pandemic.
Labra A, Cuello C, Akl E, et al. The GRADE evidence-to-decision
Many of the examples highlight how flexibility was needed framework: a report of its testing and application in 15 international
on the side of the developer in terms of maintaining trust- guideline panels. Implement Sci 2016;11:93.
worthy methodology. [6] Thayer KA, Schunemann HJ. Using GRADE to respond to health
Having recommendations for situations that might recur questions with different levels of urgency. Environ Int 2016;92-93:
on an urgent basis may reduce the need for development of 585e9.
[7] Schunemann HJ, Hill SR, Kakad M, Bellamy R, Uyeki TM,
recommendations within urgent timeframes. For example, Hayden FG, et al. WHO Rapid Advice Guidelines for pharmacolog-
Public Health England has existing guidelines for handling ical management of sporadic human infection with avian influenza A
nuclear incidents that are regularly updated, so that this (H5N1) virus. Lancet Infect Dis 2007;7:21e31.
10 E.A. Akl et al. / Journal of Clinical Epidemiology 129 (2021) 1e11

[8] Vernooij RW, Sanabria AJ, Sola I, Alonso-Coello P, Martinez [26] Morgan RL, Whaley P, Thayer KA, Schunemann HJ. Identifying the
Garcia L. Guidance for updating clinical practice guidelines: a sys- PECO: a framework for formulating good questions to explore the as-
tematic review of methodological handbooks. Implement Sci 2014; sociation of environmental and other exposures with health out-
9:3. comes. Environ Int 2018;121(Pt 1):1027e31.
[9] Garritty CM, Norris SL, Moher D. Developing WHO rapid advice [27] Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, et al.
guidelines in the setting of a public health emergency. J Clin Epide- GRADE guidelines: 2. Framing the question and deciding on impor-
miol 2017;82:47e60. tant outcomes. J Clin Epidemiol 2011;64:395e400.
[10] World Health Organization. Guidelines for WHO guidelines. Geneva: [28] Analysis of epidemiological and clinical features in older patients
World Health Organization; 2003. with Corona Virus Disease 2019 (COVID-19) out of Wuhan. Clin
[11] Florez ID, Morgan RL, Falavigna M, Kowalski SC, Zhang Y, Etxean- Infect Dis 2020;25.
dia-Ikobaltzeta I, et al. Development of rapid guidelines: 2. A quali- [29] Alhazzani W, Moller MH, Arabi YM, Loeb M, Gong MN, Fan E,
tative study with WHO guideline developers. Health Res Policy Syst et al. Surviving sepsis campaign: guidelines on the management of
2018;16(1):62. critically ill adults with Coronavirus Disease 2019 (COVID-19).
[12] Kowalski SC, Morgan RL, Falavigna M, Florez ID, Etxeandia- Intensive Care Med 2020;46(5):854e87.
Ikobaltzeta I, Wiercioch W, et al. Development of rapid guidelines: [30] Clinical observation and management of COVID-19 patients. Emerg
1. Systematic survey of current practices and methods. Health Res 2020;9(1):687e90.
Policy Syst 2018;16(1):61. [31] European Food Safety Authority. Statement of EFSA on risks for
[13] Morgan RL, Florez I, Falavigna M, Kowalski S, Akl EA, Thayer KA, public health due to the presence of melamine in infant milk and
et al. Development of rapid guidelines: 3. GIN-McMaster Guideline other milk products in China. EFSA J 2008;6(9):807.
Development Checklist extension for rapid recommendations. Health [32] Darzi A, Harfouche M, Arayssi T, Alemadi S, Alnaqbi KA,
Res Policy Syst 2018;16(1):63. Badsha H, et al. Adaptation of the 2015 American College of Rheu-
[14] World Health Organization. Clinical management of severe acute res- matology treatment guideline for rheumatoid arthritis for the Eastern
piratory infection when novel coronavirus (2019-nCoV) infection is Mediterranean Region: an exemplar of the GRADE Adolopment.
suspected: interim guidance, 28 January 2020. World Health Organi- Health Qual Life Outcomes 2017;15(1):183.
zation; 2020. Available at https://apps.who.int/iris/handle/10665/ [33] Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M,
330893. Accessed October 17, 2020. et al. GRADE guidelines: 8. Rating the quality of evidence–indirect-
[15] Sch€unemann HJ, Wiercioch W, Brozek J, Etxeandia-Ikobaltzeta I, ness. J Clin Epidemiol 2011;64:1303e10.
Mustafa RA, Manja V, et al. GRADE Evidence to Decision (EtD) [34] Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F,
frameworks for adoption, adaptation, and de novo development of Feder G, et al. AGREE II: advancing guideline development, report-
trustworthy recommendations: grade-adolopment. J Clin Epidemiol ing and evaluation in health care. CMAJ 2010;182(18):E839e42.
2017;81:101e10. [35] Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al.
[16] World Health Organization. Infection Prevention and Control guid- AMSTAR 2: a critical appraisal tool for systematic reviews that
ance for Long-Term Care Facilities in the context of COVID-19: include randomised or non-randomised studies of healthcare inter-
interim guidance, 21 March 2020. World Health Organization; ventions, or both. BMJ 2017;358:j4008.
2020. Available at https://apps.who.int/iris/handle/10665/331508. [36] Whiting P, Savovic J, Higgins JP, Caldwell DM, Reeves BC, Shea B,
Accessed October 17, 2020. et al. ROBIS: a new tool to assess risk of bias in systematic reviews
[17] Bhimraj A, Morgan RL, Shumaker AH, Lavergne V, Baden L, was developed. J Clin Epidemiol 2016;69:225e34.
Cheng VC, et al. Infectious Diseases Society of America Guidelines [37] Garner P, Hopewell S, Chandler J, MacLehose H, Schunemann HJ,
on the treatment and management of patients with COVID-19. Clin Akl EA, et al. When and how to update systematic reviews:
Infect Dis 2020. consensus and checklist. BMJ 2016;354:i3507.
[18] Sultan S, Lim JK, Altayar O, Davitkov P, Feuerstein JD, [38] National Toxicology Program. National Toxicology Program research
Siddique SM, et al. AGA institute rapid recommendations for gastro- project: West Virginia chemical spill. Board of Scientific Counselors
intestinal procedures during the COVID-19 pandemic. Gastroenter- December 9e10, 2014 meeting. 2014. Available at https://ntp.niehs.
ology 2020. nih.gov/ntp/about_ntp/bsc/2014/dec/bsc_dec2014_minutes_508.pdf.
[19] Qaseem A, Yost J, Etxeandia-Ikobaltzeta I, Miller MC, Abraham GM, Accessed October 17, 2020.
Obley AJ, et al. Should clinicians use chloroquine or hydroxychloro- [39] National Toxicology Program. Nomination Summary for West Vir-
quine alone or in combination with azithromycin for the prophylaxis ginia Elk River chemical spill (N21408). 2014. Available at https://
or treatment of COVID-19? Ann Intern Med 2020. ntp.niehs.nih.gov/getinvolved/nominate/summary/nm-n21408.html?
[20] Cardiac involvement in a patient with coronavirus disease 2019 utm_source5direct&utm_medium5prod&utm_campaign5ntpgolinks
(COVID-19). JAMA Cardiol 2020. &utm_term5nm-n21408. Accessed October 17, 2020.
[21] Alonso-Coello P, Sch€unemann HJ, Moberg J, Brignardello- [40] Schunemann HJ, Zhang Y, Oxman AD, Expert Evidence in Guide-
Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision lines G. Distinguishing opinion from evidence in guidelines. BMJ
(EtD) frameworks: a systematic and transparent approach to making 2019;366:l4606.
well informed healthcare choices. 1: Introduction. BMJ 2016353. [41] Schunemann HJ, Khabsa J, Solo K, Khamis AM, Brignardello-
[22] Garritty C, Gartlehner G, Kamel C, King V, Nussbaumer-Streit B, Petersen R, El-Harakeh A, et al. Ventilation techniques and risk for
Stevens A. Interim guidance from the cochrane rapid reviews transmission of coronavirus disease, including COVID-19. Ann
methods group. Cochrane Rapid Rev 2020;2020. Intern Med 2020.
[23] Clinical characteristics of children with coronavirus disease 2019 in [42] World Health Organization. Clinical management of severe acute res-
Hubei, China. Curr Med Sci 2020;24. piratory infection when Middle East respiratory syndrome coronavi-
[24] Rapid risk assessment of acute public health events. World Health rus (MERS-CoV) infection is suspected: interim guidance. World
Organization; 2012. Available at https://www.who.int/csr/resources/ Health Organization; 2019. Available at https://apps.who.int/iris/
publications/HSE_GAR_ARO_2012_1/en/. Accessed October 17, handle/10665/178529. Accessed October 17, 2020.
2020. [43] Stockman LJ, Bellamy R, Garner P. SARS: systematic review of
[25] Petkovic J, Riddle A, Akl EA, Khabsa J, Lytvyn L, Atwere P, et al. treatment effects. PLoS Med 2006;3(9):e343.
Protocol for the development of guidance for stakeholder engage- [44] Akl EA, Meerpohl JJ, Elliott J, Kahale LA, Sch€unemann HJ. Living
ment in health and healthcare guideline development and implemen- systematic reviews: 4. Living guideline recommendations. J Clin Ep-
tation. Syst Rev 2020;9(1):21. idemiol 2017;91:47e53.
 E.A. Akl et al. / Journal of Clinical Epidemiology 129 (2021) 1e11 11

[45] Schunemann HJ, Wiercioch W, Brozek J, Etxeandia-Ikobaltzeta I, [47] Nisbet AF. Public health protection in radiation emergencies. PHE
Mustafa RA, Manja V, et al. GRADE Evidence to Decision (EtD) publication; 2019. Available at https://assets.publishing.service.gov.
frameworks for adoption, adaptation, and de novo development of uk/government/uploads/system/uploads/attachment_data/file/805655/
trustworthy recommendations: grade-adolopment. J Clin Epidemiol Advice_for_Radiation_Emergencies_2019.pdf. Accessed October
2017;81:101e10. 17, 2020.
[46] Djulbegovic B, Guyatt G. Evidence vs consensus in clinical practice [48] Fineberg HV. Ten weeks to crush the curve. N Engl J Med 2020;382:
guidelines. JAMA 2019;322:725e6. e37.