International Journal of Science and Research (IJSR)
ISSN: 2319-7064
ResearchGate Impact Factor (2018): 0.28 | SJIF (2018): 7.426
Meta-Analysis: A Higher Quality of Evidence in
Clinical Research Pyramid
Dr. Sonal Jain
Consultant-Medical Writer, Mumbai, Maharashtra, India
Abstract: Meta-analysis is a statistical tool used to systematically and quantitatively estimate the pooled effect of earlierindividual
research studies to obtain conclusions about that research subject. The outcomes from a meta-analysis are more precise estimate of the
effect of treatment or risk factor for disease, than any individual study. One of the potential advantages of conducting meta-analysis is to
obtain an accurate and quantitative review of a large, complex and generally controversial topic in literature. Meticulously conducted
meta-analyses are potential tools in evidence-based medicine. The small clinical trials (small sample size) might have inconclusive
results on individual basis however, while combining these trials met-analysis increases the statistical power and thus more precise
estimate of effect size. Moreover, meta-analysis is also useful for exploring sources of publication bias and quantifying betweenstudy heterogeneity.
Keywords: meta-analysis, systematic review, evidence-based medicine, effect size, forest plot, publication bias, heterogeneity
1. Introduction
1.1 Definition
Meta-analysis is defined as a systematic review which
provides quantitative estimate for the treatment effect of an
intervention or exposure on a specific topic of interest in the
literature.1Meta-analysis use statistical methods to calculate
an overall or ‗absolute‘ effect by pooling the findings of
single, independent studies.2 The main criteria to design a
high quality meta-analysis is to recognize a medical area
with ambiguity in the effect of the treatment or exposure but
a moderately homogenous availability of literature exists.
Meta-analysis aims at standardized research based process
for investigating the existing literature on a particular and
controversial clinical issue to achieve a conclusion vis-à-vis
the effect of a treatment or exposure.1
1.2 Importance of Meta-analysis in Medical Research
In medical research, evidence-based medicine (EBM) is the
best form of evidence and the excellent evidence in EBM is
from meta-analyses. Meta-analyses are advantageous as they
are less biased with more precision in calculating an
―absolute‖ effect of different studies on a clinical issue,
increase the statistical strength by merging the results of
previous studies thereby solving the problem of small
sample sizes and insufficient statistical strength. 3,
4
Researchersutilize
a
recognised
and
systematic
methodology to identify for differences in sample size,
heterogeneity in the selected studies and treatment effects
and determines the sensitivity of the results with regards to
systematic review protocol (study selection and statistical
analysis). 2Aproperly conducted meta-analysis of clinical
studies is ranked at topmost level in the hierarchy of
evidence and is considered as decisive evidence in medical
research.3,4
systematic reviews include a meta-analysis constituent to
synthesize the data from several studies into pooled
summary of effect size (Figure 1). Often, a meta-analysis is
conducted on randomized controlled trials (RCTs) which
have a high level of evidence to attain high accuracy and
reliability of the results.5
1.4 Hierarchy of Quality of Evidence of Meta-analysis in
Evidence-based Medicine
The hierarchy ranking of studies by evidence-based practice
(EBP) is based on the rigour (strength and precision) of their
research methods. 6According to quality of evidence
pyramid, the animal research and laboratory studies
constitutes the base of the pyramid (where ideas are first
developed). On progressing up the pyramid, the volume of
information available decreases but relevance to the clinical
setting increases. There is less bias on progressing up the
pyramid with each level of evidence compared to the level
below it. Therefore, meta-analyses can be seen as the apex
of evidence-based medicine with least possible bias.7
1.3 Systematic Reviews and Meta-analysis
A systematic review process comprises of collecting all
potential studies related to a given clinical topic and design,
and analyzes their results. The quality of studies is evaluated
and a statistical meta-analysis of the study results is
performed on the basis of their quality. Generally,
Figure 1: Schematic flow of steps involved for conducting a
systematic review5
*PICO: Participants Interventions Comparisons
Outcomes
Volume 9 Issue 3, March 2020
www.ijsr.net
Licensed Under Creative Commons Attribution CC BY
Paper ID: SR20304213942
DOI: 10.21275/SR20304213942
340
International Journal of Science and Research (IJSR)
ISSN: 2319-7064
ResearchGate Impact Factor (2018): 0.28 | SJIF (2018): 7.426
Figure 2: Quality of evidence of systematic reviews and meta-analysis5
The process flow for conducting a Meta-analysis
There are mainly eight steps to conduct a systematic review
and meta-analysis (Table 1).
Table 1: Essential steps for conducting a high-quality metaanalysis
Steps for metaanalysis
1. Formulate the
review
question1,8, 9
2. Identify
inclusion and
exclusion
criteria10
Procedure
Define objectives clearly1
Include only clinically relevant and
focused study questions 1
Focus on effectiveness of treatment not
credibly demonstrated in clinical trials1
Done by Cochrane acronym: PICO(S)
question format3
From search engines:
PubMed,
EMBASE, The Cochrane Central Register
3. Create a
of Controlled Trials
search strategy
(Cochrane CENTRAL)3
and identify
studies1, 8, 9
Ideally, identify and select randomized,
controlled clinical trials(RCTs)4
Criteria used is
Grading of
4. Determine
Recommendations,
Assessment,
quality of
Development and Evaluations (GRADE)
evidence of
system(http://www.gradeworkinggroup.or
studies
g/) 9
Create a simple data extraction form/table
to assimilate the information extracted
from each study
The information should include: authors,
5. Data
publication year, number of participants,
Extraction
age range, study design, outcomes.
Data extraction by at least two reviewers
is advised for more inter-rater reliability
and avoiding data entry errors.2, 9
6. Analyzing and Create forest plots3
interpreting
Utilize fixed and random effects model3
data
Assess between study- heterogeneity :
(Statistical
Cochrane Q test, I23
analysis)
7. Determining
By methods such as funnel plot or
publication
bias
8. Disseminating
results
sensitivity analysis 1
Based on preferred reporting items for
systematic reviews and meta-analyses
(PRISMA) guidelines3
2. Study Planning
a) Formulating a research question utilizing PICO(S)
structure
The global format for formulating the question for reviews
on interventions is the PICO(S) question (European Food
Safety Authority (EFSA), 2010). PICO(S) stands for
Population(the patient or problem),Intervention or exposure,
Comparison, clinical Outcome of interest and possibly Study
design(S). The PIT (population, index test and target
population) format is considered useful for formulating
research question for diagnostic test accuracy and the PO
format (population, outcome) for research questions for
descriptive parameters (such as incidence or prevalence),
respectively (EFSA, 2010).10
Even though other models exist such as SPIDER (sample,
phenomenon of interest, design, evaluation, research type)
and SPICE (setting, perspective, intervention, comparison,
evaluation), the PICO model is most reliable and widely
used for creating clinical questions. There are three
advantages of using PICO model: 1) It makes the questioner
to emphasize on what the patient/client believes to be the
one most important outcome 2) Iteases the next step in the
process, the computer based search by allowing to select
language or key terms for their search 3) It aids in clear
identification of the problem, intervention, and outcomes
related to specific care provided to a patient.11
b) Identify inclusion and exclusion criteria
An established and predefined inclusion and exclusion
criteria is a mandate for the studies included in the metaanalysis.1One objective of defining inclusion and exclusion
criteria is to identify a homogenous set of study population
Volume 9 Issue 3, March 2020
www.ijsr.net
Licensed Under Creative Commons Attribution CC BY
Paper ID: SR20304213942
DOI: 10.21275/SR20304213942
341
International Journal of Science and Research (IJSR)
ISSN: 2319-7064
ResearchGate Impact Factor (2018): 0.28 | SJIF (2018): 7.426
for the meta-analysis. The rationale for choosing the criteria
should be clearly stated, as it may not be perceptible to the
reader. 1It is a difficult exercise to choose a selection criteria
meeting all the requirements for a quality meta-analysis.1A
wide inclusion criteria is required to demonstrate safety and
effectiveness in a large group of patients and a number of
ethical and scientific reasons should be stated to define
exclusion criteria. However, if the inclusion criteria are too
wide, there is a risk of selection of poor quality studies
which can hamper the confidence in the final result. But if
the criteria are too stringent, risk of availability of only few
studies thus making results nongeneralizable.12The
optimization and finding a balance between these two
conditions result in subjective selection of inclusion and
exclusion criteria.12The main factors for creating an
inclusion criteria are study design, sample size, and subject
characteristics. Exclusion criteria include studies not
published in English or as full-length manuscripts.
Generally, a large percentage of studies (90% or more) are
excluded by the systematic reviewers. The main reasons for
exclusion are: (1) clearly meet one or more of the exclusion
criteria, (2) incomplete or unclear methods, (3)failure to
meet a pre-established threshold for quality, or (4)not
providing sufficient statistics or data for estimating effect
sizes.12
c) Create a literature search strategy and identify
studies
A broad and comprehensive search is indispensible to secure
proper basis for evidence-based research that includes
maximum studies meeting the inclusion and exclusion
criteria.
Classically, five bibliographic databases Medline5, Embase5,
and Cochrane Central Register of Controlled Trials
(CENTRAL)5,Cochrane Database1, and Cancerlit1are used to
perform a comprehensive search of the available studies. As
a first step the abstracts of all probable studies should be
extracted followed by full review of that study appearing to
meet inclusion criteria. This process of review is often done
by at least two reviewers to establish inter-rater reliability. It
is advised for authors to maintain a record of all reviewed
studies with reasons for inclusion or exclusion, and
information needed for data pooling (e.g., means, standard
deviations). 8There are eight key stages that relate
specifically to literature searching in systematic reviews. 13
The following criteria should be considered for a
comprehensive literature search for trials:
A search not limited to English language13
Cochrane CENTRAL or at least two other electronic
databases(such as MEDLINE or EMBASE) should be
searched 13
For unpublished trials, conference abstracts, theses, trials
registers; and contacts with experts in the field should be
searched. If not all four , at least one of the search
methods should be used. 13
Figure 3: Schematic procedure of literature searching- Eight step process13
3. Quality of Evidence
Despite how well the systematic review or meta-analysis is
planned, the quality of meta-analysis gets hampered or
inaccurate results are obtained if the quality of evidence in
the studies is low. Even the analysis involves RCTs of high
quality, assessing the quality of evidence specificallyaids in
determining the strength of recommendations in the meta-
analysis. The evaluation of the quality of evidence of studies
is done by the Grading of Recommendations, Assessment,
Development and Evaluations (GRADE) system.5
Key recommendations from GRADE for evaluating
quality of evidence
GRADE proposes four levels of evidence quality: high,
moderate, low, and very low.14
Volume 9 Issue 3, March 2020
www.ijsr.net
Licensed Under Creative Commons Attribution CC BY
Paper ID: SR20304213942
DOI: 10.21275/SR20304213942
342
International Journal of Science and Research (IJSR)
ISSN: 2319-7064
ResearchGate Impact Factor (2018): 0.28 | SJIF (2018): 7.426
RCTs depict as high quality evidence and observational
studies as low quality evidence. 14
The main factors downgrading the quality are: limitations
in study design, ambiguity of estimates (wide confidence
intervals), indirectness of evidence, variability in results or
publication bias. 14
The main factors upgrading the quality are: very large
magnitude of effect, a dose-response gradient, and if all
plausible biases would reduce an apparent treatment
effect. 14
The 5-point Oxford Quality Rating Scale is used for drug
trials. This measure is considerably influenced by doubleblinding and is commonly used in Cochrane reviews for
assessing RCTs-related study quality.The standard
recommendations of CONSORT (Consolidated Standards of
Reporting Trials statement) are used for psychological
interventions.8
Extracting the data
A simple data extraction form or table can be created to
extract required information in an organized manner from
each reviewed study (e.g., authors, publication year, number
of participants, age range, study design, inclusion/exclusion
criteria and outcomes). Data extraction should be conducted
by at least two reviewers for establishing inter-rater
reliability and evading data entry errors.8
Due to difference in the size and format of each variable, the
size and format of the outcomes are different, and minor
changes may be required while combining the data. If the
differences in the size and format of the outcome variables
cause difficulties in combining the data (for example: the
use ofdifferent evaluation instruments or different evaluation
time-points), then meta-analysis should not be conducted
and the analysis may be limited to a systematic review only.
2) Type I(α) and type II (β)errors
There are two types of errors affecting the quality of metaanalysis:
a) Type I error (α) occur which rejects the null hypothesis
when it is true.
b) Type II error (β) which fails to reject the null hypothesis
when it is false.
A β = 0.2 (power 80%) generally reflects high power and
should be the targeted minimum value. The parameters on
which power mostly depends include the overall effect size,
the average group size, the number of included studies and
their heterogeneity. 15
3) Dichotomous variables and continuous variables
The outcome variables can be broadly discussed in terms of
dichotomous variables and continuous variables.
Effect measures for continuous outcomes
The two most common effect measures for continuous
outcomes are mean difference (MD) and standardized mean
difference or effect size (SMD)(Table 2).5,16
MD = Absolute difference between the mean value in two
groups5
SMD = Difference in mean outcome between groups
Standard deviation (SD) of outcome among participants
The MD is defined as the absolute difference in mean values
between the groups, and the SMD is the mean difference
between groups divided by the standard deviation (SD). 5, 15
Depending on the method of calculation of SD , the SMD
has several variations such, as Cohen's d, Glass's Δ, and
Hedges' g.17When results are in the same units(scale) in all
the studies the MD can be used, but when results are
presented in different units(scale), the SMD should be used.
5,16
9
4. Data Analysis
1) Effect Size
Effect sizes provide mean change in the dependent variable
in each study in a standardized manner. These can include
considerations of sample size. Moreover, in pooled effect
sizes as in meta-analysis, these can be weighted by the
variance of the estimate, so that the studies with lower
variance (i.e. tighter estimated effect size) are given more
weight in the data set. As there is inverse relationship
between variance and sample size (decrease in variance will
cause increase in sample size) therefore effect sizes based on
larger study populations are given greater weight. 15
The three most common statistical methods used to obtain
reliable and comparable effect sizes for meta-analysis:
a) For continuous response variables between control and
experimental groups, the mean difference(MD) between
the groups is considered as an appropriate method to
calculate effect size. 15
b) For testing for an effect of a continuous or ordinal
categorical variable on a continuous response variable,
the correlation coefficient can be used. 15
c) For dichotomous response variables the risk ratio (RR) or
odds ratio (OR) provides a measure of effect size. 15
Interpretation of MD and SMD
A value of zero (0): Effects of the new and the existing
treatment method are the same.5
A value lower than zero (0): the new treatment method is
less effective than the existing method. 5
A value greater than zero (0): the new treatment is more
effective than the existing method. 5
Effect measures for dichotomous outcomes
For pooling data for dichotomous variables, the OR, RR, or
risk difference (RD) can be used. The RR and RD can be
used as effect measures for RCTs, quasi-experimental
studies, or cohort studies, whereas the OR are appropriate
for other case-control studies or cross-sectional studies. The
dichotomous variable can be presented as the number
needed to treat (NNT), which signifies the minimum number
of patients needs to be treated in the treatment group for an
identified event to occur in at least one patient compared to
the control group. 5,18
4) Fixed-effect models and random-effect models
The analysis of effect size can be done by using two models:
a fixed-effect model or a random-effect model.5, 19
Volume 9 Issue 3, March 2020
www.ijsr.net
Licensed Under Creative Commons Attribution CC BY
Paper ID: SR20304213942
DOI: 10.21275/SR20304213942
343
International Journal of Science and Research (IJSR)
ISSN: 2319-7064
ResearchGate Impact Factor (2018): 0.28 | SJIF (2018): 7.426
Fixed-effect models
According to the fixed-effect model theory the effect of
treatment is the same in all the studies and any variation is
due to random error. Therefore, the fixed-effect model is
ideal to use for the studies with same design and
methodology, or the variability in results within a study is
small, and the variance in the studies is considered due to
random error. Three common methods used to analyse effect
size in a fixed-effect model are: 1) inverse varianceweighted estimation, 2)Mantel-Haenszel estimation, and 3)
Peto estimation (Table 2). 5, 19
Random-effect models
A random-effect model presumesheterogeneity(variation)
between the studies being combined.An important feature of
the random-effects model is that there is a distribution of
true effect sizes. 5, 19The methods commonly used to analyse
effect size in a random-effect model are: 1) the DerSimonian
and Laird method is mostly used for dichotomous variables,
2) inverse-weighted estimation is used for continuous
variables.5
Differences between fixed- and random-effects models
• The random-effects models have wider confidence
interval (CI) for the summary effect.19
• More similar study weights in the random-effects model
(small studies gain influence and large studies lose
influence). 19
• There is difference in the estimate of the effect size under
the two models.19
If the binary outcome variables are used, deciding for the
right model for analysis is crucial, as fixed and random
effects models give different results. In case of continuous
variables, fixed or random models are often produce
identical results in meta-analyses.5,19
Table 2: Recommended pooling methods 5
Fixed-effect
models
MantelHaenszel (MH)
Dichotomous Odds ratio (OR)
Inverse
variance (IV)
Peto
MantelHaenszel (MRisk ratio (RR), Risk
H)
difference (RD)
Inverse
variance (IV)
Mean difference
Inverse
(MD),
Continuous
Standardized mean variance (IV)
difference (SMD)
Data outcome
variable
Effect measure
Random-effect
models
MantelHaenszel (MH)
Inverse
variance (IV)
MantelHaenszel (MH)
Inverse
variance (IV)
Inverse
variance (IV)
5) Heterogeneity
Heterogeneity is calculated by homogeneity test which tells
if the degree of heterogeneity is greater than would be
expected to occur naturally when the effect size calculated
from several studies is higher than the sampling error.
Mostly three types of homogeneity tests can be used: 1)
forest plot, 2) Cochrane‘s Q test (chi-squared), and 3)
Higgins I2 statistics.5, 20
In the forest plot a higher homogeneity is there if there is
greater overlap between the confidence intervals. In the Q
statistic method, if the P value of the chi-squared test is less
than 0.1, a statistical heterogeneity is considered and a
random-effect can be used.5,20
However, the best approach is the I2 statistics because it
provides the quantitative estimate of the effect of
heterogeneity and a measure of the degree of inconsistency
in the studies‘ results.5, 20
The formula to calculate I2:
I2 = 100%×(Q − df)/Q
where Q= Cochran‘s heterogeneity statistic
df= the degrees of freedom.
The degree of heterogeneity as defined by the values for
I2can be categorised into three grades of low, moderate, and
high with I2 values of 25%, 50%, and 75% respectively. 5,20
Advantages of I2
Pays attention on the effect of any heterogeneity on the
meta-analysis. 20
Simple to calculate and can be generally obtained from
published meta-analyses. 20
Does not inherently depend on the number of studies in
the meta-analysis. 20
Interpretation can be similar regardless of the type of
outcome data (eg dichotomous, quantitative, or time to
event) and type of effect measure (eg ORor hazard ratio
(HR)). 20
Heterogeneity causes an asymmetry funnel plot if it induces
a correlation between study sizes and intervention
effects. For example, considerable treatment benefits might
be observed only in high risk patients and these may be
preferentially included in early and small studies. Contrarily,
the intervention may have been employed less thoroughly in
larger studies, resulting in smaller effect estimates compared
with smaller studies.21
6) Publication bias
The most common type of bias in meta-analyses is the
publication bias. This denotes to the misrepresentation of
meta-analysis outcomes due to the higher probability of
publication of statistically significant studies rather than
non-significant studies. 5
Reasons for Publication Bias
Unpublished studies. 21
Journals may be biased toward positive results because
negative results are less likely to be published. 21
Some studies get initiated but do not complete. These
studies are available in the form of conference abstracts
but not in the clinical trial study databases. Therefore they
are missed. 21
There occurs language bias because while literature
searches for systematic review, generally publications in
one language are considered (preferably only English).21
Funnel Plot
A funnel plot is a type of scatter plot of the effect estimates
from distinct studies against some measure of each study‘s
Volume 9 Issue 3, March 2020
www.ijsr.net
Licensed Under Creative Commons Attribution CC BY
Paper ID: SR20304213942
DOI: 10.21275/SR20304213942
344
International Journal of Science and Research (IJSR)
ISSN: 2319-7064
ResearchGate Impact Factor (2018): 0.28 | SJIF (2018): 7.426
precision. The standard error (SE) of the effect estimate is
plotted on the vertical axisand generally used as the measure
of study precision with a reversed scale that places the
larger, most powerful studies towards the top. Effect
estimates include relative risk(RR), risk ratio(RR), OR,
absolute risk and logarithmic transformations of these
measures. 22
When the publication bias is absent, the points will be
symmetrically distributed around the true effect in the form
of an inverted funnel (Figure 3). 23
nodes within the systematic review process which can
generate a need for a sensitivity analysis.If results remain
same across the different analyses, the results reflect the
robustness as even with different decisions they remain them
similar. However, if the results differ across sensitivity
analyses, then interpretation of the results need caution.The
best way to report sensitivity analyses in a systematic review
is by producing a summary table. It is generally not
informative to create individual forest plots for each
sensitivity analysis undertaken.24
Various decision nodes for performing a sensitivity analysis
in a meta-analysis include:24
a) Study Search
b) Eligibility criteria
c) Type of data to be analysed
d) Data analysis methods
Figure 3: A symmetrical funnel plot
When the publication bias is present the funnel plot will be
asymmetric. The pooled effect estimate will be diverged
from the true effect due to negative effect estimates from
smaller studies missing from the plot. 23
Egger’s Test
This test uses linear regression instead of correlation and
estimates for asymmetry of the funnel plot.21
If publication bias is absent, all studies on X axis (1/SE) will
be nearby the regression line of Y axis (effect size/SE), with
large sample studies away from origin and small sample
studies near origin. This indicates that there would be lesser
publication bias if the regression line has intercept value
close to zero.21
Begg and Mazumdar’s rank correlation test
The Begg and Mazumdar rank correlation test uses the
correlation between the ranks of effect sizes and the ranks of
their variances.21The four important properties of this test
are:
a) Difference between concordant and discordant ranks.21
b) Kendall‘s rank
correlation
for
testing
the
interdependence of variance and effect size. The method
utilizes Kendall‘s rank correlation rather than the
ordinary product moment correlation, because normal
distributions of effect sizes are improbable here.21
c) z-value.21
d) p-value for this correlation.21
In meta-analysis containing 75 or more studies this test for
estimating publication bias is powerfulbut shows low power
when meta-analysis has less than 25 studies. Therefore, for
small meta-analyses the results of this test must be
interpreted carefully.
7) Sensitivity analysis
A sensitivity analysis is defined as repeat of the primary
meta-analysis in which alternative decisions or ranges of
values for decisions that were unclear while performing
meta-analysis are substituted. There are many decision
Some examples for sensitivity analysis:
a) Exploring the effect of using different meta-analysis
models (random or fixed effect models). 24
b) Exploring the effect of exclusion or inclusion of studies
in meta-analysis based on sample size, methodological
quality, or variance. 24
c) If the eligibility criteria of some studies in the metaanalysis arenot fulfilled due to lack of complete required
details, sensitivity analysis might involve performing the
meta-analysis twice: first, including all studies and
second, only including those that are definitely within
eligibility criteria.24
8) Interpreting and analysing “forest plot”
The results of a meta-analysis are typically represented as
‗forest plots‘.These meta-analysis graphs can primarily be
divided into six columns. The rows represent the results of
individual studies (Figure 4). 25, 26
a) The first column represents the individual studies
included in the meta-analysis. Generally the first author
and year are shown. 25, 26
b) The second column displays intervention/treatment
groups. 25, 26
c) The third column indicates the control groups. 25, 26
d) The fourth column represents visual display of the study
results. The contents of the fourth column can be
interpreted as follows: 25, 26
The middle line is called ‗the line of no effect or null
effect‘, which has the value of either 0 for continuous
outcome variable (eg. WMD) or 1 for
binary/dichotomous outcome variable (eg. ORorRR).
If if OR or RR = 1 or WMD = 0, it indicates that no
difference between the intervention and the control
group. 25, 26
Each box represent the ―weighting‖ of that individual
study in the meta-analysis. These boxes indicate
sample size of the study. The bigger is the box the
more are the participants in the study. The boxes are
situated in line with the outcome value of the
individual studies, also called the effect sizes (eg. OR,
RR or WMD). The value axis is situated at the bottom
of the graph.25, 26
The whiskers (horizontal lines) through the boxes
portray the length of the confidence intervals (CI). The
CI is defined as: ―The range of values within which
Volume 9 Issue 3, March 2020
www.ijsr.net
Licensed Under Creative Commons Attribution CC BY
Paper ID: SR20304213942
DOI: 10.21275/SR20304213942
345
International Journal of Science and Research (IJSR)
ISSN: 2319-7064
ResearchGate Impact Factor (2018): 0.28 | SJIF (2018): 7.426
you can be 95% certain the true value lies.‖ These
CI lines can aid in identifying the precision of the
study results. The longer lines depict wider CIs and
thus the less precise the study results. A study shows
greater weight if the bigger the sample size and the
narrower the CI.Numerically, the CI does not include 0
for continuous outcome variables and 1 for binary
outcome variables.25, 26Fundamentally:
The bigger the study, the bigger is box and
the smaller the horizontal line representing the
point estimate. This can be understood that the
probability of these studies of crossing the line of
null effect is less because 95% CI should have a
much smaller range. 25, 26
The smaller the study, smaller is box and
the wider the horizontal line representing the point
estimate. This can be understood that the
probability of these studies of crossing the line of
null effect is high because 95% CI have much
bigger range. 25, 26
e) The fifth column represents the weight (in %) indicates
the influence ofthe study on the overall results of the
meta-analysis of all included studies. The influence of
the study on overall meta-analysis results will be more if
the percentage weight is higher and the box is bigger. 25,
26
f) The sixth column shows the numerical values for each
study (eg. OR or RR or WMD and 95% CI) which are
same to the graphical results in the fourth column. 25, 26
The diamond represents the pooled result of the metaanalysis. The middle of the diamond sits on the value for the
overall effect sizes (eg. OR, RR or WMD) and the width of
the diamond depicts the width of the overall CI. Statistical
significance of the overall result is also expressed by p value
in the 'test for overall effect' which is considered as
statistically significant if p<0.05. Also, in case of binary
variables, effect values are always greater than 0 but in case
of continuous variables values can be negative or positive. 25,
26
Figure 4: Example of Forest Plot
(This example is only for representation purposes to interpret forest plot)
Softwares for Creating Forest Plots
Forest plots in Review Manager (RevMan)
RevMan provides a flexible framework for preparation of
protocols and full reviews, together with text, characteristics
of studies, comparison tables, and study data. The metaanalysis of the data entered can be performed and results are
graphically represented as forest plots in the ‗Data and
analyses‘ section. The review types which can be performed
in RevMan 5.3(recent version) include intervention reviews,
diagnostic test accuracy reviews, methodology reviews,
overviews of reviews, review properties, licence for
27
publication and flexible reviews.
RevMan offers multiple selections for data analyses (e.g.
choosing between fixed and random-effects meta-analyses),
or using different effect measures and graphics (e.g. scale of
axes and ordering of studies). Default analyses are shown
unless options are ignored. The defaults are MantelHaenszel odds ratios for dichotomous data, fixed-effect
meta-analyses of mean differences for continuous data, Peto
odds ratios for ‗O–E and Variance‘ outcomes and fixedeffect meta-analyses for generic inverse variance
outcomes.28
The most commonly used version is RevMan 5.3 which has
comprehensive features for data analyses. The following
tabs are available for data analyses:27
Constructing a comparison table
Comparison properties
Outcome properties
Adding studies to an analysis
Entering data
Calculating data
Generating and Analysis forest plots
Funnel plots
Sensitivity analysis
Go to the following link to download user guide and get
more information on all tabs/ selections available in the
software:
https://community.cochrane.org/sites/default/files/uploads/in
line-files/RevMan_5.3_User_Guide.pdf
Steps for creating forest plot in RevMan
(These screen shots are created for explaining the steps
only and are not the part of any existing review)
Step 1- Creating a RevManoutline for review
1) Start RevMan
2) Select the ―Close‖ button on the right-lower corner of the
first prompt.
3) Select ―New‖in Filemenu on the menu bar on the top.
4) Select ―Next‖in New Review Wizard
5) Select type of Review and then click ―Next‖ (Figure 5)
6) Input in PICO Short Title in the last field
7) Select Review stage, and then select ―Finish‖
Volume 9 Issue 3, March 2020
www.ijsr.net
Licensed Under Creative Commons Attribution CC BY
Paper ID: SR20304213942
DOI: 10.21275/SR20304213942
346
International Journal of Science and Research (IJSR)
ISSN: 2319-7064
ResearchGate Impact Factor (2018): 0.28 | SJIF (2018): 7.426
Figure 6: Data and analyses for creating forest plot
Figure 5: Type of Review
Step 2-Entering details of included studies
1) Click ―Studies and References‖ in left side-bar
2) Click ―Add Study‖ on the main screen under ―Included
Studies‖
3) Input study details: Author, year , then click ―Next‖
4) Select Published data only (which is automatically
selected) and then click ―Next‖
5) Input year if not automatically populated and then
click ―Finish‖
6) Repeat all the above steps for all individual studies
included in the meta-analysis
Step 3-Creating forest plots
1) Click ―Data and analyses‖ in left-sidebar (Figure 6)
2) Click ― Add comparison‖ on the main screen below
―Data and Analyses‖
3) Input Name of PICO in New Comparison Wizard, eg
―Hydrotherapy vsTraditional Physiotherapy
in
Osteoarthritis (OA) in relieving pain‖ then click
―Next‖
4) Select ―Add an outcome under the new comparison‖
and then click ―Continue‖
5) Select ―Dichotomous or Continuous‖ based on Data
Type, then click ―Next‖( Figure 7)
6) Input Outcome Name, eg. ―Relieve in OA pain‖
7) Edit Group Label 1, eg. ―Hydrotherapy‖ then, Label2,
eg. ―Traditional Physiotherapy‖ and then click Next
8) Select the method for effect-measure ―Risk Ratio‖ or
―Odds Ratio‖, ― Mean Difference‖ whatever is
applicable as per data type and then click ―Next‖
9) Select Results of meta-analysis, ―Total and subtotals‖,
―90%, 95 % CI‖ and then click ―Next‖
10) Enter Forest Plot labels, then click ―Next‖
11) Select ―Add study data for the new outcome‖, then
click ―Continue‖
12) Select all studies with the outcome under review (for
relieving OA pain), then ―Finish‖
13) Enter the numerical values of outcomes for each the
controland treatment groupsin the white boxes on the
center of the screen.
14) Click the blue forest plot icon to the right of the FE
button above the forest plot which is created.
15) Click ―Add as Figure‖( A plot similar to Figure 4 will
be displayed on the screen)
16) Click ―Figures‖in left-side bar to see a list of forest
plots
17) Use the―File‖ menu to save your RevMan forest plot
file
Figure 7: Data type selection in RevMan
Forest Plots in R Studio
R package meta is a user-friendly package providing
standard methods for meta-analysis.29
For more information go to the link:http://meta-analysiswith-r.org/
R package meta provides the following statistical methods
for meta-analysis:
1) For fixed effect and random effects model 29
For continuous outcome data (metacont)
For binary outcome data (metabin)
For incidence rates (metainc)
For generic inverse variance meta-analysis (metagen)
For single correlations (metacor)
For single means (metamean)
For single proportions (metaprop)
2) Offer different types of plots for meta-analysis29
Forest plot (forest)
Funnel plot (funnel)
Galbraith plot / radial plot (radial)
L‘Abbe plot for meta-analysis with binary outcome data
(labbe)
Baujat plot to explore heterogeneity in meta-analysis
(baujat)
Bubble plot to display the result of a meta-regression
(bubble)
3) Statistical tests for funnel plot asymmetry (metabias) and
4) Trim-and-fill method (trimfill) to evaluate bias in metaanalysis29
5) Cumulative meta-analysis (metacum) and leave-one-out
meta-analysis (metainf)29
6) Meta-regression (metareg)29
Standard Practice for Reporting of Meta-analyses- The
Preferred Reporting Items for Systematic Reviews and
Meta-Analyses (PRISMA) Guidelines
The PRISMA Statement comprise of a 27-item checklist and
a four-phase flow diagram for researchers to re-use. The
Volume 9 Issue 3, March 2020
www.ijsr.net
Licensed Under Creative Commons Attribution CC BY
Paper ID: SR20304213942
DOI: 10.21275/SR20304213942
347
International Journal of Science and Research (IJSR)
ISSN: 2319-7064
ResearchGate Impact Factor (2018): 0.28 | SJIF (2018): 7.426
objective of the PRISMA Statement is to support authors to
improve thereporting of systematic reviews and metaanalyses.
For 27-item checklist go to the link:
http://prismastatement.org/PRISMAStatement/Checklist.aspx
For four-phase flow diagram go to the link:
http://prismastatement.org/PRISMAStatement/FlowDiagram.aspx
5. Conclusion
Meta-analyses are generally conducted by pharmaceutical
companies for obtaining marketing authorization from
regulatory authorities for new drugs if sometimes demand a
meta-analysis as part of the approval process. For clinicians
and applied researchers in medicine, education, psychology,
criminal justice, meta-analysis is a tool to find out which
interventions work and comparisons with other interventions
to identify the superiority of a specific drug. Other domains
where meta analysis is widely used to assess the evidence
include sociology, social psychology, sex differences,
finance and economics, marketing, political science ,ecology
and genetics,. 31The conventional medical practice trends
have been changed by the use of randomized, blinded,
multicenter clinical trials and meta-analysis, leading to
development of ―EBM‖. The key players in initiating this
change have been the Cochrane Collaboration who have
developed standard practice guidelines for conducting
systematic reviews and meta-analyses and more recently the
PRISMA statement has been introduced to improve
reporting of systematic reviews and meta-analyses. One
major issue in determining whether studies can be pooled is
the extent of heterogeneity among individual studies. The
graphical statistical tools for assessing heterogeneity such as
Cochrane‘s Q test (chi-squared), and Higgins I2 statistics
have been found appropriate approaches for conducting high
quality meta-analyses. Publication bias is anotherbasic issue
which can be estimated by funnel plot, Begg and
Mazumdar's rank correlation, and Egger regression methods.
Despite of all these, meta-analysis is an infallible tool with
apex hierarchy in clinical research pyramid with least
possible bias.
References
[1] Russo
MW.
How
to
Review
a
Metaanalysis.GastroenterolHepatol (N Y). 2007; 3(8): 637–
642.
[2] Shorten Aand Shorten B. What is meta-analysis?
Evidence Based Nursing. 2013; 16(1):3-4.
[3] Lee YH. An overview of meta-analysis for
clinicians.Korean J Intern Med.2018;33:277-283.
[4] Haidich
AB.
Meta-analysis
in
medical
research.HIPPOKRATIA. 2010; 14 (Suppl 1): 29-37.
[5] Ahn E and Kang H. Introduction to systematic review
and meta-analysis.Korean J Anesthesiol. 2018; 71(2):
103-112.
[6] National
Health
and
Medical
Research
Council. NHMRC additional levels of evidence and
grades for recommendations for developers of
guidelines.
2009.
Available
at
URL:
https://www.mja.com.au/sites/default/files/NHMRC.lev
els.of.evidence.2008-09.pdf (Accessed on November
30, 2019).
[7] Meta-analysis:
What,
Why,
and
How.
https://uk.cochrane.org/news/meta-analysis-what-whyand-how (Accessed on November 30, 2019).
[8] Uman LS. Systematic Reviews and Meta-Analyses. J
Can Acad Child Adolesc Psychiatry. 2011; 20(1):57-59.
[9] Ahn EJ and Kang H. Introduction to systematic review
and meta-analysis.Korean J Anesthesiol.2018; 71(2):
103-112.
[10] O‘Connor AM, Anderson KM, Goodell CK and
Sargeant JM. Conducting Systematic Reviews of
Intervention Questions I: Writing the Review Protocol,
Formulating the Question and Searching the Literature.
Zoonoses and Public Health.2014; 61 (suppl. 1): 28–38.
[11] Eriksen MB and FrandsenTF.The impact of patient,
intervention, comparison, outcome (PICO) as a search
strategy tool on literature search quality: a systematic
review.J Med Libr Assoc. 2018; 106(4): 420–431.
[12] Meline T. Selecting Studies for Systematic Review:
Inclusion and Exclusion Criteria.CONTEMPORARY
ISSUES IN COMMUNICATION SCIENCE AND
DISORDERS. 2006; 33: 21–27.
[13] Cooper C, Booth A, Varley-Campbell J, Britten N and
Garside R. Defining the process to literature searching
in systematic reviews: a literature review of guidance
and supporting studies.BMC Medical Research
Methodology.2018; 18:85.
[14] Guyatt GH, Oxman AD, EVist G, Kunz R,Falck-Ytter
Y and Schünemann HJ. GRADE: what is ―quality of
evidence‖
and
why
is
it
important
to
clinicians?BMJ.2008;336: 995-998.
[15] Harrison F. Getting started with meta-analysis. Methods
in Ecology and Evolution.2011; 2:1–10.
[16] Zlowodzki M, Poolman RW, Kerkhoffs GM, Tornetta P
III, andBhandar M. How to interpret a meta-analysis
and judge its value as a guide for clinical
practice?ActaOrthopaedica. 2007; 78 (5): 598–609.
[17] Takeshima N, SozuT ,Tajika A , Ogawa Y , Hayasaka
Y and Toshiaki A Furukawa TA.Which is more
generalizable, powerful and interpretable in metaanalyses, mean difference or standardized mean
difference? BMC Medical Research Methodology.
2014; 14(30): 1-7.
[18] Borenstein M, Hedges LV, Higgins JPT and Rothstein
HR. Chapter 5.Effect Sizes Based on Binary Data (2X 2
Tables) Introduction to Meta-Analysis.2009; John
Wiley & Sons, Ltd.
[19] Borensteina M, Hedges LV, Higgins JPT and Rothstein
HR. A basic introduction to fixed-effect and randomeffects models for meta-analysis. Res. Syn. Meth.
2010;1: 97—111.
[20] Higgins JPT, Thompson SG, Deeks JJ and Altman DG.
Measuring
inconsistency
in
meta-analyses.
BMJ.2003;327(7414) : 557–60.
[21] Kale J and Nirpharake A. Comparison of Different
Methods
of
Detecting
Publication
Bias.https://www.phusewiki.org/docs/Conference%202
017%20AS%20Papers/AS04.pdf
(Accessed
on
February 3, 2020)
Volume 9 Issue 3, March 2020
www.ijsr.net
Licensed Under Creative Commons Attribution CC BY
Paper ID: SR20304213942
DOI: 10.21275/SR20304213942
348
International Journal of Science and Research (IJSR)
ISSN: 2319-7064
ResearchGate Impact Factor (2018): 0.28 | SJIF (2018): 7.426
[22] Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones
DR, Lau J etal. Recommendations for examining and
interpreting funnel plot asymmetry in meta-analyses of
randomised controlled trials. BMJ.2011;342:d4002.
[23] Liu JLY.The role of the funnel plot in detecting
publication
and
related
biases
in
metaanalysis.Evidence-Based Dentistry
[24] 2011; 12: 121-122.
[25] Sensitivity
analyses.https://handbook-51.cochrane.org/chapter_9/9_7_sensitivity_analyses.htm(
Accessed on February 3, 2020).
[26] Ried K. Interpreting and understandingmeta-analysis
graphs. Reprinted from Australian Family Physician.
2006; 35(8): 635-638.
[27] How
to
read
a
forest
plot?https://uk.cochrane.org/news/how-read-forestplot(Accessed on February 6, 2020).
[28] RevMan
5.3
User
Guide.
https://community.cochrane.org/sites/default/files/uploa
ds/inline-files/RevMan_5.3_User_Guide.pdf(Accessed
on February 19, 2020).
[29] Forest Plots in RevMan. https://handbook-51.cochrane.org/chapter_11/11_3_2_1_forest_plots_in_r
evman.htm(Accessed on February 19, 2020).
[30] Schwarzer G etal.Package ‗meta‘. General Package for
Meta-Analysis.
2020.
https://cran.rproject.org/web/packages/meta/meta.pdf(Accessed on
February 25, 2020).
[31] Moher D, Liberati A, Tetzlaff J,Altman DG and The
PRISMA Group. Preferred Reporting Items for
Systematic Reviews andMeta-Analyses: The PRISMA
Statement.PLoS Med. 2009; 6(7): e1000097.
[32] Why perform a meta-analysis?https://www.metaanalysis.com/pages/why_do.php?cart=(Accessed
on
February 27, 2020)
Volume 9 Issue 3, March 2020
www.ijsr.net
Licensed Under Creative Commons Attribution CC BY
Paper ID: SR20304213942
DOI: 10.21275/SR20304213942
349