Annals of Internal Medicine Research and Reporting Methods

Random-Effects Meta-analysis of Inconsistent Effects: A Time

for Change
John E. Cornell, PhD; Cynthia D. Mulrow, MD, MSc; Russell Localio, PhD; Catharine B. Stack, PhD, MS; Anne R. Meibohm, PhD;
Eliseo Guallar, MD, DrPH; and Steven N. Goodman, MD, PhD

A primary goal of meta-analysis is to improve the estimation of ing universal use of the DL estimator with analyses based on a
treatment effects by pooling results of similar studies. This article critical synthesis that recognizes the uncertainty in the evidence,
explains how the most widely used method for pooling heteroge- focuses on describing and explaining the probable sources of vari-
neous studies—the DerSimonian–Laird (DL) estimator— can pro- ation in the evidence, and uses random-effects estimates that pro-
duce biased estimates with falsely high precision. A classic example vide more accurate confidence limits than the DL estimator.
is presented to show that use of the DL estimator can lead to
erroneous conclusions. Particular problems with the DL estimator
are discussed, and several alternative methods for summarizing Ann Intern Med. 2014;160:267–270. www.annals.org
heterogeneous evidence are presented. The authors support replac- For author affiliations, see end of text.

T he basic premise of meta-analysis is that the average of

estimates provided by a group of studies is closer to the
truth than the estimate provided by an individual study.
frequently used to illustrate different methods for estimat-
ing a common treatment effect when the body of evidence
is heterogeneous (12, 13). The effect estimates from the
This premise rests on the assumption that each study is a individual studies range from a more than 4-fold statisti-
near-replication of a single experiment and that differences cally significant decrease in the odds of eclampsia with
among study results are due only to chance. The technical diuretics observed in the study by Fallis and colleagues (5)
jargon for this fundamental assumption is that each of the to an almost 3-fold nonsignificant increase in the study by
studies is estimating the same “fixed effect,” and the corre- Tervilä and Vartiainen (9). A visual clue that these studies
sponding meta-analytic approach is dubbed the “fixed- are statistically heterogeneous is that the confidence limits
effects model.” of several pairs of studies do not overlap.
When studies are statistically heterogeneous and dif- The Figure shows that different statistical approaches
ferences among their results cannot be explained by chance to combining data can produce results leading to different
alone, the meta-analyst faces a conundrum. Qualitative conclusions. The fixed-effects model, which is not appro-
heterogeneity among study designs, patient characteristics, priate for these data, shows a summary effect of 0.67, with
and treatment and comparator regimens may be so great 95% confidence limits (0.56 and 0.80) that are 19% less
that it does not make sense to combine studies to derive a than and greater than that value. The DL random-effects
single summary estimate. However, when the qualitative estimate shows a slightly larger effect (odds ratio, 0.60),
and quantitative heterogeneity is not so great that a single but the confidence limits are substantially wider—33% less
number summarizing the evidence would be misleading, than (0.40) and greater than (0.89) the summary effect,
statistical models that incorporate the extra variability albeit still highly statistically significant. Use of any of the
across studies not believed to be due to chance may be used other 3 random-effects estimators depicted in the Figure
to summarize the data. These models assume that the ob- shows identical point estimates for the odds ratio of 0.60
served treatment effect for a study is a combination of a but dramatically wider confidence limits that are 73% less
treatment effect common to all studies plus a component than and greater than 0.60, with the upper limits all ex-
specific to that study alone. This extra, study-specific com- ceeding 1.00. The corresponding P values range from less
ponent is assumed to be random, hence the jargon that it is than 0.001 for the fixed-effects model to 0.011 for the DL
a “random effect,” with accompanying mathematical mod- estimator and 0.070 or greater for the other random-effects
els dubbed “random-effects models.” The most widely used models.
random-effects model is based on an estimator developed
by DerSimonian and Laird in the mid-1980s and is known
as the DerSimonian–Laird (DL) estimator (1).
STATISTICAL HETEROGENEITY AND UNCERTAINTY
The differences noted in the example are due to the
AN EXAMPLE ways that the models handle statistical heterogeneity. Sta-
The Figure depicts a statistically heterogeneous set of tistical heterogeneity refers to variation in the true effects
studies followed by several methods of estimating their av- being estimated by each study. We characterize this varia-
erage effect. The example is from a 1985 meta-analysis by tion by its SD, a statistic called ␶. Assuming normality, we
Collins and colleagues on the effect of administering a di- expect 95% of true effects to fall within ⫾ 2 ⫻ ␶ of the
uretic to women at risk for preeclampsia (11), and it is central estimate. When odds ratios or relative risks are
© 2014 American College of Physicians 267

Downloaded From: http://annals.org/pdfaccess.ashx?url=/data/journals/aim/929769/ by a University of California San Diego User on 01/13/2017

 Research and Reporting Methods Random-Effects Meta-analysis of Inconsistent Effects

Key Summary Points

THE DL ESTIMATOR AND ALTERNATIVE APPROACHES
The DL method appeared in the literature just as
The decision to calculate a summary estimate in a meta-
analysis should be based on clinical judgment, the number
meta-analytic methods were being adopted to help review-
of studies, and the degree of variation among studies. ers quantitatively summarize evidence about medical inter-
ventions. It was relatively simple to compute and is still the
A random-effects model is a meta-analytic approach that standard estimator programmed into many meta-analysis
incorporates study-to-study variability beyond what would software packages, including the RevMan software devel-
be expected by chance. oped by the Cochrane Collaboration (15). As statisticians
The DerSimonian–Laird (DL) method, the earliest and most began in the 1990s to recognize the problems with the
commonly used random-effects model, is the default DL approach, they—including DerSimonian and Kacker
method in many software packages. (16)—proposed a wide range of alternatives that better
capture the uncertainty associated with statistical heteroge-
The DL method produces confidence bounds that are too neity. These included random-effects estimators based on
narrow (and P values that are typically too small) when small-sample adjustments, such as the Knapp–Hartung ap-
the number of studies is small or when there are substan- proach (17), likelihood-based methods (13, 18, 19), and
tive differences among study estimates. hierarchical Bayesian models (20).
Alternative random-effects estimates based on small- The Knapp–Hartung approach, one of the more re-
sample adjustments, the profile likelihood, or hierarchical cent methods, assumes that variances are estimated from
Bayesian models that perform better than the DL method small samples, makes small-sample adjustments to the vari-
are readily available in software packages. ance estimates, and constructs confidence limits based on
the t distribution with k ⫺ 1 degrees of freedom. This
When it is appropriate to pool studies whose estimates
estimator produces a wider confidence limit than the DL
vary widely, meta-analytic methods that provide a better
estimate. It may slightly overestimate the amount of uncer-
accounting of uncertainty than the DL estimator should
be used.
tainty in some cases, particularly when dealing with 5 or
fewer studies. It is available in some specialized meta-
analysis programs and packages, such as the metareg pro-
gram (21) in Stata (StataCorp, College Station, Texas) and
the metafor package (22) in R (R Foundation for Statistical
used, the normality is on a log scale, so that true study Computing, Vienna, Austria).
Likelihood estimates, which are readily available in
odds ratios or relative risks fall within a range of the esti-
such commonly used statistical packages as SAS (SAS In-
mate multiplied by e⫾2⫻␶. In the example, ␶ equals 0.48,
stitute, Cary, North Carolina), are computed using stan-
so the true study effects are estimated to fall within
dard mixed-effects linear models (18, 19). The profile like-
0.60 ⫻ e⫾0.96, or 2.6 times greater than or less than 0.60
lihood is a good method for computing confidence
(0.23 to 1.56). This range should be smaller than the ac-
bounds. Unlike estimators based on maximum likelihood
tual smallest and largest study estimates, as is the case in
or restricted maximum likelihood methods, the profile
this example, with the remainder of the variation assumed likelihood allows for asymmetrical intervals and uncer-
to be due to chance. tainty in estimation of the between-study variance (␶2).
The models vary in their assumption of how certain Simulation studies show that it provides a substantially
we are about ␶; this uncertainty is included in the meta- better accounting of uncertainty than the DL estimator
analytic CIs. The DL method assumes that our guess about (13, 23). The profile likelihood estimates are available in
␶ is exactly correct, with no uncertainty; thus, confidence the metaan package (24) in Stata and the metaLik package
limits are too narrow and the P values are too small. In (25) in R. The latter provides a more accurate but possibly
Collins and colleagues’ meta-analysis, which pooled a mod- conservative small-sample profile likelihood estimate of un-
est number of studies (n ⫽ 9) with statistically heteroge- certainty (26).
neous effects, the DL estimator provided the narrowest Bayesian random-effects models, which are based on
confidence limits among the random-effects options. an exact binomial distribution, perform well in many situ-
In addition to ␶, meta-analysts commonly use statisti- ations where others do poorly, particularly with sparse data
cal tests, such as the Cochran Q test, or indices, such as the and few studies (27, 28). A hierarchical Bayesian equiva-
I2 index, to help gauge heterogeneity of effects. Both the lent to the mixed-effects model can be fitted using
Cochran Q test and the I2 index are dimensionless mea- WinBugs or related packages (OpenBugs or JAGS). A hi-
sures of statistical heterogeneity. Neither conveys informa- erarchical Bayesian model augmented by careful consider-
tion about actual variation in effect size, and both have low ation of priors on ␶ may provide a better accounting of the
power to detect heterogeneity in situations involving 10 or uncertainty than non-Bayesian approaches, particularly
fewer studies (14). when the number of studies is small (29). Because selection
268 18 February 2014 Annals of Internal Medicine Volume 160 • Number 4 www.annals.org

Downloaded From: http://annals.org/pdfaccess.ashx?url=/data/journals/aim/929769/ by a University of California San Diego User on 01/13/2017

 Random-Effects Meta-analysis of Inconsistent Effects Research and Reporting Methods

Figure. Heterogeneous evidence from Collins and colleagues’ meta-analysis of the effects of diuretics on preeclampsia (11).

Study, Year (Reference) Odds Ratio (95% CI)

Weseley and Douglas, 1962 (2) 1.04 (0.48–2.28)

Flowers et al, 1962 (3) 0.40 (0.20–0.78)
Menzies, 1964 (4) 0.33 (0.14–0.74)
Fallis et al, 1964 (5) 0.23 (0.08–0.67)
Cuadros and Tatum, 1964 (6) 0.25 (0.13–0.48)
Landesman et al, 1965 (7) 0.74 (0.59–0.94)
Kraus et al, 1966 (8) 0.77 (0.39–1.52)
Tervilä and Vartiainen, 1971 (9) 2.97 (0.59–15.07)
Campbell and MacGillivray, 1975 (10) 1.14 (0.69–1.91)

Analysis
Fixed-effects ( = 0)* 0.67 (0.56–0.80)
DerSimonian–Laird ( = 0.48)* 0.60 (0.40–0.89)
Knapp–Hartung ( = 0.48)† 0.60 (0.35–1.03)
Profile likelihood ( = 0.49)‡ 0.60 (0.35–1.04)
Hierarchical Bayesian ( = 0.63)§ 0.60 (0.34–1.08)

0.05 0.25 1.00 4.00 20.00

Odds Ratio (Log Scale)

* The metafor package in R was used to compute the fixed-effects estimate and the DerSimonian–Laird random-effects estimate. † The metafor package
in R was used to compute the Knapp–Hartung small-sample adjustments, based on the DerSimonian–Laird estimate. ‡ The small-sample (Skovgaard)
estimate from the metaLik package in R was used to compute the profile likelihood estimate. The large-sample profile likelihood estimate produced a
narrower CI that indicates a statistically significant effect (95% CI, 0.37 to 0.95). § The hierarchical Bayesian estimate was computed using WinBugs and
assumed a vague uniform (10, 10) prior distribution for ␶. A sensitivity analysis assuming a vague ␥ (0.001, 0.001) on precision (1/␶ 2) produced a
slightly smaller but statistically significant 95% CI (0.36 to 0.98).

of a prior distribution for ␶ or ␶2 is critical to any Bayesian estimate. When there are too few studies to stratify by
analysis (27), it is important to conduct sensitivity analyses study-level characteristics, whether pooling is reasonable
based on different choices for the prior distribution. must be addressed. A critical synthesis that highlights the
variations in the evidence and describes the possible sources
RECOMMENDATIONS FOR MOVING FORWARD of variation will almost always be more useful than one
None of the random-effects methods provide a univer- that averages over these dimensions and can point the way
sal solution to the problem of heterogeneity. The decision toward improvement of future studies.
to summarize data mathematically depends on critical When the decision has been made to pool studies in
judgment, and the reasons for that decision should be ar- the face of heterogeneity, the extra uncertainty due to that
ticulated as part of any meta-analysis. Random-effects esti- heterogeneity must be adequately represented. All of the
mates are most appropriate when it is difficult to attribute alternative approaches to random-effects modeling more
observed heterogeneity of effects to clinical or methodolog- accurately incorporate the uncertainty associated with sta-
ical differences among the studies. Proper selection and tistical heterogeneity than does the DL estimator. With a
implementation of a random-effects model requires careful small number of studies, the Knapp–Hartung or small-
consideration of how many studies are available, the extent sample profile likelihood estimator may be the best choices,
to which estimates vary from study to study (␶), and study- even if they are conservative. The Bayesian methods are
specific clinical and methodological factors that contribute good but require knowledge of Bayesian software and per-
to heterogeneity. Large variation in study design, conduct, form best with informed choice of a prior distribution for
population, measurements, and analyses suggests that it ␶ (that is, the range of plausible values for ␶).
may be unwise to estimate an average effect. When the Insightful synthesis recognizes the qualitative and
number of studies is sufficiently large, organizing ana- quantitative heterogeneity and uncertainty of evidence; fo-
lyses around clinically or methodologically important cuses on describing and explaining the probable sources of
study-level characteristics through stratification or meta- variation in the evidence; and, when summarizing hetero-
regression may be more informative than a single summary geneous evidence quantitatively, uses random-effects esti-
www.annals.org 18 February 2014 Annals of Internal Medicine Volume 160 • Number 4 269

Downloaded From: http://annals.org/pdfaccess.ashx?url=/data/journals/aim/929769/ by a University of California San Diego User on 01/13/2017

 Research and Reporting Methods Random-Effects Meta-analysis of Inconsistent Effects

mates that properly represent statistical uncertainty. The 7. Landesman R, Aguero O, Wilson K, LaRussa R, Campbell W, Penaloza O.
The prophylactic use of chlorthalidone, a sulfonamide diuretic, in pregnancy. Br
original DL estimator from 1986 made it easy to calculate J Obstet Gynaecol. 1965;72:1004-10.
random-effects estimates with computers or spreadsheets, 8. Kraus GW, Marchese JR, Yen SS. Prophylactic use of hydrochlorothiazide in
leading to its rapid adoption and incorporation into meta- pregnancy. JAMA. 1966;198:1150-4. [PMID: 5332983]
analytic software. However, after more than 25 years of 9. Tervilä L, Vartiainen E. The effects and side effects of diuretics in the pro-
phylaxis of toxaemia of pregnancy. Acta Obstet Gynecol Scand. 1971;50:351-6.
improvement in methods and software, it is time to move [PMID: 4945572]
forward and use random-effects methods that provide a 10. Campbell DM, MacGillivray I. The effect of a low calorie diet or a thiazide
more adequate accounting of uncertainty in estimating an diuretic on the incidence of pre-eclampsia and on birth weight. Br J Obstet
Gynaecol. 1975;82:572-7. [PMID: 1096930]
average effect when heterogeneity is present. 11. Collins R, Yusuf S, Peto R. Overview of randomised trials of diuretics in
pregnancy. Br Med J (Clin Res Ed). 1985;290:17-23. [PMID: 3917318]
From University of Texas Health Science Center at San Antonio, San 12. Biggerstaff BJ, Tweedie RL. Incorporating variability in estimates of heter-
Antonio, Texas; University of Pennsylvania and American College of ogeneity in the random effects model in meta-analysis. Stat Med. 1997;16:753-
Physicians, Philadelphia, Pennsylvania; Johns Hopkins School of Public 68. [PMID: 9131763]
Health, Baltimore, Maryland; and Stanford University School of Medi- 13. Hardy RJ, Thompson SG. A likelihood approach to meta-analysis with
cine, Stanford, California. random effects. Stat Med. 1996;15:619-29. [PMID: 8731004]
14. Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis.
Stat Med. 2002;21:1539-58. [PMID: 12111919]
Potential Conflicts of Interest: Disclosures can be viewed at www 15. Review Manger (RevMan) [computer program]. Version 5.2. Copenhagen,
.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum⫽M13 Denmark: Nordic Cochrane Center, Cochrane Collaboration; 2012.
16. DerSimonian R, Kacker R. Random-effects model for meta-analysis of clin-
-2886.
ical trials: an update. Contemp Clin Trials. 2007;28:105-14. [PMID: 16807131]
17. Knapp G, Hartung J. Improved tests for a random effects meta-regression
Requests for Single Reprints: John E. Cornell, PhD, Department of with a single covariate. Stat Med. 2003;22:2693-710. [PMID: 12939780]
18. Normand SL. Meta-analysis: formulating, evaluating, combining, and report-
Epidemiology and Biostatistics, University of Texas Health Science Cen-
ing. Stat Med. 1999;18:321-59. [PMID: 10070677]
ter at San Antonio, 7703 Merton Minter Boulevard, San Antonio, TX 19. Whitehead A. Meta-analysis of Controlled Clinical Trials. New York: J Wi-
78229; e-mail, cornell@uthscsa.edu. ley; 2002.
20. Higgins JP, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-
effects meta-analysis. J R Stat Soc Ser A Stat Soc. 2009;172:137-59. [PMID:
Current author addresses and author contributions are available at 19381330]
www.annals.org. 21. Sterne JAC. Meta-analysis in Stata: An Updated Collection from the Stata
Journal. 1st ed. College Station, Texas: Stata Pr; 2009.
22. Viechtbauer W. Conducting meta-analysis in R with the metafor package.
J Stat Softw. 2010;36:1-48.
References 23. Brockwell SE, Gordon IR. A comparison of statistical methods for meta-
1. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. analysis. Stat Med. 2001;20:825-40. [PMID: 11252006]
1986;7:177-88. [PMID: 3802833] 24. Kontopantelis E, Reeves D. metaan: Random-effects meta-analysis. Stata J.
2. Weseley AC, Douglas GW. Continuous use of chlorothiazide for prevention 2010;10:395-407.
of toxemia of pregnancy. Obstet Gynecol. 1962;19:355-8. [PMID: 14006267] 25. Guolo A, Varin C. The R package metaLik for likelihood inference in meta-
3. Flowers CE Jr, Grizzle JE, Easterling WE, Bonner OB. Chlorothiazide as a analysis. J Stat Softw. 2012;50:1-14.
prophylaxis against toxemia of pregnancy. A double-blind study. Am J Obstet 26. Guolo A. Higher-order likelihood inference in meta-analysis and meta-
Gynecol. 1962;84:919-29. [PMID: 13893680] regression. Stat Med. 2012;31:313-27. [PMID: 22173666]
4. Menzies DN. Controlled trial of chlorothiazide in treatment of early pre- 27. Warn DE, Thompson SG, Spiegelhalter DJ. Bayesian random effects meta-
eclampsia. Br Med J. 1964;1:739-42. [PMID: 14102017] analysis of trials with binary outcomes: methods for the absolute risk difference
5. Fallis NE, Plauche WC, Mosey LM, Langford HG. Thiazide versus placebo and relative risk scales. Stat Med. 2002;21:1601-23. [PMID: 12111922]
in prophylaxis of toxemia of pregnancy in primigravid patients. Am J Obstet 28. Lambert PC, Sutton AJ, Burton PR, Abrams KR, Jones DR. How vague is
Gynecol. 1964;88:502-4. [PMID: 14123429] vague? A simulation study of the impact of the use of vague prior distributions in
6. Cuadros A, Tatum HJ. The prophylactic and therapeutic use of bendroflu- MCMC using WinBUGS. Stat Med. 2005;24:2401-28. [PMID: 16015676]
methiazide in pregnancy. Am J Obstet Gynecol. 1964;89:891-7. [PMID: 29. Higgins JP, Whitehead A. Borrowing strength from external trials in a meta-
14207556] analysis. Stat Med. 1996;15:2733-49. [PMID: 8981683]

270 18 February 2014 Annals of Internal Medicine Volume 160 • Number 4 www.annals.org

Downloaded From: http://annals.org/pdfaccess.ashx?url=/data/journals/aim/929769/ by a University of California San Diego User on 01/13/2017

 Annals of Internal Medicine
Current Author Addresses: Dr. Cornell: Department of Epidemiology Author Contributions: Conception and design: J.E. Cornell, C.D.
and Biostatistics, University of Texas Health Science Center at San An- Mulrow, R. Localio, S.N. Goodman.
tonio, 7703 Merton Minter Boulevard, San Antonio, TX 78229. Analysis and interpretation of the data: J.E. Cornell, R. Localio.
Dr. Mulrow: University of Texas Health Science Center at San Antonio, Drafting of the article: J.E. Cornell, C.D. Mulrow, R. Localio, C.B.
7703 Floyd Curl Drive, San Antonio, TX 78229. Stack, A.R. Meibohm, S.N. Goodman.
Dr. Localio: Center for Clinical Epidemiology and Biostatistics, Univer- Critical revision of the article for important intellectual content: J.E.
sity of Pennsylvania, 635 Blockley Hall, 423 Guardian Drive, Philadel- Cornell, C.D. Mulrow, R. Localio, C.B. Stack, A.R. Meibohm, E. Gual-
phia, PA 19104-6021. lar, S.N. Goodman.
Drs. Stack and Meibohm: American College of Physicians, 190 N. In- Final approval of the article: J.E. Cornell, C.D. Mulrow, R. Localio,
dependence Mall West, Philadelphia, PA 19106.
C.B. Stack, A.R. Meibohm, E. Guallar, S.N. Goodman.
Dr. Guallar: Welch Center for Prevention, Epidemiology and Clinical
Statistical expertise: J.E. Cornell, R. Localio, A.R. Meibohm, S.N. Good-
Research, Johns Hopkins Medical Institutions, 2024 East Monument
man.
Street, Room 2-645, Baltimore, MD 21287.
Administrative, technical, or logistic support: C.D. Mulrow.
Dr. Goodman: Stanford University School of Medicine, 259 Campus
Drive, T265 Redwood Building/HRP, Stanford, CA 94305. Collection and assembly of data: J.E. Cornell.

www.annals.org 18 February 2014 Annals of Internal Medicine Volume 160 • Number 4

Downloaded From: http://annals.org/pdfaccess.ashx?url=/data/journals/aim/929769/ by a University of California San Diego User on 01/13/2017