PUBLICATION BIAS IN PSYCHOLOGY
Meta-Analyses in Psychology Often Overestimate
Evidence for and Size of Effects
František Bartoš1∗ , Maximilian Maier2∗ , David R. Shanks2 , T.D. Stanley3,4 , Martina
Sladekova5 , Eric-Jan Wagenmakers1
* Both authors contributed equally
1 Department of Psychological Methods, University of Amsterdam
2 Department of Experimental Psychology, University College London
3 Deakin Laboratory for the Meta-Analysis of Research (DeLMAR), Deakin University
4 Department of Economics, School of Business, Deakin University
5 School of Psychology, University of Sussex
Author Note
Correspondence concerning this article should be addressed to František Bartoš,
Department of Psychological Methods, University of Amsterdam E-mail:
f.bartos96@gmail.com
1
PUBLICATION BIAS IN PSYCHOLOGY
2
Abstract
Adjusting for publication bias is essential when drawing meta-analytic inferences. However,
most methods that adjust for publication bias are sensitive to the particular research
conditions, such as the degree of heterogeneity in effect sizes across studies. Sladekova
et al. (2022) tried to circumvent this complication by selecting the methods that are most
appropriate for a given set of conditions, and concluded that publication bias on average
causes only minimal over-estimation of effect sizes in psychology. However, this approach
suffers from a Şcatch-22Ť problem Ů to know the underlying research conditions, one needs
to have adjusted for publication bias correctly, but to correctly adjust for publication bias,
one needs to know the underlying research conditions. To alleviate this problem we
conduct an alternative analysis, Robust Bayesian meta-analysis (RoBMA), which is not
based on model-selection but on model-averaging. In RoBMA, models that predict the
observed results better are given correspondingly higher weights. A RoBMA reanalysis of
Sladekova et al.’s data reveals that more than 60% of meta-analyses in psychology notably
overestimate the evidence for the presence of the meta-analytic effect and more than 50%
overestimate its magnitude. Our results highlight the need for robust bias correction when
conducting meta-analyses and for the adoption of publishing formats such as Registered
Reports that are less prone to publication bias.
Keywords: publication bias, RoBMA, model-selection, model-averaging, Bayesian
inference
PUBLICATION BIAS IN PSYCHOLOGY
3
Meta-Analyses in Psychology Often Overestimate
Evidence for and Size of Effects
Meta-analysis is widely regarded as the best way to combine and summarize
seemingly conĆicting evidence across a set of primary studies. However, publication bias Ů
the preferential publishing of statistically signiĄcant studies Ů often causes meta-analyses
to overestimate mean effect sizes (Borenstein et al., 2009; Rosenthal & Gaito, 1964; Vevea
& Hedges, 1995). Therefore, a key question concerns the extent to which meta-analytic
estimates represent reliable indicators even when publication bias is left unaccounted for.
To address this question, Sladekova, Webb, and Field (2022; henceforth SWF) compiled an
extensive data set of 433 meta-analyses from the Ąeld of psychology and assessed the
typical overestimation of effect sizes using methodologically advanced techniques and a
model-selection procedure recently developed by Carter et al. (2019). SWF concluded that
on average, effect size estimates were only marginally lower after accounting for publication
bias. The most aggressive average adjustment was provided by precision effect test (PET)
models, ∆r = −0.032, 95% CI [−0.055, −0.009]); moreover, meta-analyses comprised of few
studies often exhibited an anomalous upward adjustment.
In their analyses, SWF speciĄed four plausible data-generating processes and
selected the best estimator for each based on the Ąndings of a simulation study by Carter
et al. (2019). As different publication bias adjustment methods are generally found to
perform well under different conditions, Carter et al. (2019) provided code that allows
researchers to select the most suitable publication bias correction method based on speciĄc
assumed research conditions, such as the true degree of heterogeneity in the effect sizes
included in the meta-analysis. Whereas this approach presents a substantial improvement
over the common practice of applying bias correction methods with little regard to the
observed meta-analytic conditions, we believe that in empirical practice it is nigh
impossible to execute as intended. The reason is that critical research characteristics (i.e.,
the true effect size, heterogeneity, and degree and type of publication bias) cannot be
PUBLICATION BIAS IN PSYCHOLOGY
4
accurately estimated unless one Ąrst adjusts for publication bias. Alternatively, specifying
multiple conditions might result in different estimates, leaving the analyst with
incompatible conclusions. Therefore, the approach by Carter et al. (2019), as employed by
SWF, creates a catch-22 problem (Heller, 1961): to correctly adjust for publication bias,
one needs to know the underlying research conditions; however, in order to know the
underlying research conditions, one needs to have adjusted correctly for publication bias.1
A second challenge for the Şselect-the-best-estimatorŤ approach is that the Carter
et al. (2019) simulation is based on speciĄc assumptions about the data generating process.
As with all simulations, the question is how well the data generating process actually
corresponds to publication bias as it operates in the real world (Stanley et al., 2021). In
their discussion, SWF point out that an alternative solution is provided by Bayesian
model-averaging (Hinne et al., 2020; Hoeting et al., 1999). Bayesian model-averaging (e.g.,
robust Bayesian meta-analysis or RoBMA; Bartoš et al., in press; Bartoš, Maier,
Wagenmakers, et al., 2021; Maier et al., 2022) simultaneously considers an entire ensemble
of models for publication selection and potential research conditions. The data then guides
the inference to be based most strongly on those models that best predict the observed
research results. In this way the Bayesian model-averaging of publication bias models
alleviates the catch-22 problem outlined above. SWF discuss how RoBMA would be a
good alternative approach; here, we follow SWF’s suggestion and re-analyse their dataset
with RoBMA. To preview, a very different (and, we argue, more credible) conclusion
emerges from this re-analysis.
A third challenge for the for the Şselect-the-best-estimatorŤ approach is that
investigations based on empirical data show that the speciĄc correction methods employed
1
The type and degree of publication bias as well as the true effect size is generally unknown, a problem
which SWF sidestepped by calculating four possible models of publication bias and effect size. Moreover,
the random-effects’ heterogeneity estimates that are required to select the best method depends on the
degree and type of publication bias (Augusteijn et al., 2019; Hönekopp & Linden, 2022)
PUBLICATION BIAS IN PSYCHOLOGY
5
by SWF do not adjust for publication bias sufficiently. In particular, Kvarven et al. (2020)
compared estimates from publication bias-adjusted meta-analyses to Registered Replication
Reports on the same topic (Chambers, 2013; Chambers et al., 2015). Registered Reports
are a publication format in which a submitted manuscript receives peer review and Şin
principleŤ acceptance based on the introduction and methods section alone. Hence the
journal commits itself to publishing the report independent of the outcome, as long as the
data pass pre-speciĄed outcome-neutral quality checks. Therefore, Registered Reports are
not affected by publication bias and can be considered the Şgold standardŤ of evidence.
Consequently, a publication bias adjustment method that works well ought to produce an
effect size estimate that is similar to the one from a Registered Report on the same topic.
By comparing Registered Reports to associated meta-analyses, Kvarven et al. (2020)
showed that the publication bias correction methods employed in SWF lead to substantial
overestimation of effect size and under-estimation of the required correction (but see Lewis
et al. (2020) for a criticism of this approach, which argues that the difference might partly
be explained by genuine effect heterogeneity rather than publication bias). In contrast,
Bartoš, Maier, Wagenmakers, et al. (2021) demonstrated that RoBMA results in estimates
that are less biased and have considerably lower root mean square errors.
Finally, in their work SWF focus solely on the impact of publication bias
adjustment on meta-analytic effect size. In practice, researchers also wish to know whether
there is a genuine effect in the Ąrst place (Jeffreys, 1961; Jeffreys, 1973). A Bayesian
analysis allows us to quantify the evidence for a non-null effect and assess its posterior
probability, while circumventing problems of frequentist signiĄcance testing (e.g.,
Wagenmakers, 2007; Wagenmakers et al., 2016).
In sum, by applying multiple models to the data simultaneously, RoBMA avoids the
catch-22 problem that plagues the Şselect-the-best-estimatorŤ approach. Moreover,
RoBMA does not underadjust for publication bias (Bartoš, Maier, Wagenmakers, et al.,
2021), and offers a Bayesian way to quantify how much publication bias inĆates the
PUBLICATION BIAS IN PSYCHOLOGY
6
evidence for the presence of an overall effect.
In the next sections, we apply RoBMA to the meta-analysis dataset compiled by
SWF. The RoBMA re-analysis shows that many meta-analyses suffer from publication bias
in the sense that both the effect size and the evidence for the presence of the effect are
substantially overestimated (52.7% and 60.8%, respectively).
Method
The RoBMA Model Ensemble
Here we describe how we employed the robust Bayesian model-averaging
methodology. The remaining publication bias adjustment methods used in SWF are
explained in more detail therein and in Carter et al. (2019).
The complete RoBMA-PSMA model ensemble (as implemented in Bartoš, Maier,
Wagenmakers, et al., 2021; simply referred below as RoBMA) employs models that can be
categorized along three research dimensions: presence vs. absence of the effect,
heterogeneity across reported effects, and publication selection bias. Each of these
hypotheses is assigned a prior model probability of 1/2, reĆecting a position of equipoise.
The individual models speciĄed within the RoBMA ensemble then represent a combination
of these research characteristics with prior model probabilities corresponding to the
product of prior probabilities of each corresponding hypothesis. For models representing
the presence of publication bias, the prior model probability is equally split amongst the
various selection models and PET-PEESE, then further split equally among the different
selection models or between PET and PEESE. The complete RoBMA-PSMA ensemble
consists of 36 different models.
The hypothesis about the absence of the effect is represented by a point prior
distribution on the effect size at 0, µ = 0, and the hypothesis about the presence of the
effect is represented by a standard normal prior distribution on Cohen’s d effect size,
µ ∼ Normal(0, 1), representing the most common range of effect sizes in psychology. The
PUBLICATION BIAS IN PSYCHOLOGY
7
hypothesis about the absence of heterogeneity is represented by a point prior distribution
on the heterogeneity at 0, τ = 0, and the hypothesis about the presence of heterogeneity is
represented by an inverse-gamma distribution, τ ∼ Inverse-Gamma(1, 0.15) (with scale and
shape parameterization; corresponding to Cohen’s d effect sizes), based on empirical
heterogeneity estimates from the Ąeld of psychology (van Erp et al., 2017). The hypothesis
about the absence of publication bias is instantiated by not applying any publication bias
corrections, and the hypothesis about the presence of publication bias is instantiated by
applying a set of six weight functions (Larose & Dey, 1998; Maier et al., 2021; Vevea &
Hedges, 1995), and both the PET and PEESE models (Stanley et al., 2017) to adjust for
publication bias. The weight functions are speciĄed as a combination of cutoffs on
signiĄcant, marginally signiĄcant p-values and the direction of the effect. The cumulative
unit Dirichlet prior distributions enforce a decreasing relative prior probability with
increasing p-values which further helps with performance of selection models. The PET
and PEESE models are speciĄed as meta-regressions of the effect sizes on the standard
errors or standard errors squared with a truncated Cauchy distributions on the PET and
PEESE regression coefficients, PET ∼ Cauchy+ (0, 1), PEESE ∼ Cauchy+ (0, 5), which
enforce a positive relationship between standard errors and effect sizes. More details on the
RoBMA speciĄcation are presented in Bartoš, Maier, Wagenmakers, et al. (2021).
The performance of RoBMA has been evaluated extensively in simulation studies as
well as empirical comparisons. In particular, Bartoš, Maier, Wagenmakers, et al. (2021)
reanalysed a large simulation study by Hong and Reed (n.d.), which itself combined four
different previous simulation environments comprising 1,640 separate experimental
conditions (Alinaghi & Reed, 2018; Bom & Rachinger, 2019; Carter et al., 2019; Stanley
et al., 2017). In these simulations, RoBMA outperformed other methods for publication
bias correction in terms of bias and root mean squared error. RoBMA was also evaluated
empirically by comparing meta-analyses that are linked to Registered Replication Reports
in Kvarven et al. (2020). As discussed above, comparing meta-analysis bias corrections to a
PUBLICATION BIAS IN PSYCHOLOGY
8
Şground truthŤ as revealed by Registered Reports allows us to evaluate whether a given
correction sufficiently adjusts for likely publication bias. In the Kvarven et al. (2020)
comparison of meta-analyses and Registered Reports, RoBMA was shown to provide the
best adjustment for publication bias when evaluated by average bias and/or root mean
square error by Bartoš, Maier, Wagenmakers, et al. (2021). Nonetheless, RoBMA and
Bayesian model-averaging are only as good as the models incorporated in the ensemble.
Since none of the meta-analytic models employed in RoBMA directly adjusts for p-hacking,
RoBMA can exhibit downward bias in cases with strong p-hacking (Bartoš, Maier,
Wagenmakers, et al., 2021).
Effect Size Transformation
In contrast to SWF, we analyzed the effect sizes using the Fisher z scale (and
subsequently transformed the meta-analytic estimates back to the correlation scale for
interpretation). We prefer the Fisher z scale for two reasons. First, it is unbounded (i.e.,
not restricted to the (−1, 1) interval) and the sampling distribution is approximately
normal, which corresponds to the likelihoods used by meta-analytic models (this also
prevents adjusted meta-analytic correlation estimates falling outside of (−1, 1), which is
anomalous).2 Second, the Fisher z score and its standard error are by deĄnition
orthogonal, which is an important assumption for models adjusting for the relationship
between effect sizes and standard errors as in PET-PEESE (this was not an issue in SWF
as they used standard errors of Fisher’s z alongside the correlation effect sizes).
The use of the Fisher’s z scale results in slight differences in (a) selected methods
for each condition (as the reduced range of correlation effect sizes limits the possible
heterogeneity), and (b) effect sizes estimates of those selected methods. These differences,
however, do not change the qualitative conclusions.
2
Use of the Fisher z transformation did necessitate the removal of 51 reported correlation coefficients
equal to 1 from 4 meta-analyses.
PUBLICATION BIAS IN PSYCHOLOGY
9
Effects of Publication Bias on Evidence and on Effect Size
We extended the SWF results by Ąrst assessing the extent to which publication bias
inĆates the evidence for the presence of an effect. Then, similarly to SWF, we also
evaluated and compared the effect of publication bias on the meta-analytic estimates of
effect size.
To evaluate the change in evidence for the presence of the effect, we compared the
posterior probability for the presence of the effect under RoBMA to the posterior
probability for the presence of the effect under RoBMA after excluding the models that
adjust for publication bias. The publication bias unadjusted version of RoBMA
corresponds to the Bayesian model-averaged meta-analysis (BMA; e.g., Bartoš, Gronau,
et al., 2021; Gronau & Wagenmakers, 2018; Gronau et al., 2017). For both RoBMA and
BMA, the prior model probability for the presence of the effect is set to 1/2. Furthermore,
we summarize the results as the change in the percentage of meta-analyses that provide at
least moderate or strong evidence for either the null or alternative hypothesis based on the
Şrule of thumbŤ Bayes factor categories that have been proposed to facilitate the
interpretation of Bayes factors (i.e., BF > 3 is moderate evidence, and BF > 10 is strong
evidence; Jeffreys, 1939; Lee & Wagenmakers, 2013).
To evaluate the change in the meta-analytic estimate of effect size, we compared the
model-averaged posterior mean obtained from RoBMA to effect size estimates from two
meta-analytic methods that do not adjust for publication bias. The Ąrst comparison is to a
random effects meta-analysis (reMA) which is regarded as the default meta-analytic
method in behavioral research. The comparison of reMA and RoBMA estimates therefore
quantiĄes the reduction in effect size that obtains when researchers use RoBMA instead of
the standard methodology. The second comparison is to a different version of Bayesian
model-averaged meta-analysis (BMA; Bartoš, Gronau, et al., 2021; Gronau et al., 2021;
Gronau et al., 2017) that is identical to RoBMA apart from the fact that BMA lacks the
models that adjust for publication bias; consequently, the comparison of BMA and RoBMA
PUBLICATION BIAS IN PSYCHOLOGY
10
estimates quantiĄes the reduction in effect size that can be attributed solely to publication
bias adjustment.
Finally, we compare the effect size adjustments due to RoBMA against the
adjustments due to the methods presented by SWF. We employ the same Bayesian
hierarchical models as SWF to estimate the mean publication bias adjustment, for SWF’s
model selection and RoBMA separately. When analyzing results from the SWFs model
selection, we combined 3PSM and 4PSM into a single category (PSM) and only estimated
Ąxed effects for methods that were selected at least 20 times.
We performed the analysis in R R Core Team, 2021 using the RoBMA R package
Bartoš and Maier, 2020 and additional R functions adopted from SWF and Carter et al.
(2019). The analysis scripts and results are available at https://osf.io/7yzut/.
Results
Evidence for the Presence of the Effect
First, we used RoBMA to evaluate inĆation of the posterior probability for the
presence of the effect. Figure 1 shows the estimates for the presence of an effect before
(x-axis) and after (y-axis) the publication bias adjustment. The dotted diagonal line
highlights the points of no change in posterior probability of the alternative hypotheses due
to publication bias. For many meta-analyses the evidence for the presence of an effect is
considerably lower after adjusting for publication bias, which is further exempliĄed by the
marginal densities of the posterior probabilities on the right and top sides of the Ągure.
Across all meta-analyses, the median posterior probability drops from 0.97, interquartile
range (IRQ; 0.44, 1.00), to 0.53, IQR (0.26, 0.91), indicating considerable inĆation of
evidence due to publication bias. Nevertheless, for 39.2% of the meta-analyses the posterior
probability for the presence of the effect did not change by more than 0.05, indicating that
a notable proportion of psychology meta-analyses are relatively robust to publication bias.
Furthermore, the percentage of meta-analyses providing strong or at least moderate
PUBLICATION BIAS IN PSYCHOLOGY
11
Figure 1
Posterior probability for the presence of the effect from the publication bias adjusted vs.
unadjusted models.
1.0
Adjusted P(H1 | Data)
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Unadjusted P(H1 | Data)
Note. Some meta-analyses show more evidence for an effect after publication bias adjustment. This anomaly occurs when the correction methodology adjusts a nearly zero effect
size (showing evidence for the null hypothesis) further down to be slightly negative. This
increases the evidence for an effect but in the opposite direction compared to the estimate
of the original meta-analysis.
evidence for the alternative hypothesis (i.e., BF10 > 10 and BF10 > 3) decreased from
55.7% to 24.9% and from 64.3% to 36.9%, respectively. Interestingly, the proportion of
meta-analyses providing strong or at least moderate evidence for the null hypothesis (i.e.,
BF10 < 1/10 and BF10 < 1/3) increased only marginally, from 4.7% and 5.2% and from
PUBLICATION BIAS IN PSYCHOLOGY
12
18.5% to 23.9%, respectively 3 . Most of the change in evidence was due to the increase in
the Śundecided’ evidence category, (i.e., 1/3 > BF10 > 3), from 17.2% to 39.2%.
Effect Size Estimates
In addition to the impact on the posterior probability for the presence of the effect,
we can also quantify the degree to which publication bias impacts the effect size estimates.
Figure 2 and Figure 3 show the impact of adjusting for publication bias on the
meta-analytic estimates. The dotted diagonal lines in Figure 2 highlight the points of no
change in the effect size estimates due to publication bias. After adjusting for publication
bias, many estimates are considerably lower. SpeciĄcally, the publication bias unadjusted
meta-analytic effect sizes corresponded mostly to small to medium sized effects based on
random effects meta-analyses r = 0.17, IQR (0.09, 0.30), and BMA r = 0.15, IQR
(0.04, 0.28). However, the publication bias adjustment provided by RoBMA reduced the
estimates to predominantly small sized effects (i.e., r = 0.07, IQR (0.01, 0.22)).
Whereas the distributions of the publication bias unadjusted and adjusted effect
size estimates were notably different, the distribution of differences between the estimates
was highly skewed with many meta-analyses undergoing only small publication bias
adjustments (right panel of Figure 4). The median adjustment from random effect
meta-analyses to RoBMA was r = −0.07, IQR (−0.12, −0.04), and the median adjustment
from BMA to RoBMA was r = −0.04, IQR (−0.08, −0.01). Interestingly, the comparison
of BMA and RoBMA, quantifying the adjustment attributable only to the publication bias
adjustment part, revealed that 47.3% of meta-analytic effect size estimates are adjusted by
less than r = 0.03,4 again indicating that not all meta-analytic estimates are distorted by
3
This can be partly explained by the difficulty of Ąnding evidence for the null using priors centered on zero
as in RoBMA (and most other applications of Bayesian testing). We still chose to use these priors, as they
have other desirable properties and they have been evaluated extensively in applied examples and
simulation studies (Bartoš, Maier, Wagenmakers, et al., 2021).
4
Other possible ŤthresholdsŞ would results in 24.1% with r = 0.01, 37.7% with r = 0.02, 54.4% with
PUBLICATION BIAS IN PSYCHOLOGY
13
Figure 2
RoBMA effect size (correlations)
Effect size estimates from the publication bias adjusted vs. unadjusted models.
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
−0.2
−0.2
−0.2
0.0
0.2
0.4
0.6
0.8
reMA effect size (correlations)
1.0
−0.2
0.0
0.2
0.4
0.6
0.8
1.0
BMA effect size (correlations)
Note. Model-averaged posterior mean effect size estimates based on RoBMA (y-axis) vs.
mean effect size estimates based on random effects meta-analysis (x-axis, left panel) and
model-averaged posterior mean effect size estimates based on BMA (x-axis, right panel).
One outlier adjusted to −0.46 (from 0.15 with BMA and 0.16 with reMA) is omitted from
the display.)
publication bias.
Van Aert et al. (2019) argue that meta-analyses with low heterogeneity show little
evidence of publication bias. To assess the impact of heterogeneity, we conducted an
exploratory regression analysis predicting the effect size adjustment attributable to
publication bias from heterogeneity with the unadjusted effect size estimate as a covariate
(to account for the fact that meta-analyses with larger effect sizes on average show larger
absolute bias and larger τ ). We found that publication bias and heterogeneity were indeed
associated, BF10 = 8.96 × 108 , b = −0.20 95% CI [−0.26, −0.15]. Contrary to the
r = 0.04, and 63.5% with r = 0.05
PUBLICATION BIAS IN PSYCHOLOGY
14
Figure 3
Comparison of densities of the effect size estimates from the publication bias adjusted vs.
unadjusted models.
RoBMA − Random effects
RoBMA − BMA
Density
Random effects
BMA
RoBMA
−0.2
0.0
0.2
0.4
0.6
Effect size (correlations)
0.8
1.0
−0.4
−0.3
−0.2
−0.1
0.0
0.1
0.2
Effect size change (correlations)
Note. Densities of meta-analytic estimates under each method (left) and densities of differences between the unadjusted and adjusted estimates (right). One outlier adjusted to −0.46
is omitted from the display.
conclusions of van Aert et al. (2019), we obtain moderate evidence in favor of the effect size
overestimation even amongst homogeneous studies (i.e., no heterogeneity, tested via the
coefficient for intercept), BF10 = 8.66, Fisher’s z = −0.02, 95% CI [−0.03, 0.00]. However,
the effect size adjustment in homogeneous meta-analyses is much smaller than the average
effect size adjustment across all meta-analyses (Fisher’s z = −0.06, 95% CI
[−0.07, −0.05]).5
5
We calculated estimates and CIs using the a standard linear model in R and corresponding Bayes factors
using a normal approximation (e.g., Bartoš & Wagenmakers, 2022; Dienes, 2014; Spiegelhalter et al., 2004)
specifying centered normal prior distributions with standard deviation of 0.6. This prior corresponds to a
standard deviation of 0.3 on Cohen’s d scale, which tests for small effects.
PUBLICATION BIAS IN PSYCHOLOGY
15
Comparison to Results from SWF
We compared the effect size adjustment based on RoBMA to the adjustment based
on the model selected under different assumptions about the incidence of publication bias
and effect size called ŞModel 1Ť though ŞModel 4Ť by SWF. In short, ŞModel 1Ť speciĄed
the presence of moderate publication bias and small effect sizes, ŞModel 2Ť speciĄed the
presence of high publication bias and small effect sizes, ŞModel 3Ť speciĄed the presence of
moderate publication bias and large effect sizes, and ŞModel 4Ť speciĄed the presence of
high publication bias and large effect sizes.
Figure 4
Comparison of the publication bias adjustment generated by RoBMA and the remaining
methods under different modes of publication bias as designed by Sladekova et al. (2022)
1.00
WAAP−WLS
PET
PEESE
PET−PEESE
PSM
0.75
Effect size adjustment (r)
0.50
0.25
0.00
−0.25
−0.50
−0.75
−1.00
RoBMA
vs. reMA
RoBMA
vs. BMA
Model 1
Model 2
Model 3
Model 4
Figure 4 compares the effect size adjustments in the individual studies by RoBMA
PUBLICATION BIAS IN PSYCHOLOGY
16
and under the different models of SWF. The most noticeable difference between the effect
size adjustments is that RoBMA did not correct any of the effect size estimates in the
opposite direction. As reported before, the median effect size adjustment of RoBMA,
r = −0.07, IQR (−0.12, −0.04), and r = −0.04, IQR (−0.08, −0.01), when comparing to
random effects meta-analysis and BMA, was larger than adjustments of the other methods,
0.00 IQR (−0.04, 0.03), 0.00 IQR (−0.08, 0.05), 0.00 IQR (−0.04, 0.03), 0.00 IQR
(−0.06, 0.03) for Models 1, 2, 3, and 4, respectively. The proportion of meta-analyses where
RoBMA (compared to BMA) adjusted by less than r = 0.03 (reported earlier, 47.3%) was
higher than in three out of four Models; 44.8%, 30.3%, 50.0%, and 41.9%, for Models 1, 2,
3, and 4, respectively (but not for adjustment from random effects meta-analysis to
RoBMA, 23.2%). In other words, while RoBMA, on average, adjusted effect sizes more
aggressively than other methods, it targeted the adjustment to a lower proportion of
meta-analyses than the remaining methods.
Furthermore, the RoBMA effect size adjustments were less dispersed than the effect
size adjustments under the different models of publication bias; even when considering the
spread of the absolute values of the effect size adjustments, the standard deviation of the
absolute values of the effect size adjustments by RoBMA was 0.071 vs. 0.131, 0.142, 0.133,
and 0.136 under Models 1, 2, 3, and 4, respectively (and 0.076 for adjustment from random
effects meta-analysis to RoBMA). This further suggests that RoBMA is sensitive to the
speciĄc research conditions seen in each individual meta-analysis and adjusts accordingly.
Finally, we estimated a three-level Bayesian model describing the effects of different
publication bias adjustments with the same speciĄcation as in (Sladekova et al., 2022)
(combining the 3 and 4PSM category into PSM) for RoBMA and each Model separately.
Figure 5 compares the Ąxed effect estimates of the different methods under the different
Models. The Ąxed effect estimate of the RoBMA adjustment, βRoBMA = −0.04, 95% CI
[−0.05, −0.03] is notably more negative than the adjustments of the remaining methods
under the Model 1: βPEESE = −0.01, 95% CI [−0.03, 0.02], βPSM = 0.00, 95% CI
PUBLICATION BIAS IN PSYCHOLOGY
17
Figure 5
Comparison of the publication bias adjustment performed by RoBMA and the remaining
methods under different models of publication bias, as constructed by SWF
Model 1
Model 2
Model 3
Model 4
RoBMA
WAAP−WLS
PEESE
PET
PSM
−0.100 −0.075 −0.050 −0.025
0.000
0.025
0.050
0.075
0.100
Change in effect size (r)
[−0.01, 0.01], Model 2: βPEESE = 0.00, 95% CI [−0.05, 0.04], βPET = −0.01, 95% CI
[−0.03, 0.02], βPSM = 0.00, 95% CI [−0.02, 0.02], Model 3: βPSM = 0.00, 95% CI
[−0.01, 0.01], βWAAP-WLS = 0.00, 95% CI [−0.01, 0.01], or Model 4: βPEESE = 0.00, 95% CI
[−0.02, 0.01], βPSM = −0.02, 95% CI [−0.04, −0.01], βWAAP-WLS = 0.00, 95% CI
[−0.01, 0.02].
Concluding Comments
It is widely accepted that different meta-analysis methods perform well under
different conditions. Hence it can be risky to employ a single method to estimate the
extent to which meta-analyses in general over-estimate effect sizes. SWF attempted to
PUBLICATION BIAS IN PSYCHOLOGY
18
circumvent this complication by selecting different adjustment methods for four plausible
conditions based on heterogeneity estimates indicated by a naive random effect
meta-analysis. Their article was a much needed contribution to the bias adjustment
literature, being the Ąrst comprehensive review that tried to select estimators appropriate
for different data generating scenarios on an impressively large and representative dataset.
Here we outlined an alternative approach based on Bayesian model-averaging. Rather than
selecting a single model for each case and assumed data generating process, our robust
Bayesian meta-analysis simultaneously considers multiple models, with their contribution
to the meta-analytic inference determined by their predictive accuracy.
The difference is not a point of methodological pedantry but has a considerable
impact on the conclusions regarding the necessary degree of publication bias adjustment.
Whereas SWF found little overestimation of effect sizes due to publication bias, similarly
to van Aert et al. (2019), and for some methods, even larger effects after adjustment,
RoBMA often corrects more strongly and reveals the presence of notable bias. In addition,
RoBMA also allowed us to assess the amount of spurious evidence, indicating that evidence
for meta-analytic effect sizes is considerably weaker after publication bias is accounted for.
The difference between the effect size correction provided by RoBMA and the
remaining methods cannot be solely attributed to a downward bias of RoBMA. First,
RoBMA did not adjust meta-analytic effect size downward in many of the analyzed
meta-analyses. Second, in Appendix E of Bartoš, Maier, Wagenmakers, et al. (2021), we
applied RoBMA to 28 meta-analyses from Many Labs 2 (Klein et al., 2018), a multi-lab
Registered Replication Report, where we know that publication bias is absent. Therefore,
if a method still detects publication bias or notably corrects the estimate downwards, this
is likely indicative of bias. When we applied RoBMA to the Many Labs 2 dataset, we
found no notable downward bias, unlike other publication bias adjustment methods.
Our analysis shows that it is important to employ multi-model methods when
adjusting for publication bias, as model selection is problematic in the absence of strong
PUBLICATION BIAS IN PSYCHOLOGY
19
knowledge about the data generating process. Our extension of the SWF work suggests
that the effects of publication bias are more deleterious than previously estimated.
However, it remains the case that for a sizeable proportion of studies, the correction is
relatively modest. The considerable overestimation of effect sizes and evidence for the
effect highlights the importance of using appropriate bias correction methods and the
imperative to adopt publishing formats that are robust to publication bias, such as
Registered Reports (Chambers et al., 2015).
Data Availability Statement
The data and R scripts for performing the analyses are openly available on OSF at
https://osf.io/7yzut/.
Conflicts of Interest
The authors declare that there were no other conĆicts of interest with respect to the
authorship or the publication of this article.
Acknowledgements
This work was supported by The Netherlands Organisation for ScientiĄc Research
(NWO) through a Vici grant (#016.Vici.170.083) to Eric-Jan Wagenmakers.
PUBLICATION BIAS IN PSYCHOLOGY
20
References
Alinaghi, N., & Reed, W. R. (2018). Meta-analysis and publication bias: How well does the
FAT-PET-PEESE procedure work? Research Synthesis Methods, 9 (2), 285Ű311.
https://doi.org/10.1002/jrsm.1298
Augusteijn, H. E., van Aert, R., & van Assen, M. A. (2019). The effect of publication bias
on the Q test and assessment of heterogeneity. Psychological Methods, 24 (1),
116Ű134. https://doi.org/10.1037/met0000197
Bartoš, F., Gronau, Q. F., Timmers, B., Otte, W. M., Ly, A., & Wagenmakers, E.-J.
(2021). Bayesian model-averaged meta-analysis in medicine. Statistics in Medicine,
40 (30), 6743Ű6761. https://doi.org/10.1002/sim.9170
Bartoš, F., & Maier, M. (2020). RoBMA: An R package for robust Bayesian meta-analyses
[R package version 2.1.1]. https://CRAN.R-project.org/package=RoBMA
Bartoš, F., Maier, M., Quintana, D., & Wagenmakers, E.-J. (in press). Adjusting for
publication bias in JASP and R Ů Selection models, PET-PEESE, and robust
Bayesian meta-analysis. Advances in Methods and Practices in Psychological
Science. https://doi.org/10.31234/osf.io/75bqn
Bartoš, F., Maier, M., Wagenmakers, E.-J., Doucouliagos, H., & Stanley, T. D. (2021).
Robust Bayesian meta-analysis: Model-averaging across complementary publication
bias adjustment methods. https://doi.org/10.31234/osf.io/kvsp7
Bartoš, F., & Wagenmakers, E.-J. (2022). Fast and accurate approximation to informed
Bayes factors for focal parameters. arXiv preprint arXiv:2203.01435.
https://doi.org/10.48550/arXiv.2203.01435
Bom, P. R., & Rachinger, H. (2019). A kinked meta-regression model for publication bias
correction. Research Synthesis Methods, 10 (4), 497Ű514.
https://doi.org/10.1002/jrsm.1352
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to
meta-analysis. John Wiley & Sons.
PUBLICATION BIAS IN PSYCHOLOGY
21
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019). Correcting for bias
in psychology: A comparison of meta-analytic methods. Advances in Methods and
Practices in Psychological Science, 2 (2), 115Ű144.
https://doi.org/10.1177/2515245919847196
Chambers, C. D. (2013). Registered reports: A new publishing initiative at cortex. Cortex,
49 (3), 609Ű610. https://doi.org/10.1016/j.cortex.2012.12.016
Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P., & Willmes, K. (2015).
Registered reports: Realigning incentives in scientiĄc publishing. Cortex, 66, A1ŰA2.
Dienes, Z. (2014). Using Bayes to get the most out of non-signiĄcant results. Frontiers in
Psychology, 5:781. https://doi.org/10.3389/fpsyg.2014.00781
Gronau, Q. F., & Wagenmakers, E.-J. (2018). Bayesian evidence accumulation in
experimental mathematics: A case study of four irrational numbers. Experimental
Mathematics, 27, 277Ű286. https://doi.org/10.1080/10586458.2016.1256006
Gronau, Q. F., Heck, D. W., Berkhout, S. W., Haaf, J. M., & Wagenmakers, E.-J. (2021).
A primer on Bayesian model-averaged meta-analysis. Advances in Methods and
Practices in Psychological Science, 4 (3).
https://doi.org/10.1177%5C%2F25152459211031256
Gronau, Q. F., van Erp, S., Heck, D. W., Cesario, J., Jonas, K. J., & Wagenmakers, E.-J.
(2017). A Bayesian model-averaged meta-analysis of the power pose effect with
informed and default priors: The case of felt power. Comprehensive Results in Social
Psychology, 2 (1), 123Ű138. https://doi.org/10.1080/23743603.2017.1326760
Heller, J. (1961). Catch-22. Simon; Schuster.
Hinne, M., Gronau, Q. F., van den Bergh, D., & Wagenmakers, E.-J. (2020). A conceptual
introduction to Bayesian model averaging. Advances in Methods and Practices in
Psychological Science, 3 (2), 200Ű215. https://doi.org/10.1177/2515245919898657
PUBLICATION BIAS IN PSYCHOLOGY
22
Hoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999). Bayesian model
averaging: A tutorial. Statistical Science, 14 (4), 382Ű401.
https://doi.org/10.1214/SS%5C%2F1009212519
Hönekopp, J., & Linden, A. H. (2022). Heterogeneity estimates in a biased world. PloS
One, 17 (2), 1Ű21. https://doi.org/10.1371/journal.pone.0262809
Hong, S., & Reed, W. R. (n.d.). Using monte carlo experiments to select meta-analytic
estimators. Research Synthesis Methods, 12 (2), 192Ű215.
https://doi.org/https://doi.org/10.1002/jrsm.1467
Jeffreys, H. (1939). Theory of probability (1st Edition). Oxford University Press.
Jeffreys, H. (1961). Theory of probability (3rd Edition). Oxford University Press.
Jeffreys, H. (1973). Scientific inference (3rd Edition). Cambridge University Press.
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams Jr, R. B., Alper, S.,
Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018). Many Labs 2:
Investigating variation in replicability across samples and settings. Advances in
Methods and Practices in Psychological Science, 1 (4), 443Ű490.
https://doi.org/10.1177/515245918810225
Kvarven, A., Strømland, E., & Johannesson, M. (2020). Comparing meta-analyses and
preregistered multiple-laboratory replication projects. Nature Human Behaviour,
4 (4), 423Ű434. https://doi.org/10.1038/s41562-019-0787-z
Larose, D. T., & Dey, D. K. (1998). Modeling publication bias using weighted distributions
in a Bayesian framework. Computational Statistics & Data Analysis, 26 (3),
279Ű302. https://doi.org/10.1016/S0167-9473(97)00039-X
Lee, M. D., & Wagenmakers, E.-J. (2013). Bayesian cognitive modeling: A practical course.
Cambridge University Press.
Lewis, M., Mathur, M., VanderWeele, T., & Frank, M. C. (2020). The puzzling relationship
between multi-lab replications and meta-analyses of the rest of the literature.
https://doi.org/10.31234/osf.io/pbrdk
PUBLICATION BIAS IN PSYCHOLOGY
23
Maier, M., Bartoš, F., & Wagenmakers, E.-J. (2022). Robust Bayesian meta-analysis:
Addressing publication bias with model-averaging. Psychological Methods.
10.1037/met0000405
Maier, M., VanderWeele, T., & Mathur, M. (2021). Using selection models to assess
sensitivity to publication bias: A tutorial and call for more routine use.
https://doi.org/10.31222/osf.io/tp45u
R Core Team. (2021). R: A language and environment for statistical computing. R
Foundation for Statistical Computing. Vienna, Austria. https://www.R-project.org/
Rosenthal, R., & Gaito, J. (1964). Further evidence for the cliff effect in interpretation of
levels of signiĄcance. Psychological Reports, 15 (2), 570.
https://doi.org/10.2466/pr0.1964.15.2.570
Sladekova, M., Webb, L. E., & Field, A. P. (2022). Estimating the change in meta-analytic
effect size estimates after the application of publication bias adjustment methods.
Psychological Methods.
Spiegelhalter, D. J., Abrams, K. R., & Myles, J. P. (2004). Bayesian approaches to clinical
trials and health-care evaluation. John Wiley & Sons.
Stanley, T. D., Doucouliagos, H., Ioannidis, J. P. A., & Carter, E. C. (2021). Detecting
publication selection bias through excess statistical signiĄcance. Research Synthesis
Methods, 12 (6), 776Ű795. https://doi.org/10.1002/jrsm.1512
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. (2017). Finding the power to reduce
publication bias. Statistics in Medicine, 36 (10), 1580Ű1598.
https://doi.org/10.1002/sim.7228
van Aert, R. C., Wicherts, J. M., & Van Assen, M. A. (2019). Publication bias examined in
meta-analyses from psychology and medicine: A meta-meta-analysis. PloS one,
14 (4), e0215052. https://doi.org/10.1371/journal.pone.0215052
van Erp, S., Verhagen, J., Grasman, R. P., & Wagenmakers, E.-J. (2017). Estimates of
between-study heterogeneity for 705 meta-analyses reported in Psychological
PUBLICATION BIAS IN PSYCHOLOGY
24
Bulletin from 1990Ű2013. Journal of Open Psychology Data, 5 (1), Article 4.
http://doi.org/10.5334/jopd.33
Vevea, J. L., & Hedges, L. V. (1995). A general linear model for estimating effect size in
the presence of publication bias. Psychometrika, 60 (3), 419Ű435.
https://doi.org/10.1007/BF02294384
Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values.
Psychonomic Bulletin & Review, 14 (5), 779Ű804.
https://doi.org/10.3758/BF03194105
Wagenmakers, E.-J., Morey, R. D., & Lee, M. D. (2016). Bayesian beneĄts for the
pragmatic researcher. Current Directions in Psychological Science, 25 (3), 169Ű176.
https://doi.org/10.1177/0963721416643289