Educational Psychologist

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/hedp20

Replication is important for educational

Jonathan A. Plucker & Matthew C. Makel

To cite this article: Jonathan A. Plucker & Matthew C. Makel (2021) Replication is important for
educational psychology: Recent developments and key issues, Educational Psychologist, 56:2,
90-100, DOI: 10.1080/00461520.2021.1895796

To link to this article: https://doi.org/10.1080/00461520.2021.1895796

© 2021 The Author(s). Published with

Published online: 31 Mar 2021.

Submit your article to this journal

Article views: 2801

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

Replication is important for educational psychology: Recent developments and

ABSTRACT
Replication is a key activity in scientific endeavors. Yet explicit replications are rare in many fields,
including education and psychology. In this article, we discuss the relevance and value of replica-
tion in educational psychology and analyze challenges regarding the role replications can and
should play in research. These challenges include philosophical, methodological, professional, and
utility concerns about replication in education and the social sciences more broadly. Finally, we
discuss strategies that may address these concerns in educational psychology research.

Replication is the intentional repetition of previous research 2014)—a debate we explore in greater depth later in this art-
to confirm or disconfirm the previous results, serving as a icle—these two categories are the most common and
de facto reliability check on previous research. Informing straightforward way of thinking about types of replication.
stakeholders about which results can be repeated—and in Regardless of form, replication is rare, perhaps far too
what circumstances—are the chief value that replications rare. In a series of studies, we have found low replication
contribute to research and the public at large. Successful rates in the published research bases of psychology, educa-
replication of research on positive outcomes associated with tion, special education, gifted education, and criminology,
a reading intervention, for example, provides educators and ranging from 0.13% in education to 1.07% in psychology
policymakers with confidence that can justify the investment (Makel & Plucker, 2014b, 2015; Makel et al., 2012, 2016;
of scarce public resources in implementation of that inter- Pridemore et al., 2018). Other researchers, using slightly dif-
vention. Conversely, a sensational research result that cannot ferent methods, have arrived at roughly similar rates
be replicated provides information to stakeholders that may (Lemons et al., 2016; McNeeley & Warner, 2015). Although
prevent unnecessary resource and opportunity costs. we have not specifically examined the presence of replication
Replicability is therefore a cornerstone of the research studies within educational psychology, the field’s major jour-
endeavor in educational psychology. It tends to occur in one nals were included in the education study (Makel & Plucker,
of two forms, direct or conceptual replications. When 2014b), suggesting replication is uncommon in educational
attempting a direct replication, researchers are attempting to psychology. If replication is a foundational activity within
follow the original study’s methods as closely as possible in the field yet rarely occurs, it is fair to question whether the
an effort to arrive at similar results. The goal of a direct rep- field’s impact is being unnecessarily limited.
lication is not a thumbs-up/thumbs-down decision; rather, While exploring issues related to replication over the past
as Simons (2014) notes, “The end result is not a judgment decade, we often found ourselves in faculty meetings, con-
of whether a single replication attempt succeeded or failed— ference sessions, and casual conversations with colleagues
it is a robust estimate of the size and reliability of the who asked questions about the idea of replication (philoso-
original finding” (p. 76). In contrast, the purpose of a con- phy of replication), strategies for conducting replication
ceptual replication is to examine the theoretical soundness of studies (methodology of replication), the professional feasi-
a particular finding or set of findings, with less focus on bility of replication (professional implications of replication),
repeating exact methods from the original study. Conceptual and replication’s utility in the field of educational psych-
replications purposefully alter factors such as participant ology (utility of replication). In this article, we attempt to
demographics, operationalization of dependent variables, or summarize researchers’ current understandings in all four
study context (see Schmidt, 2009, Figure 1). Although there areas, with the caveat that researchers have not yet answered
is considerable debate about the value of direct versus con- all of the concerns and questions about replication success-
ceptual replication (see Simons, 2014; Stroebe & Strack, fully. In each section, we provide summaries and analyses of

CONTACT Jonathan A. Plucker jplucker@jhu.edu Johns Hopkins University, 5801 Smith Avenue, Suite 400, Baltimore, MD 21209.
ß 2021 The Author(s). Published with license by Taylor & Francis Group, LLC.
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/
4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in
any way.
 EDUCATIONAL PSYCHOLOGIST 91

the most recent thinking and research on replication, in all students, of all ages, in all schools?” Second, to determine
addition to an examination of questions that have yet to be the need for some replication, one can ask, “Can we assume
answered. Our hope is that this article equips readers with with a reasonable degree of confidence that we know when
perspective to form deeper understanding of how replication results would generalize across specific contexts, ages, and
should factor into their own work as well as the field cultures, or do we need to collect data to be certain?” This
more broadly.1 establishes scenarios where replications add value. Third, to
establish confidence in the reliability of existing research,
one could ask, “Do you believe that the existing academic
Philosophy of replication publication process is 100% error free?” This helps demon-
Replication has well-defined, epistemological purposes strate fallibility with published findings. A fourth question,
to determine confidence in the field’s body of research on a
Some educational psychologists have questioned the philo- specific topic, is: “Do you think having more confidence in
sophical basis for replication. However, the rationale for rep- how research on a given topic generalizes, in which settings
lication research has strong epistemological foundations and for which students, would help you make more effect-
related to the nature of scientific knowledge. Indeed, Collins ive decisions?”
(1985) called replication the Supreme Court of science. Although we acknowledge that epistemological questions
Schmidt (2017) is more direct, noting that, “ … a single remain about replication and how to appropriately interpret
observation cannot be trusted,” and that “replication … is and decide when to use them (see, e.g., Gervais, 2021), these
capable of transforming an observation into a fact, or piece questions have evolved over the past generation from “Do
of knowledge” (p. 236, emphasis in original). Several scholars replications have value?” to “When do we need replications
have recently argued replication plays an important role in and how can we structure them to provide maximum val-
theory building and theory assessment (Guest & Martin, ue?” This is a non-trivial, philosophical advance that helps
2021; Irvine, 2021; van Rooij & Baggio, 2021). the field provide greater value via real world application.
Perhaps the most cynical framing of the epistemological
value of replication can be drawn from Planck’s (1949)
observation that science advances one funeral at a time. In Increased use of replication is needed
his 1949 autobiography, he wrote that, “a new scientific
The field has provided value for many decades, why is an
truth does not triumph by convincing its opponents and
enhanced focus on replication needed now? Calls for repli-
making them see the light, but rather because its opponents
cation are not new, and problems in other, related fields
eventually die and a new generation grows up that is famil-
suggest educational psychology could learn from errors in
iar with it” (p. 33). More to the point, he observed, “An those fields rather than waiting to suffer from them itself.
important scientific innovation rarely makes its way by For example, concerns about replicability have been raised
gradually winning over and converting its opponents” (p. about the research base in a broad range of fields, from eco-
97), an idea reinforced by both philosophers of science (e.g. nomics to medicine to the life sciences (see Zwaan et al,
Kuhn, 2012) and empirical evidence (Azoulay et al., 2019). 2018). Within the social sciences, psychology has long been
But need this historical reality be destiny? Although
considered to have a replication crisis (Pashler & Harris,
Azoulay et al. also note the potential value of having emi-
2012; Rosenthal, 1969; Schlosberg, 1951). This has gained
nent gatekeepers control the flow and dominance of ideas,
attention in recent years due to suspicions of research mis-
especially within nascent fields, educational psychology is a conduct by several prominent researchers. Whether one
mature field. Regular and planned replication is one tool believes this situation to be important or overblown, a good
that can help education science self-correct more quickly. crisis should never be wasted. In this vein, Vazire (2018) has
Why relegate advances in educational psychology to the pas- reframed the replicability crisis into a credibility revolution
sage of time when we have open science tools at our dis- that embraces several methodological strategies, including
posal, replication first among them, that can help advance conducting more replication research.
the field’s knowledge faster and more efficiently? Doing so The presence of a credibility crisis is certainly applicable
will allow the field to be more transparent and democratic to education research. The first author once testified before
about what we know, what we do not know, and when the a state senate education committee and was surprised to
field is heading down theoretical rabbit holes that do not hear senators sarcastically noting that researchers can make
translate to practice. their studies say whatever they want. From an empirical per-
The philosophical value of replications in educational spective, Merk and Rosman (2019) found evidence that stu-
psychology is underscored via a series of intentionally straw- dent-teachers held a “smart but evil” stereotype about
person questions. First, to examine constraints on generality education researchers, “as the authors of scientific studies
(Simons et al., 2017) one can ask, “If we conducted every … are perceived not only as less benevolent, with less
study with undergrads, could we generalize those results to integrity, but also as having more expertise in contrast to
practitioners. This is an intriguing finding, as it suggests
1
After we had written most of this article, we noted the unintended similarity that student-teachers hold a kind of distrust in scientists” (p.
in structure to Zwaan et al. (2018), who used potential concerns about
replication to organize their comments. Although our framing is unique, we 6). We do not believe such perceptions by relevant stake-
acknowledge the overlap in general structure. holders help the scientific endeavor in educational
92 J. A. PLUCKER AND M. C. MAKEL

psychology. An increase in credibility among stakeholders is That said, conceptual replications may be more inform-
one possible pathway toward gaining appreciation for the ative when they are conducted systematically through pur-
accuracy and usefulness of our efforts. posefully altering a single variable rather than through
changing many variables. If the population, independent
variables, and outcome variables are all changed, making
Most studies are not replications strong conclusions about replicability of original studies
Several scholars have argued that most psychological studies becomes complicated if not impossible. In a related vein,
are de facto replication studies, given that they investigate haphazard conceptual replications may provide little value.
similar theoretical constructs using different methods than For example, is there theoretical rationale for assuming there
earlier studies; in other words, most papers are conceptual may be a difference in how left-handed and right-handed
replications (Smith et al., 2017; Stroebe & Strack, 2014). students respond to math tutoring intervention? If not, such
Chhin et al. (2018), in a study of IES-funded projects, found a conceptual replication may not be worth pursuing.
no direct replications but concluded that nearly half of Regardless of how educational psychologists feel about
funded applications involved conceptual replications. conceptual versus direct replication (or any other classifica-
We understand the temptation to consider most papers tion system for replication), they should explicitly state their
to be undeclared, conceptual replications. But failing to label intent; when they fail to state their intent to replicate a the-
them as such (and take advantage of open science strategies oretical position or empirical finding, it becomes difficult
such as preregistration) disrupts the research process, mak- for the field to move forward. Explicit intent can be easily
ing it harder for consumers of research to know what has systematized, with authors including the following language
and has not been replicated (Hodges, 2015; Reich, 2021/ in their papers: “We are attempting to [directly/conceptu-
this issue). ally] replicate the methods used by [citation] in their study
For these and other reasons, Simons (2014) argued that of [key concepts or interventions].” In addition, direct repli-
direct replications are the only way to verify the reliability cations should state, “We kept all methods as similar to the
of results, a position that is attracting growing levels of sup- original as possible, with the following exceptions.”
port (e.g. Machery, 2019; Nosek & Errington, 2019). Conceptual replications should state, “Our study methods
H€uffmeier et al. (2016) offer a more nuanced typology differ from the study to be replicated in the following ways.”
involving exact replications (direct replication conducted by Such language would add substantial clarity with minimal
same researchers), close replications (also direct but by dif- length and should be included in both the abstract and body
ferent researchers), constructive (also direct but a similar of research papers.
study modified in a small number of ways to assess robust-
ness of original effect), conceptual under lab conditions The contextual nature of education is not a barrier to
(conceptual attempt to study theory), and conceptual under replication
field conditions (also conceptual and attempting to study
robustness of theoretical effect). An advantage of the Given that children are unique and, therefore, have unique
H€uffmeier et al. approach2 is that even if one assumes most experiences, concerns about generalizability (or lack thereof)
empirical studies are conceptual replications, this typology are important issues given the variability of students and
stresses the importance of systematic and sequential replica- their contexts. A belief that no research finding can general-
tion approaches to advancement of psychological knowledge, ize may be the most extreme version of this concern. We
and that conceptual replication without earlier forms of sup- have encountered such views, often sociocultural in nature
portive, direct replication are not adding meaningfully to a (see Turner & Nolen, 2015), but do not find them compel-
field’s research base (see Zwaan et al., 2018, for a similar ling. There are numerous educational findings that have
argument). Similarly, Irvine (2021) argued that even the best been replicated and generalized across contexts, including
conceptual replications have a low capacity for theoretical the results of sociocultural inquiry (e.g. Coalition for
payoff in most circumstances. Psychology in Schools and Education, 2015). Anyone who
This importance of direct replication in no way implies lacked confidence in generalizability would likely have to
that conceptual replications lack value, as they help assess believe that educational psychology should consist only of
generalizability and establish boundary conditions for empir- case studies or action research. Moreover, from this perspec-
tive, publishing these case studies and action research would
ical effects. Comparing the relative merit of each type of
have little value beyond descriptive biography because they
replication should not be about which is better or more
would not inform practice in other contexts.
important. Rather, which form of replication is most useful
for a given effect at a given time? If a finding is wholly
novel, direct replication may be more useful prior to con- Methodology of replication
ceptual replication. If a finding has been observed several
times in one population, a conceptual replication may be Both replication and meta-analysis are necessary
more informative. Meta-analysis and replication address research quality issues
and are complementary processes, but they have distinct
2
See, in particular, Table 1 and Figure 1 in H€
uffmeier et al. (2016). purposes and therefore address research quality in different
 EDUCATIONAL PSYCHOLOGIST 93

ways (Patall, 2021/this issue; Valentine, 2019; Williams et al., important. For example, does replication apply at all to
2017). Meta-analyses synthesize previous research, whereas studies using grounded theory or critical race theory? How
replications seek to verify whether previous research findings can qualitative research approach independent replication
are reproducible and, therefore, accurate. Wide variance in when individual subjectivity or background personal context
construct definition, instrumentation, sampling, and data plays a central role in many approaches to qualitative inter-
analysis, among other factors, can result in a diverse pool of pretation? How can qualitative replication help build or
studies within a meta-analysis, none of which may have assess theory? Developing answers to these questions will
been previously replicated. A carefully conducted meta-ana- help inform when and how replication can play a role in
lysis of irreproducible studies is of no value (see Carter qualitative research.
et al., 2019). In an examination of replication within qualitative
Moreover, meta-analyses and replications solve different research, Leppink (2017) concludes that “perhaps we should
problems. Meta-analyses help solve the problem of heteroge- no longer think in terms of qualitative–quantitative divides
neous results (which may be driven by moderators such as but rather in terms of more-less replicable distinctions, and
using different samples or measures). However, replications do all that is possible to document all choices and decisions
help assess (and address) experimenter bias. Namely, if a made throughout a study to enable others to replicate our
researcher has a bias (e.g. wants to find a specific result, will work” (p. 100). This recommendation is applicable to all
be rewarded if certain results are obtained), meta-analyzing forms of research within educational psychology and the
multiple studies from the same lab will amplify the bias. learning sciences. If a research result (whether quantitative
From this perspective, independent replication is necessary or qualitative) is so narrow and fragile that it can never be
but insufficient for meaningful meta-analysis. found again (even by the same research team), that result
would be of little use to practitioners and policymakers.
Conversely, for example, a series of case studies that all
Replication is relevant to all forms of research found evidence, hypothetically speaking, that teachers find
Some may view replication as applying only to experimental differentiation difficult in heterogeneously-grouped class-
research. From this perspective, replication may be viewed rooms should be viewed as successful conceptual replica-
as “other people’s problem” by researchers who do not con- tions—and would provide important information to
duct experimental research. However, replicability and reli- stakeholders.
ability of results are important across all empirical
approaches to study design and data collection. Across a
Research rigor involves more than replication
range of disciplines, plentiful examples are available of quan-
titative, non-experimental research being subject to replica- We often encounter colleagues who note that increased use
tion attempts (e.g. Kanai, 2016; Piffer, 2019). For example, of replication within educational psychology will not auto-
Ebersole et al. (2016), as part of a Many Labs collaboration, matically lead to huge increases in the quality of the field’s
attempted to replicate both experimental and correlational research. Indeed, increased use of replication is a necessary
effects in social psychology across undergraduate research but insufficient strategy for improving research quality (see
pools to determine whether variations in participation in also Nosek et al., 2021; Schmidt, 2017). In recent years, a
research across the course of a semester produced signifi- range of strategies have been suggested and refined for
cantly different effects, finding little evidence they did. improving how research is conducted and communicated,
Assessing whether the timing of an event influences conse- many falling under the banner of open science. These wide-
quences is of great relevance to all of education and educa- ranging approaches include preregistration of hypotheses
tional psychology. Ironically, we once had a descriptive (i.e. publicly sharing a study’s methods before the study is
study (on replication, no less!) replicated by a team that was conducted), open data, and meta-analysis, among many
studying the same issues at the same time for the same jour- others (Chambers, 2017; Cr€ uwell et al., 2019; Fleming et al.,
nal special issue (see Lemons et al., 2016; Makel 2021/this issue; Gehlbach & Robinson, 2021/this issue;
et al., 2016). Makel & Plucker, 2017; Nosek et al., 2015; Reich, 2021/this
Regarding qualitative methods, most readers will not be issue; Spellman, 2015). To facilitate these practices, web
surprised that this is an area of considerable debate, with services are available that allow for posting of preregistra-
some strong, negative views of the importance of replication tions, data sharing, and posting of pre-prints (e.g. https://
(and even the concept of replicability) to qualitative research osf.io/, https://edarxiv.org/). Many of these approaches to
(Pratt et al., 2020). However, perspectives are emerging improving the rigor of research have been embraced by edu-
among qualitative researchers that replication is important. cational psychologists.
Qualitative researchers appear to be focusing on the clear For example, a recent issue of AERA Open was dedicated
communication of methods to facilitate replicable qualitative to publishing registered reports and included replications
work (Anczyk et al., 2019; Schindler et al., 2020; (Reich et al., 2020), with the vast majority being educational
Steinhardt, 2020). psychology research. The studies were primarily inferential
The application of replication to qualitative research is and experimental in nature but included descriptive work
relatively recent, and therefore the number of unanswered (e.g. Peters et al., 2019). In the ensuing commentaries (see
questions in this area of scholarship are numerous and Reich et al., 2020), many of the authors noted the
94 J. A. PLUCKER AND M. C. MAKEL

advantages of approaching research from an open science/ failed replication attempts tend to capture media attention
heightened credibility perspective. For example, Merk and and be noted on social media, replications in the social sci-
Rosman (2020) noted that preregistration requires more ences tend to be confirmatory. Successful replication rates
detailed methods sections in papers, and that the discussion tend to be over 65% in education, psychology, economics,
section was “more honest and vivid as we could, for and special education, with some rates in excess of 80%
example, give sharper opinions” (p. 1). Another benefit is (Camerer et al., 2016; Klein et al., 2014; Lemons et al., 2016;
that the detailed method sections typically found in preregis- Makel & Plucker, 2014a; Makel et al., 2012, 2016; cf.
tered studies facilitate future replication attempts. Any pro- Camerer et al., 2018). Although self-replications tend to be
gress toward heightened research credibility, regardless of more successful than third-party replications (Makel et al.,
whether one refers to it as “open science” or some other 2012; Makel & Plucker, 2014b), they are still often successful
term, is a positive development for educational psychologists even when conducted by third-party researchers (71 vs.
and the educators and families whom our work benefits. 54%, respectively, Makel & Plucker, 2014b, Table 2). But we
Development and testing of theory may be an even more note that these estimates are built on replications that were
important component of improving research rigor, but that generally not preregistered. Replicability estimates of prereg-
does not mean that replication is irrelevant (Wentzel, 2021/ istered replications are often lower (e.g. 35%; Open Science
this issue). Some have proposed that replications are part of Collaboration, 2015), which may result from the many ways
the process of strengthening the empirical portion that success of a replication can be determined (see Gervais,
informs the cycle of theory development and assessment 2021, for 11 such examples).
(van Rooij & Baggio, 2021). Irvine (2021) argued that with- Replication research is also increasingly attractive to
out sufficiently advanced theory, replications may have lim- external funders. Howe and Perfors (2018) note that “grant
ited value but that they can serve as a tool to improve agencies greatly value novelty, but they even more greatly
theory. van Rooij and Baggio (2021) share the vivid example value reliable science; a novel finding can have a long-term
of knowing apples fall from trees (a replicable effect) but impact only if it is true” (p. 25). For this reason, it is not
needing the theory of gravity to explain the effect and pro- surprising that the National Science Foundation and the
vide true understanding. In addition, they argued that the Institute of Educational Sciences (IES) now have regular
development of theory is not only the ultimate priority but funding competitions for replication research. Replications
also the foundation of all future empirical efforts (cf. Eronen have become such a prevalent part of funding projects that
& Bringmann, 2021). these agencies have jointly published companion guidelines
on conducting replication research (National Science
Foundation & Institute of Educational Sciences, U.S.
Professional implications of replication Department of Education, 2018). This type of action also
Replications are good for one’s career suggests a cultural shift is occurring, making replications
both a public good for the field and good for one’s car-
Regardless of whether one believes replications can benefit eer, too.
the field, they may wonder if conducting replications will Finally, replication research may lower opportunity costs
further their own careers.3 Sufficient incentives exist within within the busy careers of educational psychologists.
education to allay this concern: Replications are cited, Consider some worst case scenarios: A researcher spends
unsuccessful replications can be well received, and external years working on a concept that, over time, others cannot
funding for replications is growing. replicate. Or a group of graduate students devote their
As shown in Makel and Plucker’s (2014b) study of repli- research time pursuing topics that appear enticing but
cation in the Top 100 education journals as ranked by ultimately fail to replicate. Neither situation is good for
impact factor, the median number of citations for articles one’s career, but contrast those scenarios with that of an
being replicated was 31 (range from 1 to 7,644) while the early career researcher who conducts a series of replications
median citation count for the replicating paper was 5 (range on foundational studies on a particular construct, some suc-
from 0 to 135). Although the replicating papers were cited cessful and some not, that help improve theory on that con-
far less often, the median citation count for those papers struct and create more effective interventions. The latter
was higher than the impact factor for all of the 100 journals scenario is of greater benefit to both the researcher and
at that time. the field.
Some researchers may be concerned that conducting
unsuccessful replications will give one the reputation of
being a critic within their field, which could cause problems Replication is pro-innovation
when going up for promotion, being considered for fellow- Researchers work in communities that reward creativity and
ships and other honors, and other aspects of a profession innovation. Some may be concerned that replications dis-
where peer assessment is valued and important. Although tract or detract from creative contributions. Under no cir-
cumstances is this accurate. This question is often asked or
3
In economics, this is called the tragedy of the commons: Replication may be implied by journal editors, who note that they are resistant
a public good (benefitting all), but individuals may act as free riders,
benefitting when others perform replications but not acting to conduct to publishing replication papers because they want their
replications themselves. journal to focus on creative additions to the research
 EDUCATIONAL PSYCHOLOGIST 95

literature. The attraction to the shiny new object in research considerable evidence of stereotype threat (Smerdon et al.,
is well-known (e.g. Fanelli, 2010; Hodges, 2015; Howe & 2020). The author of the original study acknowledged the
Perfors, 2018; Makel, 2014), but this attitude confuses nov- failed replication, noted ways in which the replication
elty with creativity. Although definitions of creativity almost improved on his original study, provided even-handed ana-
always involve characteristics of novelty, originality, or lysis on the possible causes for the divergent results, and
uniqueness, they also require usefulness or utility (e.g. even defended the replication authors against subsequent
Plucker et al., 2004; Simonton, 2012). criticisms of the replication (Stafford, 2020). This commit-
If an idea cannot be replicated, arguing that the idea is ment to building an accurate research base rather than
useful, especially in an applied field such as educational reflexively defend one’s personal investment in their
psychology, is a difficult case to make (Makel & Plucker, research is commendable and serves as a model for educa-
2014a). Furthermore, given that innovation can be concep- tional psychologists.
tualized as creativity taken to scale, a finding that cannot be Many researchers have proposed ways to make replica-
replicated—regardless of one’s chosen definition of replica- tion less adversarial, from systematic approaches involving
tion—can never inform innovative practice. At best, it can changing research culture to specific replication strategies.
only misinform practice and mislead educators and policy- Gernsbacher (2018) has pointed to reciprocal replica-
makers. An irreplicable research result within educational tions—in which teams of researchers attempt to replicate
psychology is neither creative nor innovative; a replicable each other’s work—as one path forward. Tierney et al.
result is likely to be creative, and if taken to (2020) proposed a creative destruction approach that con-
scale, innovative. ceptualizes replications as the act of replacing original
A related aspect involves the primacy effect as applied to results with revised results that are more powerful or
research, in which the first study published on a topic is more precise. Regardless of the approach, having more
assumed to be the most valuable. Gelman (2017, 2018) has frequent replications in educational psychology will help
proposed a time-reversal heuristic—a thought experiment in make them less of an aberration, more difficult to inter-
which researchers consider how their evaluation of a theor- pret as a personal attack, more of a key aspect of the
etical effect changes if an unsupportive direct replication educational psychology enterprise, and more successful in
were published first, and the original, exploratory, support- improving interventions and the theories on which they
ive study were published second. Most people would be are based.
skeptical of the second study’s results, when in fact both
should be considered equally when evaluating the research
on the effect. The time-reversal heuristic would have us con- Educational psychology is starting to value and support
sider usefulness before novelty, an admittedly creative replication
approach to assessing creativity in research. As the preceding sections suggest, the conventional wis-
dom on the value of replication within educational psych-
ology is changing, albeit slowly and unevenly. There are
Replication need not be an adversarial act
reasons to be optimistic about changing attitudes within
Is conducting a replication an aggressive act? Not necessar- the field toward replication and open science practices
ily, given that replication attempts can be perceived as a (see Fleming et al., 2021/this issue; Mellor, 2021/this
form of flattery, in that a researcher’s colleagues are paying issue). For example, editors from some specialty journals
attention to their work. In a field with little to no replica- in and related to the field have published editorials
tion, it is human nature to find any attempt to replicate endorsing and implementing open science practices (e.g.
your work to be suspicious if not adversarial. Until the Adelson & Matthews, 2019; Hodges, 2015; Spector et al.,
research culture changes to embrace replication and other 2015), and journals that feature work from the field, such
open science strategies, replication will always have at least a as AERA Open, The British Journal of Educational
tinge of an adversarial feel. But researchers can influence the Psychology, Exceptional Children, and Journal of
degree to which replication is viewed as constructive ver- Educational Psychology, accept registered reports. But to
sus aggressive. our knowledge few other educational psychology journals
For example, compare the reactions of two distinct sets have taken similar steps and acceptance of replications is
of authors whose work has recently been the subject of less clear. When all of the field’s journal editors state
unsuccessful replication attempts. In the first, a study of the unequivocal support for replication and open science
impact of human-like avatars on decision-making in a tech- approaches to research, many of the professional concerns
nology context was replicated with mixed and generally surrounding replication will dissipate.
negative results (Simmons & Nelson, 2020). The authors of Increasing calls for innovation in how we teach research
the original study responded politely, but their response methods at the graduate level will also hopefully result in
included several paragraphs exploring why the replication lasting culture change (Gernsbacher, 2018; Spector at al.,
was likely flawed. 2015). For example, Kochari and Ostarek (2018) have called
However, in the other example, Stafford (2018) authored for making direct replications a required research activity
a study that did not find evidence of stereotype threat for doctoral students, and Yeo-Teh and Tang (2021) have
among chess players. A replication of the study found suggested a research ethics and integrity course for doctoral
96 J. A. PLUCKER AND M. C. MAKEL

students that emphasizes the responsible conduct of any assessment process and not to rely solely on formal stat-
research, including issues such as preregistration and other istical comparison. As with original findings, we believe con-
open science practices. Using graduate education as a major text issues such as theory, measurement validity, and
intervention point allows educational psychologists to pre- relevant previous findings all need to play a vital role in
pare the next generation of innovators as opposed to slowly interpreting replication success.
adopting and adapting new research practices after everyone Interestingly, researchers have spent considerable time
has already entered the field. debating the flip-side of this coin, the definitions of “failed”
replications. Again, the issue of context is especially relevant
to this discussion, as one can usually explain away a failed
Utility of replication
or mixed replication attempt as being due to differences in
It is possible to determine if a replication is “successful” subtle context or hidden moderators. That may be the case,
but the authors of the original study then bear responsibility
We once led a replication workshop at a federal agency for not adequately describing key contextual factors in the
where the participants questioned whether it was possible to success of the original intervention. Indeed, an intervention
determine if a replication attempt was successful. However, whose success is so highly dependent on subtle variations of
in the absence of replication, researchers routinely make context is unlikely to be of great use to practitioners who
judgments about the accuracy of collective bodies of
attempt to use the intervention. Gelman (2018) argues simi-
research, dealing with variables in study quality and meth-
larly, noting that “various concerns about the difficulty of
odology, but with little information about whether a given
replication should, in fact, be interpreted as arguments in
finding is replicable. If we can determine success and failure
favor of replication … if effects can vary by context, this
of original studies, we can determine success of replications.
provides more reason why replication is necessary for scien-
Simple heuristics such as comparing whether p-values are
tific progress” (p. 19, emphasis in original). In other words,
similar have their limitations, including ignoring whether
context happens, and finding interventions that are useful
the magnitude of the effect is the same (Valentine, 2019).
across contextual differences (or, more to the point, are use-
But this limitation is not unique to replications; it holds true
ful only within certain, definable contexts) is a goal of edu-
for original research as well. Several approaches that could
cational psychology.
be taken to assess replication success have been proposed,
In a related vein, some may wonder whether replications
each with its strengths and limitations (Schauer & Hedges,
“fail” because the replicators simply are not as skilled as the
2021), including interpreting confidence intervals (Jacob
researchers involved with the original study. But there is no
et al., 2019; Zwaan et al., 2018), a combination of sample
evidence that replications fail because replicators lack experi-
and effect sizes (Simonsohn, 2015), and a replication Bayes
ence or expertise (Nosek, 2020; Protzko & Schooler, 2020).
factor (Zwaan et al., 2018). Others have noted that more
Moreover, in an applied field like educational psychology, it
than one replication may often be needed for unambiguous
interpretation of effects (Hedges & Schauer, 2019) or that is important to know whether special skill is required to
reproducible results may not lead to a “convergence to sci- elicit a particular effect. What if not all teachers or schools
entific truth” (Devezer et al., 2019, p. 17) because of have this elusive trait? Should they expect effects or not?
research and statistical weaknesses as well as the complex Those are the questions practitioners want and need
nature of the world. answered. If researchers can develop even marginally
We have no issue with attempts to create objective crite- informative practices, they would be immediately useful to
ria for evaluating replication results but offer a broad educators who are experiencing initiative fatigue.
approach that is more direct: Do the results give the reader
more or less confidence in the validity of the original find- There is no magic number for “enough” replications
ings? More objective and statistical approaches, such as
those mentioned above, may inform the answer to this ques- Assessing the need for replication is not about a particular
tion, but in the end stakeholders will make subjective judg- replication rate, although we believe 1% is too low. Rather
ments (e.g. importance of topic, theoretical underpinnings than focusing on replications rates, other criteria must be
of an intervention, quality of study design, magnitude of considered. If one were to ask how many people should be
effects) about the results of any study, and replications are on anti-cholesterol medication, the answer would not be a
no different. Just as a specific p value should not automatic- percentage of the population. Instead, the answer would be
ally equate to action, nor should any other generic statistic. based on the proportion of individuals who met specified
A more interpretive approach allows educators to consider criteria (e.g. age, cholesterol levels, family history of cardio-
local context when evaluating, for example, whether efficacy vascular disease) associated with problems and benefits rele-
research on a specific intervention has been replicated. For vant to anti-cholesterol medication. The percentage of
example, the replications may be sufficiently supportive of individuals who meet that criteria may vary over time and
efficacy to allow the intervention to be used in a small-scale, place, but it is those contextual factors that determine
experimental program but not sufficiently supportive to appropriateness and value. Similarly, Irvine (2021) argued,
expand the intervention to every school in a large district. "as the current state of knowledge informs what counts as a
Meehl (1978) similarly emphasized the importance of using good replication, what counts as a good replication can
relevant theory and other background knowledge as part of change" (p. 8). For these reasons, we find a percent target
 EDUCATIONAL PSYCHOLOGIST 97

for educational psychology to be a less-than-ideal lens structure within higher education and the broader domain
through which to assess appropriateness of quantity of of academic research (Mellor, 2021/this issue). What gets
replications. published and funded as well as what helps get people hired,
Rather, we favor a focus on the use of the following crite- promoted, and honored needs rethinking. Or perhaps more
ria for determining whether studies should be replicated. accurately, perceptions of what gets rewarded matter, and
From an empirical perspective, highly-cited studies should those perceptions are still largely misaligned with replication
be prioritized for replication. If a study on a new construct and other strategies for improving educational psychology
or intervention is cited several dozen or several hundred research. Additionally, undergraduate and graduate methods
times in its first year after publication, those citations can be courses—as well as courses for pre-service teachers and
interpreted as the field voting with its feet, so to speak, administrators—need to include replication and its import-
about the importance of the study. Regarding qualitative evi- ance. Another system-level step involves educational psych-
dence, a set of research findings that are about to be scaled ology journals and professional organizations adopting the
up for broad implementation or are being included in text- Transparency and Openness Promotion (TOP) Guidelines
books and course materials should be targeted for (Nosek et al., 2015), which include standards on replication.
replication. And venerable groups within the field, such as Division 15
Viewed conceptually, a replication adds value if it helps of the American Psychological Association, should offer
build theory (Guest & Martin, 2021), narrow or assess a awards for high-quality replication attempts (Gorgolewski
given theory (Irvine, 2021), or assess whether an explanation et al., 2018).
has particular boundaries on effects (van Rooij & Baggio, The current discussion and debate about improved
2021). In particular, high rates of replication may be needed research methodology in educational psychology and other
in studies based on weak theory, a lot of variability in results fields should in no way give the impression that adoption of
across contexts, or theory that predicts many complex fac- open science methods is the sole solution for improving the
tors affecting outcomes. In the end, the question about repli- field’s impact. The best studies, even with huge sample sizes,
cation prevalence boils down to whether a field has impeccable design and measurement, and cutting-edge ana-
replicated its most important studies or needs them as part lysis techniques, will not be useful if based on weak theoret-
of the cycle of theory development, rather than whether ical foundations. Theory matters (Gehlbach & Robinson,
researchers have replicated a specific percent of the field’s 2021/this issue; Smith et al., 2017; Wentzel, 2021/this issue),
empirical output. and having carefully constructed theories that build on theo-
ries and research of the past serve as the foundation for all
we do as applied psychologists (Vartanian, 2017). The grow-
Increased use of replication is feasible ing discussion of replication’s role in theory development
Regarding more frequent use of replication within educa- and assessment (e.g. Guest & Martin, 2021; Irvine, 2021; van
tional psychology, we need to “Make it so” (Picard, 2366). Rooij & Baggio, 2021) will likely have a major impact on
An ideal educational system is informed by research evi- replication use. For example, making decisions about when
dence that practitioners and policymakers are confident will replications are needed and what types of replication should
lead to desired outcomes. We struggle to see any realistic be used may be informed by what they contribute to the
path toward this goal that does not include greater use of development of a particular theory.
replication in educational psychology research. The idea that Making gains on both people and system challenges to
science is self-correcting is well-established, but such correc- increase use of replication will likely occur in stages.
tion does not occur magically. Science is self-correcting Developing momentum behind changing cultural norms will
when scientists are self-correcting. Replication attempts help require coalition-building and sustained effort of many
inform the practices in which we should have confidence within their research communities and departments. To
and the practices that need correcting. Frequent replication achieve this culture change, several universities in the
attempts will help accomplish this more quickly than trad- United Kingdom have jointly agreed to appoint research
itional, largely replication-free approaches to research. quality officers (Munaf o, 2019), sending a clear message
To make replication more common within the field, we across institutions and stakeholder groups about the need to
see two challenges of adaptation: people and systems. People act collectively to increase research quality and impact. We
challenges (like those mentioned above), often stem from see no reason why educational psychologists, and education
individuals needing to admit there is a problem. Educational researchers in general, cannot act so boldly.
psychology is not alone in needing to change behaviors to
live up to norms (see Vazire, 2018). Psychology, even fMRI
Conclusion
research, suffers from reliability and replicability problems
(Elliott et al., 2020). Another people challenge to replications Replication is a necessary cornerstone for effective scientific
is researchers giving undue trust and credit to original stud- endeavors, yet explicit replications are too rare within edu-
ies simply because they were published first (again, cational psychology. In concert with other open science
Gelman’s time-reversal heuristics). strategies, increased use of strategically planned and timed
System challenges that prevent greater replication preva- replications would improve the overall quality and value of
lence include things as fundamental as the incentive educational psychology research. Of course, the field needs
98 J. A. PLUCKER AND M. C. MAKEL

to explore several aspects of replication to help maximize its Chambers, C. (2017). The seven deadly sins of psychology: A manifesto
potential benefits, such as how replication efforts can best for reforming the culture of scientific practice. Princeton University
Press.
inform theory development, the extent to which replication
Chhin, C. S., Taylor, K. A., & Wei, W. S. (2018). Supporting a culture
can be applied to various forms of qualitative research, and of replication: An examination of education and special education
how replication can be most effectively incentivized, among research grants funded by the Institute of Education Sciences.
other topics. But as Leppink (2017) noted in an examination Educational Researcher, 47(9), 594–605. https://doi.org/10.3102/
of replication across all types of research methodologies, the 0013189X18788047
Coalition for Psychology in Schools and Education. (2015). Top 20
goal of replication is to “allow us to work together toward
principles from psychology for preK-12 teaching and learning.
stronger conclusions and implications for future research American Psychological Association.
and practice” (p. 100). This point is well-taken, and we see Collins, H. M. (1985). Changing order: Replication and induction in sci-
it as a fitting goal for educational psychologists, too. By col- entific practice. University of Chicago Press.
laborating and working together to improve the quality of Cr€uwell, S., van Doorn, J., Etz, A., Makel, M. C., Moshontz, H.,
our research, educational psychologists will create positive Niebaum, J., Orben, A., Parsons, S., & Schulte-Mecklenbeck, M.
(2019). 7 easy steps to open science: An annotated reading list.
outcomes for both research and practice in education and Zeitschrift F€ur Psychologie, 227(4), 237–248. https://econtent.hog-
learning. Replication—in conjunction with strong theoretical refe.com/doi/10.1027/2151-2604/a000387
foundations and the widespread use of other open science Devezer, B., Nardin, L. G., Baumgaertner, B., & Buzbas, E. O. (2019).
practices—will help us achieve this goal. These collabora- Scientific discovery in a model-centric framework: Reproducibility,
tions will help us understand existing work better, conduct innovation, and epistemic diversity. Plos One, 14(5), Article
e0216125. https://doi.org/10.1371/journal.pone.0216125
future work more efficiently and effectively, and provide
Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M.,
greater value to practitioners and policymakers. Allen, J. M., Banks, J. B., Baranski, E., Bernstein, M. J., Bonfiglio,
D. B. V., Boucher, L., Brown, E. R., Budiman, N. I., Cairo, A. H.,
Capaldi, C. A., Chartier, C. R., Chung, J. M., Cicero, D. C.,
Acknowledgment Coleman, J. A., Conway, J. G., … Nosek, B. A. (2016). Many Labs
The authors acknowledge and appreciate the constructive criticisms and 3: Evaluating participant pool quality across the academic semester
suggestions for improvement provided by the guest editors and reviewers. via replication. Journal of Experimental Social Psychology, 67, 68–82.
https://doi.org/10.1016/j.jesp.2015.10.012
Elliott, M. L., Knodt, K. R., Ireland, D., Morris, M. L., Poulton, R., S.
Sison, M. L, R., Moffitt, T. E., Caspi, A., & Hariri, A. R. (2020).
What is the test-retest reliability of common task-fMRI measure?
ORCID New empirical evidence and a meta-analysis. Psychological Science,
Jonathan A. Plucker http://orcid.org/0000-0002-5327-0851 31(7), 792–806. https://doi.org/10.1177/0956797620916786
Matthew C. Makel http://orcid.org/0000-0002-3837-0088 Eronen, M. I., & Bringmann, L. F. (2021). The theory crisis in psychology:
How to move forward. Perspectives on Psychological Science. Advance
online publication. https://doi.org/10.1177/1745691620970586
References Fanelli, D. (2010). Do pressures to publish increase scientists’ bias? An
empirical support from US States Data. PLoS One, 5(4), e10271.
Adelson, J. L., & Matthews, M. S. (2019). Gifted Child Quarterly’s com- https://journals.plos.org/plosone/article?id=10.1371/journal.pone.
mitment to transparency, openness, and research improvement. 0010271
Gifted Child Quarterly, 63(2), 83–85. https://doi.org/10.1177/ Fleming, J. I., Wilson, S. E., Hart, S. A., Therrien, W. J., & Cook, B. G.
0016986218824675 (2021/this issue). Open accessibility in education research:
Anczyk, A., Grzymała-Moszczy nska, H., Krzysztof-Swiderska, A., & Enhancing the credibility, equity, impact, and efficiency of research.
Prusak, J. (2019). The replication crisis and qualitative research in Educational Psychologist, 56(2), 110–121. https://doi.org/10.1080/
the psychology of religion. The International Journal for the 00461520.2021.1897593
Psychology of Religion, 29(4), 278–291. https://doi.org/10.1080/ Gehlbach, H., & Robinson, C. D. (2021/this issue). From old school to
10508619.2019.1687197 open science: The implications of new research norms for educa-
Azoulay, P., Fons-Rosen, C., & Graff Zivin, J. S. (2019). Does science tional psychology and beyond. Educational Psychologist, 56(2),
advance one funeral at a time? The American Economic Review, 79–89. https://doi.org/10.1080/00461520.2021.1898961
109(8), 2889–2920. https://doi.org/10.1257/aer.20161574 Gelman, A. (2017). Beyond “power pose”: Using replication failures
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., and a better understanding of data collection and analysis to do
Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., better science. https://statmodeling.stat.columbia.edu/2017/10/18/
Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., beyond-power-pose-using-replication-failures-better-understanding-
Pfeiffer, T., Razen, M., & Wu, H. (2016). Evaluating replicability of data-collection-analysis-better-science/
laboratory experiments in economics. Science, 351(6280), 1433–1436. Gelman, A. (2018). Don’t characterize replications as successes or fail-
https://doi.org/10.1126/science.aaf0918 ures. The Behavioral and Brain Sciences, 41, e128–20. https://doi.
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., org/10.1017/S0140525X18000638
Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., Gernsbacher, M. A. (2018). Three ways to make replication main-
Altmejd, A., Buttrick, N., Chan, T., Chen, Y., Forsell, E., Gampa, A., stream. Behavioral and Brain Sciences, 41, 20. https://doi.org/10.
Heikensten, E., Hummer, L., Imai, T., … Wu, H. (2018). Evaluating 1017/S0140525X1800064X
the replicability of social science experiments in Nature and Science Gervais, W. M. (2021). Practical methodological reform needs good
between 2010 and 2015. Nature Human Behaviour, 2(9), 637–644. theory. Perspectives on Psychological Science. Advance online publi-
https://doi.org/10.1038/s41562-018-0399-z cation. https://doi.org/10.1177/1745691620977471
Carter, E. C., Sch€onbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019). Gorgolewski, K. J., Nichols, T., Kennedy, D. N., Poline, J. B., &
Correcting for bias in psychology: A comparison of meta-analytic Poldrack, R. A. (2018). Making replication prestigious. Behavioral
methods. Advances in Methods and Practices in Psychological and Brain Sciences, 41, Article e131. https://doi.org/10.1017/S01405
Science, 2(2), 115–144. https://doi.org/10.1177/2515245919847196 25X18000663
 EDUCATIONAL PSYCHOLOGIST 99

Guest, O., & Martin, A. (2021). How computational modeling can Necessary but far too rare. Remedial and Special Education, 37(4),
force theory building in psychological science. Perspectives on 205–212. https://doi.org/10.1177/0741932516646083
Psychological Science. Advance online publication. https://doi.org/10. Makel, M. C., Plucker, J. A., & Hegarty, C. B. (2012). Replications in
1177/1745691620970585 psychology research: How often do they really occur? Perspectives on
Hedges, L. V., & Schauer, J. M. (2019). More than one replication Psychological Science A Journal of the Association for Psychological
study is needed for unambiguous tests of replication. Journal of Science, 7(6), 537–542. https://doi.org/10.1177/1745691612460688
Educational and Behavioral Statistics, 44(5), 543–570. https://doi.org/ McNeeley, S., & Warner, J. J. (2015). Replication in criminology:
10.3102/1076998619852953 Necessary practice. European Journal of Criminology, 12(5), 581–597.
Hodges, C. (2015). Replication studies in educational technology. https://doi.org/10.1177/1477370815578197
TechTrends, 59(4), 3–4. https://doi.org/10.1007/s11528-015-0862-2 Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir
Howe, P. D., & Perfors, A. (2018). An argument for how (and why) to Ronald, and the slow progress of soft psychology. Journal of
incentivise replication. Behavioral and Brain Sciences, 41, 25–26. Consulting and Clinical Psychology, 46(4), 806–834. https://doi.org/
https://doi.org/10.1017/S0140525X18000705 10.1037/0022-006X.46.4.806
H€uffmeier, J., Mazei, J., & Schultze, T. (2016). Reconceptualizing repli- Mellor, D. (2021/this issue). Improving norms in research culture to
cation as a sequence of different studies: A replication typology. incentivize transparency and rigor. Educational Psychologist, 56(2),
Journal of Experimental Social Psychology, 66, 81–92. https://doi.org/ 122–131. https://doi.org/10.1080/00461520.2021.1902329
10.1016/j.jesp.2015.09.009 Merk, S., & Rosman, T. (2019). Smart but evil? student-teachers’ per-
Irvine, E. (2021). The role of replication studies in theory building. ception of educational researchers’ epistemic trustworthiness. AERA
Perspectives on Psychological Science. Advance online publication. Open, 5(3). https://doi.org/10.1177/2332858419868158
https://doi.org/10.1177/1745691620970558 Merk, S., & Rosman, T. (2020). Reflections on the registered report
Jacob, R. T., Doolittle, F., Kemple, J., & Somers, M.-A. (2019). A process for “Smart but evil? Student-teachers’ perception of educa-
framework for learning from null results. Educational Researcher, tional researchers’ epistemic trustworthiness. AERA Open, 6(2).
48(9), 580–589. https://doi.org/10.3102/0013189X19891955 https://doi.org/10.1177/2332858420918158
Kanai, R. (2016). Open questions in conducting confirmatory replica- Munaf o, M. (2019). Raising research quality will require collective
tion studies: Commentary on Boekel et al., 2015. Cortex; a Journal action. Nature, 576(7786), 183. https://doi.org/10.1038/d41586-019-
Devoted to the Study of the Nervous System and Behavior, 74, 03750-7
343–347. https://doi.org/10.1016/j.cortex.2015.02.020 National Science Foundation & Institute of Educational Sciences,
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahnık, S., U.S. Department of Education. (2018). A supplement to the common
Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, guidelines for education research and development. https://ies.ed.gov/
C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, pdf/CompanionGuidelinesReplicationReproducibility.pdf
T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., … Nosek, B. A. [@BrianNosek]. (2020, November 13). Summary press
Nosek, B. A. (2014). Investigating variation in replicability: a “many release [Twitter thread]. https://twitter.com/BrianNosek/status/
labs” replication project. Social Psychology, 45(3), 142–152. https:// 1327296776865525761
doi.org/10.1027/1864-9335/a000178 Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D.,
Kochari, A. R., & Ostarek, M. (2018). Introducing a replication-first Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G.,
rule for PhD projects (commmentary on Zwaan et al., ‘Making rep- Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R.,
lication mainstream’). Behavioral and Brain Sciences, 41, 28. https:// Goroff, D., Green, D. P., Hesse, B., Humphreys, M., … Yarkoni, T.
doi.org/10.1017/S0140525X18000730 (2015). SCIENTIFIC STANDARDS. Promoting an open research
Kuhn, T. S. (2012). The structure of scientific revolutions (4th ed.). culture. Science, 348(6242), 1422–1425. https://doi.org/10.1126/sci-
University of Chicago Press. ence.aab2374
Lemons, C. J., King, S. A., Davidson, K. A., Berryessa, T. L., Gajjar, Nosek, B. A., & Errington, T. M. (2019, September 10). What is repli-
S. A., & Sacks, L. H. (2016). An inadvertent concurrent replication. cation?. https://doi.org/10.1371/journal.pbio.3000691
Same roadmap, different journey. Remedial and Special Education, Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S.,
37(4), 213–222. https://doi.org/10.1177/0741932516631116 Dreber, A., Fidler, F., Hilgard, J., Kline, M., Nuijten, M. B.,
Leppink, J. (2017). Revisiting the quantitative-qualitative-mixed meth- Rohrer, J., Romero, F., Scheel, A., Scherer, L., Sch€ onbrodt, F., &
ods labels: Research questions, developments, and the need for repli- Vazire, S. (2021). Replicability, robustness, and reproducibility in psy-
cation. Journal of Taibah University Medical Sciences, 12(2), 97–101. chological science [preprint]. PsyArXiv. https://psyarxiv.com/ksfvq/
https://doi.org/10.1016/j.jtumed.2016.11.008 download?format=pdf
Machery, E. (2019, October 10). What is a replication? https://doi.org/ Open Science Collaboration. (2015). Estimating the reproducibility of
10.31234/osf.io/8x7yn psychological science. Science, 349(6251), Article aac4716. https://
Makel, M. C. (2014). The empirical march: Making science better at doi.org/10.1126/science.aac4716
self-correction. Psychology of Aesthetics, Creativity, and the Arts, Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown?
8(1), 2–7. https://doi.org/10.1037/a0035803 Three arguments examined. Perspectives on Psychological Science: A
Makel, M. C., & Plucker, J. A. (2014a). Creativity is more than novelty: Journal of the Association for Psychological Science, 7(6), 531–536.
Reconsidering replication as a creative act. Psychology of Aesthetics, https://doi.org/10.1177/1745691612463401
Creativity, and the Arts, 8(1), 27–29. https://doi.org/10.1037/ Patall, E. A. (2021/this issue). Implications of the open science era for
a0035811 educational psychology research syntheses. Educational Psychologist,
Makel, M. C., & Plucker, J. A. (2014b). Facts are more important than 56(2), 142–160. https://doi.org/10.1080/00461520.2021.1897009
novelty: Replication in the education sciences. Educational Peters, S. J., Rambo-Hernandez, K., Makel, M. C., Matthews, M., &
Researcher, 43(6), 304–316. https://doi.org/10.3102/ Plucker, J. A. (2019). The effect of local norms on racial and ethnic
0013189X14545513 representation in gifted education. AERA Open, 5(2). https://doi.org/
Makel, M. C., & Plucker, J. A. (2015). An introduction to replication 10.1177/2332858419848446
research in gifted education: Shiny and new is not the same as use- Piffer, D. (2019). Evidence for recent polygenic selection on educational
ful. Gifted Child Quarterly, 59(3), 157–164. https://doi.org/10.1177/ attainment and intelligence inferred from Gwas hits: A replication
0016986215578747 of previous findings using recent data. Psych, 1(1), 55–75. https://
Makel, M. C., & Plucker, J. A. (Eds.). (2017). Toward a more perfect doi.org/10.3390/psych1010005
psychology: Improving trust, accuracy, and transparency in research. Planck, M. K. (1949). Scientificautobiography and other papers.
American Psychological Association. Philosophical Library.
Makel, M. C., Plucker, J. A., Freeman, J., Lombardi, A., Simonsen, B., Plucker, J. A., Beghetto, R. A., & Dow, G. T. (2004). Why isn’t creativ-
& Coyne, M. (2016). Replication of special education research: ity more important to educational psychologists? Potentials, pitfalls,
100 J. A. PLUCKER AND M. C. MAKEL

and future directions in creativity research. Educational Psychologist, Spector, J. M., Johnson, T. E., & Young, P. A. (2015). An editorial on
39(2), 83–96. https://doi.org/10.1207/s15326985ep3902_1 replication studies and scaling up efforts. Educational Technology
Pratt, M. G., Kaplan, S., & Whittington, R. (2020). Editorial essay: The Research and Development, 63(1), 1–4. https://doi.org/10.1007/
tumult over transparency: Decoupling transparency from replication s11423-014-9364-3
in establishing trustworthy qualitative research. Administrative Spellman, B. A. (2015). A short (Personal) Future History of
Science Quarterly, 65(1), 1–19. https://doi.org/10.1177/ Revolution 2.0. Perspectives on Psychological Science: A Journal of the
0001839219887663 Association for Psychological Science, 10(6), 886–899. https://doi.org/
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2018). Replication in 10.1177/1745691615609918
criminology and the social sciences. Annual Review of Criminology, Stafford, T. (2018). Female chess players outperform expectations when
1(1), 19–38. https://doi.org/10.1146/annurev-criminol-032317-091849 playing men. Psychological Science, 29(3), 429–436. https://doi.org/
Protzko, J., & Schooler, J. W. (2020). No relationship between 10.1177/0956797617736887
researcher impact and replication effect: An analysis of five studies Stafford, T. [@TomStafford]. (2020, May 20). Is stereotype threat in
with 100 replications. PeerJ., 8, e8014. https://doi.org/10.7717/peerj. chess real after all? [Twitter thread] https://twitter.com/tomstafford/
8014
status/1263013074556071936
Reich, J. (2021/this issue). Preregistration and registered reports.
Steinhardt, I. (2020). Learning open science by doing open science. A
Educational Psychologist, 56(2), 101–109. https://doi.org/10.1080/
reflection of a qualitative research project-based seminar. Education
00461520.2021.1900851
for Information (preprint). https://content.iospress.com/download/
Reich, J., Gehlbach, H., & Albers, C. (2020, May). AERA Open special
education-for-information/efi190308?id=education-for-information%
topic on preregistered reports. AERA Open. https://journals.sagepub.
com/page/ero/collections/registered-reports 2Fefi190308
Rosenthal, R. (1969). On not so replicated experiments and not so null Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of
results. Journal of Consulting and Clinical Psychology, 33(1), 7–10. exact replication. Perspectives on Psychological Science: A Journal of
https://doi.org/10.1037/h0027231 the Association for Psychological Science, 9(1), 59–71. https://doi.org/
Schauer, J. M., & Hedges, L. V. (2021). Reconsidering statistical meth- 10.1177/1745691613514450
ods for assessing replication. Psychological Methods, 26(1), 127–139. Tierney, W., Hardy, J. H., Ebersole, C. R., Leavitt, K., Viganola, D.,
https://doi.org/10.1037/met0000302 Clemente, E. G., Gordon, M., Dreber, A., Johannesson, M., Pfeiffer,
Schindler, C., Veja, C., Hocker, J., Kminek, H., & Meier, M. (2020). T., Hiring Decisions Forecasting Collaboration., & Uhlmann, E. L.
Collaborative open analysis in a qualitative research environment. (2020). Creative destruction in science. Organizational Behavior and
Education for Information, 36(3), 215–247. https://doi.org/10/3233/ Human Decision Processes, 161, 291–309. https://doi.org/10.1016/j.
EFI-190261 obhdp.2020.07.002
Schlosberg, H. (1951). Repeating fundamental experiments. American Turner, J. C., & Nolen, S. B. (2015). Introduction: The relevance of the
Psychologist, 6(5), 177–177. https://doi.org/10.1037/h0056148 situative perspective in educational psychology. Educational
Schmidt, S. (2009). Shall we really do it again? The powerful concept Psychologist, 50(3), 167–172. https://doi.org/10.1080/00461520.2015.
of replication is neglected in the social sciences. Review of General 1075404
Psychology, 13(2), 90–100. https://doi.org/10.1037/a0015108 Valentine, J. (2019). Expecting and learning from null results.
Schmidt, S. (2017). Replication. In M. C. Makel & J. A. Plucker (Eds.), Educational Researcher, 48(9), 611–613. https://doi.org/10.3102/
Toward a more perfect psychology: Improving trust, accuracy, and 0013189X19891440
transparency in research (pp. 215–232). American Psychological van Rooij, I., & Baggio, G. (2021). Theory before the test: How to build
Association. https://doi.org/10.1037/0000033-015 high-verisimilitude explanatory theories in psychological science.
Simmons, J., Nelson, L. (2020, May 20). Do human-like products inspire Perspectives on Psychological Science. Advance online publication.
more holistic judgments? http://datacolada.org/87 https://doi.org/10.1177/1745691620970604
Simons, D. J. (2014). The value of direct replication. Perspectives Vartanian, O. (2017). The contributions of theory choice, cumulative
on Psychological Science, 9(1), 76–80. https://doi.org/10.1177/ science, and problem finding to scientific innovation and research
1745691613514755 quality. In M. C. Makel & J. A. Plucker (Eds.), Toward a more per-
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on
fect psychology: Improving trust, accuracy, and transparency in
Generality (COG): A proposed addition to all empirical papers.
research (pp. 13–31). American Psychological Association. https://
Perspectives on Psychological Science: A Journal of the Association for
doi.org/10.1037/0000033-002
Psychological Science, 12(6), 1123–1128. https://doi.org/10.1177/
Vazire, S. (2018). Implications of the credibility revolution for product-
1745691617708630
ivity, creativity, and progress. Perspectives on Psychological Science,
Simonsohn, U. (2015). Small telescopes: detectability and the evaluation
of replication results. Psychological Science, 26(5), 559–569. https:// 13(4), 411–417. https://doi.org/10.1177/1745691617751884
doi.org/10.1177/0956797614567341 Wentzel, K. R. (2021/this issue). Open science reforms: Strengths, chal-
Simonton, D. K. (2012). Taking the U.S. Patent Office criteria seriously: lenges, and future directions. Educational Psychologist, 56(2),
A quantitative three-criterion creativity definition and its implica- 161–173. https://doi.org/10.1080/00461520.2021.1901709
tions. Creativity Research Journal, 24(2–3), 97–106. https://doi.org/ Williams, R. T., Polanin, J. R., & Pigott, T. D. (2017). Meta-analysis
10.1080/10400419.2012.676974 and reproducibility. In. M. C. Makel & J. A. Plucker (Eds.), Toward
Smerdon, D., Hu, H., McLennan, A., von Hippel, W., & Albrecht, S. a more perfect psychology (pp. 255–270). American Psychological
(2020). Female chess players show typical stereotype-threat effects: Association.
Commentary on Stafford (2018). Psychological Science, 31(6), Yeo-Teh, N. S. L., & Tang, B. L. (2021). Research ethics courses as a vac-
756–759. https://doi.org/10.1177/0956797620924051 cination against a toxic research environment or culture. Research
Smith, J. K., Smith, L. F., & Smith, B. K. (2017). The reproducibility Ethics, 17(1), 55–65. https://doi.org/10.1177/1747016120926686
crisis in psychology: Attack of the clones or phantom menace? In Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making
M. C. Makel & J. A. Plucker (Eds.), Toward a more perfect psych- replication mainstream. Behavioral and Brain Sciences, 41, e120.
ology (pp. 273–287). American Psychological Association. https://doi.org/10.1017/S0140525X17001972