Topics / Read the research
The replication crisis: when a finding repeats
Published November 20, 2026. 2 sources at the foot of the page.
What is replication?
Replication is the process of running a study again, with new people and the original method, to see whether the same result appears. The Open Science Collaboration (2015), writing up the largest such effort in psychology to that date, puts the principle first: reproducibility is a core principle of scientific progress, and scientific claims should not gain credence because of the status or authority of their originator but by the replicability of their supporting evidence. Their term for the method is direct replication, the attempt to recreate the conditions believed sufficient for obtaining a previously observed finding, and it is the means of establishing reproducibility of a finding with new data. The word new matters. Reanalyzing the same spreadsheet checks the arithmetic; only new data can check the finding.
Where you might meet it
- Another laboratory repeats a famous experiment with fresh volunteers, and the effect that made headlines comes back at half its original size.
- A teacher tries a classroom technique from a well known paper, gets nothing, and wonders whether the paper or the classroom is to blame.
- Two laboratories study the same question, one finds an effect and one does not, and each has a story about why the other went wrong.
Klein and colleagues (2018) open Many Labs 2 with exactly that third scene. Suppose a researcher, Josh, conducts an experiment and finds that academic performance is reduced among participants who experience threat compared with those in a control condition. Another researcher, Nina, conducts the same study at her institution and finds no effect. The explanations that come to mind first are about people and places: Nina may have used a sample that differed in important ways from Josh’s sample, and the situational context in Nina’s lab might have differed in theoretically important but nonobvious ways from the context in Josh’s lab. Both explanations could be true. A less interesting, but real, possibility is that one of the researchers made an error in design or procedure that the other did not. Finally, the different results may be a function of sampling error: Nina’s result could be a false negative, or Josh’s result could be a false positive.
The Open Science Collaboration (2015) lists the same roads. A direct replication may not obtain the original result for a variety of reasons: known or unknown differences between the replication and original study may moderate the size of an observed effect, the original result could have been a false positive, or the replication could produce a false negative. Even research of exemplary quality may have irreproducible empirical findings because of random or systematic error. That is why one study, however carefully run, is a start rather than a verdict. As the Collaboration’s conclusion puts it, a single study almost never provides definitive resolution for or against an effect and its explanation. The two chapters that follow are about what happened when psychologists ran hundreds of studies again on purpose.
What the Reproducibility Project found
Starting in November 2011, the Open Science Collaboration (2015) built a protocol for selecting and conducting high quality replications and then followed it one hundred times. The sampling frame was the 2008 volume of three journals: Psychological Science, the Journal of Personality and Social Psychology, and the Journal of Experimental Psychology: Learning, Memory, and Cognition. Replication teams took articles from a small pool at a time, by default replicated the last experiment in each article, contacted the original authors for materials, wrote a protocol and an analysis plan, had both reviewed by the original authors and by other members of the project, registered the protocol publicly, ran the study and archived everything. In total, 100 replications were completed by 270 contributing authors, with an average statistical power of 0.92 to detect the original effect sizes.
Ninety-seven of the 100 original studies had reported a positive result. If every original effect were true and accurately estimated, that power would predict approximately 89 positive results in the replications. There were 35, or 36 percent. Mean effect sizes, expressed as correlations, fell from 0.403 in the original studies to 0.197 in the replications, and of the 99 pairs where both could be computed, 82 showed a stronger effect in the original. The authors warn against reading the first number alone: a key weakness of counting significant results is that it treats the 0.05 threshold as a bright-line criterion between replication success and failure. So they report four more indicators. The original effect size fell inside the replication’s 95 percent confidence interval in 47 percent of cases; the replication teams themselves answered yes, when asked whether the result had replicated, for 39 of the 100; and when original and replication were combined in a meta-analysis, 68 percent of the effects that could be combined still differed from zero. No single indicator sufficiently describes replication success, they write, but collectively these results offer a clear conclusion: a large portion of replications produced weaker evidence for the original findings despite using materials provided by the original authors, review in advance for methodological fidelity, and high statistical power to detect the original effect sizes.
Which findings held up? The strength of the original evidence predicted it. Among original studies with a p value below 0.02, 26 of 63, or 41 percent, reached significance again; of those with a p value between 0.02 and 0.04, 6 of 23 did, 26 percent; and of those above 0.04, 2 of 11, 18 percent. Almost two thirds, 20 of 32, of original studies with a p value below 0.001 had a significant replication. Larger original effects were more likely to come back. Tests of a main or simple effect replicated in 23 of 49 cases, 47 percent, while tests of an interaction did in just 8 of 37, 22 percent. Cognitive psychology effects replicated at 50 percent, 21 of 42, and social psychology effects at 25 percent, 14 of 55, partly because the original social effects were weaker. Surprising effects were less reproducible, as were effects for which it was more challenging to conduct the replication. And there was little evidence that perceived importance of the effect, expertise of the original or replication teams, or self-assessed quality of the replication accounted for meaningful variation in reproducibility.
The Collaboration offers one explanation for the general shrinkage. Low-power research designs combined with publication bias favoring positive results together produce a literature with upwardly biased effect sizes, which anticipates that replication effect sizes would be smaller than original studies on a routine basis, not because of differences in implementation but because the original study effect sizes are affected by publication and reporting bias, and the replications are not. The project asked a broad and shallow question, one replication each of many effects, and the authors point to a complementary narrow and deep approach: the Many Labs projects, in which many laboratories replicate a few effects at once. Klein and colleagues (2018) ran the second of those, and it is the next chapter. Innovation points out paths that are possible; replication points out paths that are likely; progress relies on both.
What Many Labs 2 found
Klein and colleagues (2018) replicated 28 classic and contemporary published findings with protocols that were peer reviewed in advance, before any data were collected, in the Registered Report format, and preregistered on the Open Science Framework. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories, 39 samples from the United States and 86 from elsewhere. The point was variation: if an effect depends on who is tested and where, running the same procedure in a hundred places should show it.
Using the conventional criterion of statistical significance, 15 of the 28 replications, 54 percent, provided evidence of a statistically significant effect in the same direction as the original finding. With a strict criterion of p below.0001, 14 still did, a reflection of the extremely high-powered design. Seven replications yielded effect sizes larger than the original ones, and 21 yielded smaller ones. The median Cohen’s d was 0.60 for the original findings and 0.15 for the replications. Sixteen replication effects were small, below 0.20, and 9 were in the direction opposite the direction of the original effect. Because there is no simple decision rule for declaring success or failure in replication, the authors scored each effect against five criteria: 10 effects were successfully replicated according to all the criteria that could be applied, 13 were unsuccessfully replicated according to all of them, and 5 varied in replication success depending on the criterion used, usually because the replication effect size was substantially smaller than the original effect size. The framing effect first reported by Tversky and Kahneman in 1981 with 181 people came back in 7,228 with a d of 0.40 and passed every criterion.
The main purpose of the project was to see how much effect sizes vary by sample and setting, and the answer was: less than the field assumes. Knowing the effect being studied provides much more information about effect size than does knowing the sample being studied. Just 11 of the 28 effects showed significant heterogeneity across samples, and most of these were among the effects with the largest overall magnitude; 19 of the 28 had an estimated tau near zero, an indication of minimal heterogeneity. Where no effect was observed overall, there was also very little evidence for heterogeneity among the samples, so, at least for the variation investigated here, heterogeneity across samples does not provide much explanatory power for failures to replicate. There was little evidence for systematic effects of task order or of running a study in the laboratory versus online. For 12 of the 28 effects, the original authors or other experts had named in advance a moderator or sample characteristic that might be needed to see the effect; evidence consistent with the hypothesized moderation appeared for one of the 12, weak or partial evidence for two, and little for the other nine. This does not mean that moderating effects do not occur, but it may mean that psychological theory is not yet advanced enough to predict them reliably.
That finding has a sharp edge. When a replication fails, the first defense is often a hidden moderator, something about the new sample or setting that switched the effect off, and the authors note how often that appeal is made when there have been no empirical tests of whether such moderators are operative. The present study suggests that dismissing failures to replicate as a consequence of such moderators without conducting independent tests of the hypothesized moderators is unwise. Their closing implication is a plain one: it is not sufficient to presume that moderating influences account for observed variation in a phenomenon. Cultural, sample, or procedural variation could be a reasonable hypothesis as an account for differences in observed effects, but it is not a credible hypothesis until it survives a confirmatory test. The authors are equally plain about their own limits. The 28 findings were not a random sample of any definable population, and the fact that half of them reproduced does not mean that 50 percent of all findings in psychology will be reproduced, or fail to be reproduced, under similar circumstances.
The questions to ask of a finding
Neither project hands a reader a rule, and both say so. What they give instead is a set of questions, each with evidence behind it, that separate a result that has been tested from a result that has been reported once. Before trusting a single finding, ask whether anybody has run the study again with new people, and what the effect did when they had.
Six questions
- Has it been directly replicated? Direct replication mainly provides evidence for the reliability of a result, the Open Science Collaboration (2015) writes, and reliability is the first thing to know. A result from one laboratory, one sample and one analysis has been reported, which is different from having been tested.
- How strong was the original evidence? In the Reproducibility Project, replication success was more consistently related to the original strength of evidence, such as original p value, effect size, and effect tested, than to characteristics of the teams and implementation of the replication. A p value just under.05, a small sample and an interaction effect are three warning signs in one.
- Was the result surprising? Surprising effects were less reproducible. A finding that upends what everyone believed is exactly the one that most needs a second run before it is believed.
- Were the replications fair tests? The replications in both projects used original materials, were reviewed by the original authors or by peers before data collection, were preregistered with analysis plans fixed in advance, and had high statistical power. A failed replication that lacked those things says less; a failed replication that had them says more.
- Does the effect hold across samples and settings? In Many Labs 2, Klein and colleagues (2018) found that variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied, and that heterogeneity appeared mostly among the large, consistently detectable effects. An effect that is large in one place and absent everywhere else was the exception, so a hidden moderator is a hypothesis to test rather than an explanation to accept.
- What did the effect size do? Replication effects were about half the size of the originals in the Reproducibility Project, and the median d fell from 0.60 to 0.15 in Many Labs 2. A finding can come back real and small, and the number that was reported first is usually the larger one.
What a replication cannot tell a reader
The projects are as careful about success as about failure. It is too easy to conclude that successful replication means that the theoretical understanding of the original finding is correct; if there are alternative explanations for the original finding, those alternatives could likewise account for the replication, and understanding is achieved through multiple, diverse investigations that provide converging support for a theoretical interpretation and rule out alternative explanations. It is also too easy to conclude that a failure to replicate a result means that the original evidence was a false positive, since unanticipated factors in the sample, setting, or procedure could still have altered the observed effect magnitudes, and Klein and colleagues add that the failures to replicate do not necessarily mean that the tested hypotheses are incorrect. Asked how many of the 100 effects they had established as true, the Open Science Collaboration answered zero, and how many as false, zero again, and called that the reality of doing science rather than a limitation of the project. Nothing in this course says a finding from one laboratory is wrong. It says that a finding that has repeated has earned something a finding that has not cannot have, and that a reader who asks whether it has repeated has asked the right first question.
Where to go next on the path
Reading a study is about the choices behind a single p value, and Effect sizes and p values is about why a result that came back smaller is still a result; the three units are meant to be read together. The framing effect, taught in Think clearer, is one of the 28 findings Many Labs 2 ran again.
Sources
- Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
- Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., and 183 others, with Nosek, B. A. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443-490.
Practice it on Learn
Learn teaches replication in short games and lessons, with the same research as this page.
Practice replicationSee what Learn is
Your first lesson is free and needs no account. For adults 18 and over.
More topics
- Effect sizes and p values, explained
An effect size is the amount of a difference or a relationship a study found, stated in original units or as a number of standard deviations.
- How to read a psychology study
A false positive is the finding of an effect that is only noise, a result that clears the line for statistical significance and then fails to hold up.
- Fast and slow thinking: System 1 and System 2
Cognitive reflection is the habit of checking the answer that comes to mind first before you give it.