Topics / Read the research
How to read a psychology study
Published November 20, 2026. 2 sources at the foot of the page.
What is a false positive?
A false positive is the finding of an effect that is only noise, a result that clears the line for statistical significance and then fails to hold up. Simmons, Nelson and Simonsohn (2011) call it perhaps the most costly error in research, the incorrect rejection of a null hypothesis, and open their paper with the reasons: once in the literature a false positive is persistent, because a null result has many possible causes and failures to replicate previous findings are never conclusive, and journals rarely publish a replication at all; it draws money and effort into research programs that lead nowhere; and a field known for publishing false positives risks losing its credibility.
Where you might meet it
- A headline reports a study in which a morning routine sharpened memory, the abstract gives one p-value, and nobody can see how many comparisons were tried.
- A friend repeats a finding from a podcast with the p-value attached, as if the number had settled the matter.
- A manager reads the abstract of a study, skips the method section, and changes a policy on the strength of the result.
Gelman and Loken (2014) start with the number itself. Researchers typically express the confidence in their data as a p-value, the probability that a perceived result is actually the result of random variation. By convention, a p-value below 0.05 is considered a meaningful refutation of the null hypothesis; however, such conclusions are less solid than they appear. The idea goes back to Ronald Fisher in the 1920s, working on crop variance in Hertfordshire, England, who offered p-values as a means of protecting researchers from declaring truth based on patterns in noise. In an ironic twist, they write, p-values are now often manipulated to lend credence to noisy claims based on small samples.
An abstract shows the result that made the paper: the comparison, the direction and the p-value. What it leaves out is how many comparisons were available before that one was chosen, and Simmons, Nelson and Simonsohn state the arithmetic plainly: the likelihood of at least one of many analyses producing a falsely positive finding at the 5% level is necessarily greater than 5%. A single p-value in an abstract says nothing about how many others were computed on the way to it.
So this course reads a study in three parts and in one order: the abstract, to learn what is claimed; the method section, to count the choices; the result, to ask whether the same analysis would have been run on different data. The abstract comes first because it is the claim, and a claim is a thing to check rather than a thing to believe.
What a method section can hide
Simmons, Nelson and Simonsohn name the culprit researcher degrees of freedom: the decisions a researcher makes while collecting and analyzing data, each of which can be settled in more than one justifiable way. Should more data be collected? Should some observations be excluded? Which conditions should be combined and which ones compared? Which control variables should be considered? Should specific measures be combined or transformed or both? It is rare, and sometimes impractical, for researchers to make all these decisions beforehand, and it is common and accepted practice to explore various analytic alternatives, to search for a combination that yields statistical significance, and to then report only what worked.
None of this needs a villain. The authors trace the behavior to two factors, ambiguity in how best to make these decisions and the researcher’s desire to find a statistically significant result, and they show how much ambiguity there is with one small decision, how to handle outliers in reaction times. In roughly 30 Psychological Science articles they found responses dropped as too fast when they were in the fastest 2.5%, or more than 2 standard deviations from the mean, or faster than 100, 150, 200 or 300 milliseconds. None of these decisions is necessarily incorrect, but that fact makes any of them justifiable, and so available to a researcher who wants a particular answer.
The paper measures what that flexibility costs with computer simulations of pure noise: 15,000 samples from a design with two conditions and 20 observations per cell, in which no effect exists. Testing two dependent variables instead of one raises the false-positive rate from 5 percent to 9.5 percent, nearly double. A researcher who tests at 20 observations per cell and, when the result falls short, adds 10 more per cell and tests again raises the rate to 7.7 percent, about half again. Adding a covariate, with or without its interaction, gives 11.7 percent. Running three conditions and reporting any two or all three gives 12.6 percent. Using all four together gives a rate of 60.7 percent, which the authors call a stunning 61% false-positive rate: a researcher is more likely than not to find a significant effect in data that contain none.
They then show the same thing with real participants. In Study 1, 30 undergraduates heard either a children’s song, Hot Potato, or a control song, Kalimba, and reported how old they felt; with father’s age as a covariate, the children’s song made them feel older, p =.033. In Study 2, 20 undergraduates heard either When I’m Sixty-Four or Kalimba and then gave their birth dates; by their birth dates, listeners to the Beatles song were nearly a year and a half younger, p =.040. These two studies were conducted with real participants, employed legitimate statistical analyses, and are reported truthfully, the authors write, and Study 2 supports a hypothesis that is necessarily false. Everything reported here actually happened.
Their fuller report of Study 2 shows what the short one hid: 34 participants rather than 20, a third song dropped from the report, a dozen measures never mentioned, among them how old people felt, how much they would enjoy eating at a diner, the square root of 100, their mother’s age and their political orientation, analyses run after every session of about 10 participants with no decision in advance about when to stop, and a covariate chosen from that list. Without father’s age as a covariate the difference was smaller and p was.33. The fuller version, the authors note, would be all but impossible to publish, which is the point.
Their remedy is six requirements for authors: decide the rule for terminating data collection before data collection begins and report it; collect at least 20 observations per cell or justify the cost of collecting more; list all variables collected; report all experimental conditions, including failed manipulations; if observations are eliminated, report the result with them included; and if an analysis includes a covariate, report the result without the covariate as well. They suggest one word for the lists, so that readers know they are complete: participants reported only their age and gender.
A reader can hold the list the other way around. A method section that states how many people were recruited and why, lists every measure, names every condition and shows the result with and without each exclusion and covariate has closed most of the doors a false positive walks through. A method section that does none of this has left them open, whether or not anybody walked through, and that is what a reader is counting when the method section comes before the result.
One analysis, many possible analyses
Gelman and Loken take the problem one step further. Searching through analyses for a significant one is the behavior the 2011 paper described, and they note it has since been called p-hacking. Their point is that it is possible to have multiple potential comparisons, that is, a data analysis whose details are highly contingent on data, invalidating published p-values, without the researcher performing any conscious procedure of fishing through the data or explicitly examining multiple comparisons.
Their example is a hypothetical researcher who asks whether supporters of two parties perform differently on a short math test when it is posed in two different contexts. The pattern could show among men and not women, among women and not men, in one context and not the other, or only in a difference between the two differences, and each outcome has a ready explanation under the theory. A single overarching research hypothesis corresponds to many possible choices of a decision variable, and whichever one the data favor will look like the natural test to run.
Given a particular data set, it can seem entirely appropriate to look at the data and construct reasonable rules for data exclusion, coding, and analysis that can lead to statistical significance. In such a case, researchers need to perform only one test, but that test is conditional on the data, with the same effect as if they had deliberately fished for those results. In this garden of forking paths, whatever route you take seems predetermined, but that is because the choices are done implicitly. The mistake is in thinking that, if the particular path that was chosen yields statistical significance, this is strong evidence in favor of the hypothesis.
Gelman and Loken read a 2013 study that reported a significant interaction with no significant main effect, and observe that had the main effect appeared, in either direction, a theoretically justified explanation would have been ready for that too, and had neither appeared, other interactions were there to look at. Nothing in such a report suggests fishing, they write, and it is reasonable for scientists to refine their hypotheses in light of the data. The multiplicity is in the paths the data could have taken, whether or not anybody walked more than one.
They also report what happens when a team closes the garden before publishing. In 2013 Brian Nosek, Jeffrey Spies and Matt Motyl found a large and statistically significant relationship between political extremism and the perception of images as black or white rather than intermediate shades. Rather than declaring victory, they gathered a large new sample and performed a replication with predetermined protocols and data analysis. By their initial estimate the replication had a 99 percent chance of reaching statistical significance; its p-value was 0.59. The moral Gelman and Loken draw is that a statistically significant p-value cannot be taken at face value, even when it belongs to a comparison that fits an existing theory.
That gives a reader one question for the result section, in Gelman and Loken’s words: Would the same data-analysis decisions have been made with a different data set? Read the method section before the result, and ask of every choice in it whether the same choice would have been made with different data. A choice that would have gone another way with other numbers is a fork, and a result at the end of several forks is one path through a garden, however natural the path looks in print.
Limits, and where this goes next
A false positive is a property of a result, never a verdict on the people who published it, and this course stops where its two sources stop.
- Flexibility is not fraud. This exploratory behavior is not the by-product of malicious intent, Simmons, Nelson and Simonsohn write, and they assume that the vast majority of researchers strive for honesty. Gelman and Loken add that the very fact that scientists generally do not cheat and generally do not go fishing is what makes them vulnerable to believing a pattern robust enough to cross the 0.05 threshold.
- A reader cannot see the file drawer. The six requirements cannot reveal the degrees of freedom that come from reporting only the experiments that worked, and no method section lists the studies that were never written up.
- Small studies with perfect results deserve the hardest look. Reviewers should be more tolerant of imperfections in results, the 2011 paper says, because underpowered studies with perfect results are the ones that should invite extra scrutiny; and Gelman and Loken place the risk from multiple comparisons highest where there are small effect sizes, small sample sizes, large measurement errors, and high variation, and lowest where differences are large, samples are large and measurement is precise.
- Some repairs exist, and none is universal. Preregistration might make sense in a field like psychology, where more data are not hard to get, though Gelman and Loken say it cannot realistically be a general solution; prepublication replication pairs an exploratory study with a second one that is purely confirmatory with its own preregistered protocol; and analyzing all relevant comparisons, not just whatever happens to be statistically significant, is the repair for data that cannot be gathered again. When a justification is not compelling, a reviewer may ask for an exact replication, and the 2011 authors add that never is too selective; a reviewer may also ask to see a result under the other reasonable way of settling an arbitrary choice, because if a result holds only for one arbitrary specification, then everyone involved has learned a great deal about how robust the effect is.
- A failed replication is a reason to doubt, never a verdict. Because a null result has many possible causes, a replication that finds nothing suggests that the original was a false positive without proving it, which is why the reading in this course rests on the method section rather than on waiting for a replication.
Gelman and Loken end with a line from the statistician Frederick Mosteller: it is easy to lie with statistics, but easier without them. Nothing in this course says that a significant result is worthless. It says that a result is a claim, that the method section is where its choices live, and that a reader who counts them is harder to fool.
Where to go next on the path
Confirmation bias, on this track, is about what a reader does with a result they wanted to be true, and Two ways of thinking is about the second look that catches a tidy answer before it is accepted.
Knowing what you don’t know is about the gap between how sure a judgment feels and how often it is right, which is the gap a p-value was invented to measure.
Sources
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366.
- Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460-465.
Practice it on Learn
Learn teaches reading a study in short games and lessons, with the same research as this page.
Practice reading a studySee what Learn is
Your first lesson is free and needs no account. For adults 18 and over.
More topics
- Effect sizes and p values, explained
An effect size is the amount of a difference or a relationship a study found, stated in original units or as a number of standard deviations.
- The replication crisis: when a finding repeats
Replication is the process of running a study again, with new people and the original method, to see whether the same result appears.
- The framing effect
The framing effect is the tendency to decide the same choice differently depending on whether its outcomes are described as gains or as losses.