Topics / Read the research

How to read a psychology study

Published November 20, 2026. 2 sources at the foot of the page.

What is a false positive?

A false positive is the finding of an effect that is only noise, a result that clears the line for statistical significance and then fails to hold up. Simmons, Nelson and Simonsohn (2011) call it perhaps the most costly error in research, the incorrect rejection of a null hypothesis, and open their paper with the reasons: once in the literature a false positive is persistent, because a null result has many possible causes and failures to replicate previous findings are never conclusive, and journals rarely publish a replication at all; it draws money and effort into research programs that lead nowhere; and a field known for publishing false positives risks losing its credibility.

Where you might meet it

Gelman and Loken (2014) start with the number itself. Researchers typically express the confidence in their data as a p-value, the probability that a perceived result is actually the result of random variation. By convention, a p-value below 0.05 is considered a meaningful refutation of the null hypothesis; however, such conclusions are less solid than they appear. The idea goes back to Ronald Fisher in the 1920s, working on crop variance in Hertfordshire, England, who offered p-values as a means of protecting researchers from declaring truth based on patterns in noise. In an ironic twist, they write, p-values are now often manipulated to lend credence to noisy claims based on small samples.

An abstract shows the result that made the paper: the comparison, the direction and the p-value. What it leaves out is how many comparisons were available before that one was chosen, and Simmons, Nelson and Simonsohn state the arithmetic plainly: the likelihood of at least one of many analyses producing a falsely positive finding at the 5% level is necessarily greater than 5%. A single p-value in an abstract says nothing about how many others were computed on the way to it.

So this course reads a study in three parts and in one order: the abstract, to learn what is claimed; the method section, to count the choices; the result, to ask whether the same analysis would have been run on different data. The abstract comes first because it is the claim, and a claim is a thing to check rather than a thing to believe.

What a method section can hide

Simmons, Nelson and Simonsohn name the culprit researcher degrees of freedom: the decisions a researcher makes while collecting and analyzing data, each of which can be settled in more than one justifiable way. Should more data be collected? Should some observations be excluded? Which conditions should be combined and which ones compared? Which control variables should be considered? Should specific measures be combined or transformed or both? It is rare, and sometimes impractical, for researchers to make all these decisions beforehand, and it is common and accepted practice to explore various analytic alternatives, to search for a combination that yields statistical significance, and to then report only what worked.

None of this needs a villain. The authors trace the behavior to two factors, ambiguity in how best to make these decisions and the researcher’s desire to find a statistically significant result, and they show how much ambiguity there is with one small decision, how to handle outliers in reaction times. In roughly 30 Psychological Science articles they found responses dropped as too fast when they were in the fastest 2.5%, or more than 2 standard deviations from the mean, or faster than 100, 150, 200 or 300 milliseconds. None of these decisions is necessarily incorrect, but that fact makes any of them justifiable, and so available to a researcher who wants a particular answer.

The paper measures what that flexibility costs with computer simulations of pure noise: 15,000 samples from a design with two conditions and 20 observations per cell, in which no effect exists. Testing two dependent variables instead of one raises the false-positive rate from 5 percent to 9.5 percent, nearly double. A researcher who tests at 20 observations per cell and, when the result falls short, adds 10 more per cell and tests again raises the rate to 7.7 percent, about half again. Adding a covariate, with or without its interaction, gives 11.7 percent. Running three conditions and reporting any two or all three gives 12.6 percent. Using all four together gives a rate of 60.7 percent, which the authors call a stunning 61% false-positive rate: a researcher is more likely than not to find a significant effect in data that contain none.

They then show the same thing with real participants. In Study 1, 30 undergraduates heard either a children’s song, Hot Potato, or a control song, Kalimba, and reported how old they felt; with father’s age as a covariate, the children’s song made them feel older, p =.033. In Study 2, 20 undergraduates heard either When I’m Sixty-Four or Kalimba and then gave their birth dates; by their birth dates, listeners to the Beatles song were nearly a year and a half younger, p =.040. These two studies were conducted with real participants, employed legitimate statistical analyses, and are reported truthfully, the authors write, and Study 2 supports a hypothesis that is necessarily false. Everything reported here actually happened.

Their fuller report of Study 2 shows what the short one hid: 34 participants rather than 20, a third song dropped from the report, a dozen measures never mentioned, among them how old people felt, how much they would enjoy eating at a diner, the square root of 100, their mother’s age and their political orientation, analyses run after every session of about 10 participants with no decision in advance about when to stop, and a covariate chosen from that list. Without father’s age as a covariate the difference was smaller and p was.33. The fuller version, the authors note, would be all but impossible to publish, which is the point.

Their remedy is six requirements for authors: decide the rule for terminating data collection before data collection begins and report it; collect at least 20 observations per cell or justify the cost of collecting more; list all variables collected; report all experimental conditions, including failed manipulations; if observations are eliminated, report the result with them included; and if an analysis includes a covariate, report the result without the covariate as well. They suggest one word for the lists, so that readers know they are complete: participants reported only their age and gender.

A reader can hold the list the other way around. A method section that states how many people were recruited and why, lists every measure, names every condition and shows the result with and without each exclusion and covariate has closed most of the doors a false positive walks through. A method section that does none of this has left them open, whether or not anybody walked through, and that is what a reader is counting when the method section comes before the result.

One analysis, many possible analyses

Gelman and Loken take the problem one step further. Searching through analyses for a significant one is the behavior the 2011 paper described, and they note it has since been called p-hacking. Their point is that it is possible to have multiple potential comparisons, that is, a data analysis whose details are highly contingent on data, invalidating published p-values, without the researcher performing any conscious procedure of fishing through the data or explicitly examining multiple comparisons.

Their example is a hypothetical researcher who asks whether supporters of two parties perform differently on a short math test when it is posed in two different contexts. The pattern could show among men and not women, among women and not men, in one context and not the other, or only in a difference between the two differences, and each outcome has a ready explanation under the theory. A single overarching research hypothesis corresponds to many possible choices of a decision variable, and whichever one the data favor will look like the natural test to run.

Given a particular data set, it can seem entirely appropriate to look at the data and construct reasonable rules for data exclusion, coding, and analysis that can lead to statistical significance. In such a case, researchers need to perform only one test, but that test is conditional on the data, with the same effect as if they had deliberately fished for those results. In this garden of forking paths, whatever route you take seems predetermined, but that is because the choices are done implicitly. The mistake is in thinking that, if the particular path that was chosen yields statistical significance, this is strong evidence in favor of the hypothesis.

Gelman and Loken read a 2013 study that reported a significant interaction with no significant main effect, and observe that had the main effect appeared, in either direction, a theoretically justified explanation would have been ready for that too, and had neither appeared, other interactions were there to look at. Nothing in such a report suggests fishing, they write, and it is reasonable for scientists to refine their hypotheses in light of the data. The multiplicity is in the paths the data could have taken, whether or not anybody walked more than one.

They also report what happens when a team closes the garden before publishing. In 2013 Brian Nosek, Jeffrey Spies and Matt Motyl found a large and statistically significant relationship between political extremism and the perception of images as black or white rather than intermediate shades. Rather than declaring victory, they gathered a large new sample and performed a replication with predetermined protocols and data analysis. By their initial estimate the replication had a 99 percent chance of reaching statistical significance; its p-value was 0.59. The moral Gelman and Loken draw is that a statistically significant p-value cannot be taken at face value, even when it belongs to a comparison that fits an existing theory.

That gives a reader one question for the result section, in Gelman and Loken’s words: Would the same data-analysis decisions have been made with a different data set? Read the method section before the result, and ask of every choice in it whether the same choice would have been made with different data. A choice that would have gone another way with other numbers is a fork, and a result at the end of several forks is one path through a garden, however natural the path looks in print.

Limits, and where this goes next

A false positive is a property of a result, never a verdict on the people who published it, and this course stops where its two sources stop.

Gelman and Loken end with a line from the statistician Frederick Mosteller: it is easy to lie with statistics, but easier without them. Nothing in this course says that a significant result is worthless. It says that a result is a claim, that the method section is where its choices live, and that a reader who counts them is harder to fool.

Where to go next on the path

Confirmation bias, on this track, is about what a reader does with a result they wanted to be true, and Two ways of thinking is about the second look that catches a tidy answer before it is accepted.

Knowing what you don’t know is about the gap between how sure a judgment feels and how often it is right, which is the gap a p-value was invented to measure.

Sources

  1. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366.
  2. Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460-465.

Practice it on Learn

Learn teaches reading a study in short games and lessons, with the same research as this page.

Practice reading a studySee what Learn is

Your first lesson is free and needs no account. For adults 18 and over.

More topics