Topics / Read the research
Effect sizes and p values, explained
Published November 20, 2026. 3 sources at the foot of the page.
What is an effect size?
An effect size is the amount of a difference or a relationship a study found, stated in original units or as a number of standard deviations. Cumming (2014) puts it in one line: an effect size is simply an amount of anything of interest. Means, differences between means, frequencies, correlations, and many other familiar quantities are effect sizes. A p value, however, is not an effect size. The amount can be reported in original units, milliseconds or score points, or in a standardized, units-free measure, and effect sizes in original units may often be more readily interpreted.
Where you might meet it
- A report says children who were tutored scored eight points higher on a hundred point test, give or take five, and a parent can tell what that means.
- A headline says a new evening habit improved sleep, p =.03, and nobody in the article says by how many minutes.
- Two studies of the same question reach opposite verdicts, significant and not significant, and the difference between them turns out to be the number of people in each.
The p value is the other number, and Wasserstein and Lazar (2016), writing for the American Statistical Association, define it informally: a p value is the probability under a specified statistical model that a statistical summary of the data, such as the sample mean difference between two compared groups, would be equal to or more extreme than its observed value. Their first principle follows from that: p values can indicate how incompatible the data are with a specified statistical model, most often a model in which there is no difference between two groups or no relationship between a factor and an outcome. The smaller the p value, the greater the statistical incompatibility of the data with that null hypothesis, if the underlying assumptions used to calculate the p value hold.
The second and fifth principles say what the number leaves out. P values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone: the p value is a statement about data in relation to a specified hypothetical explanation, and is not a statement about the explanation itself. And a p value, or statistical significance, does not measure the size of an effect or the importance of a result. Statistical significance is not equivalent to scientific, human, or economic significance. Any effect, no matter how tiny, can produce a small p value if the sample size or measurement precision is high enough, large effects may produce unimpressive p values if the sample size is small or measurements are imprecise, and identical estimated effects will have different p values if the precision of the estimates differs.
The third principle is about the line at.05. Scientific conclusions and business or policy decisions should not be based only on whether a p value passes a specific threshold; a conclusion does not immediately become true on one side of the divide and false on the other, and a p value near 0.05 taken by itself offers only weak evidence against the null hypothesis. The statement closes with one sentence a reader can carry: no single index should substitute for scientific reasoning. So this course reads a result as two numbers, the effect size and the p value, and asks of each what it can say. The p value says how incompatible the data are with a model of no effect. The effect size says how big the thing is. Only the second answers the question most readers came with.
What happens to p when a study is run again
Cumming (2014) asks a simple question: if p reveals truth, and the experiment is replicated, doing everything the same except with a new random sample, should the second p value not reveal the same truth? He simulates 25 replications of an experiment with two independent groups of 32 people, in which the true difference is 10 units, a medium effect by Cohen’s convention, and the chance of reaching significance is.50, which he notes is typical of what is published in many fields in psychology. The 95% confidence intervals bounce around as expected. The p values range from less than.001 to.75. It seems, he writes, that p can take almost any value; this dance of the p values is astonishingly wide, and it persists in virtually every situation, even when power is high.
The lesson he draws is about a single study. Running a single experiment amounts to choosing randomly from an infinite sequence of replications like those in the figure, and a single p value gives virtually no information about the infinite sequence of p values. If an experiment gives a two-tailed p of.05, an 80% prediction interval for the p value of a replication runs from.00008 to.44: there is an 80% chance that the next p falls in that interval, a 10% chance it is smaller and a 10% chance it is larger, and that interval is independent of the sample size. Any calculated value of p could easily have been very different had the researchers merely taken a different sample, which is why guideline 9 of his 25 reads, in full: do not trust any p value.
Replication is usually scored the old way, as a success if the second study matches the significance status of the first, both below.05 or both above it. In the 25 simulated experiments, just 9 of the 24 pairs matched, 38 percent, and the long-run figure with power of.50 is 50 percent. Even with power of.80, only 68 percent of replications succeed by that criterion. A p value, in other words, gives only poor information about the likely result of a replication, while a 95% confidence interval, as the last chapter shows, is an 83% prediction interval for where the next estimate will land.
Why does the number keep its grip? Cumming, following Kirk, writes that null hypothesis significance testing prompts researchers to see the world as black or white, and to formulate research aims and conclusions in dichotomous terms: an effect is statistically significant or it is not, it exists or it does not. He suspects that the seductive appeal of declaring an effect statistically significant, the apparent but illusory certainty, is a large part of the problem, and he adds that the word itself does damage: a statistically significant effect in the results section becomes significant in the discussion or abstract, and significant shouts important. Kline recommended saying a statistical difference and omitting the word significant; Cumming’s safest plan is never to use the word at all. The alternative he names is estimation thinking, asking how large an effect is rather than whether it exists.
How big, in what units
The first step of Cumming’s eight-step strategy is to formulate research questions in estimation terms: ask how large is the effect, or to what extent, and avoid dichotomous expressions such as test the hypothesis of no difference. The second step is to identify the effect size that best answers the question; if the question asks about the difference between two means, that difference is the effect size. Asking how much naturally prompts a quantitative answer, and the answer comes in two kinds of units.
Original units are the units a thing was measured in: six points on a test, forty milliseconds, three nights a month, a share of fifty-three percent. Cumming notes that effect sizes in original units may often be more readily interpreted, and a reader who knows the test knows what six points on it means. A standardized effect size divides the difference by a standard deviation, so that studies using different measures can be compared and combined, which is usually necessary for meta-analysis. The best known is Cohen’s d, the difference between two means expressed as a number of standard deviations, which Cumming describes as a kind of z score. A d of 1 says the two group averages sit one standard deviation of the scores apart.
Cohen (1988), in his book on statistical power analysis, suggested 0.2, 0.5 and 0.8 as small, medium and large values of d, and.1,.3 and.5 for small, medium and large correlations, as Cumming (2014) reports them. Cohen emphasized that making a judgment in context should be preferred to these fallback benchmarks, and Cumming agrees: interpretation of effect sizes requires informed judgment in context, taking full account of the research situation, the manipulation, the participants and the research aims, and when interpreting an effect size, give reasons. A reliability correlation of.7 can be highly statistically significant, since zero is a completely inappropriate reference for a reliability figure, and may be judged in context to be terrible.
In everyday terms, then, an effect size answers the question a reader actually has. A medium effect of d = 0.5 says the typical member of one group sits half a standard deviation above the typical member of the other, so the two groups overlap heavily; a small effect of 0.2 says they overlap almost completely, and even a large effect of 0.8 leaves many people in the two groups looking alike. Cumming’s simulated experiment used exactly a medium effect, and its p values still ran from.001 to.75, which is the whole point: the size of an effect and the size of a p value are different facts about a study, and the first is the one that says what was found.
The more useful number
Cumming opens his case for estimation with a poll. Suppose support for Proposition X is 53%, in a poll with an error margin of 2%. Most readers immediately understand that the 53% came from a sample and conclude that it is a fair estimate of support in the population, and that the 2% suggests the largest likely error. Reporting the result that way, or as 53% with a 95% confidence interval of [51, 55], is natural and informative, and more informative than stating that support is statistically significantly greater than 50%, p below.01. The 53% is the point estimate, and the interval is the interval estimate, whose length indicates the precision of the estimate.
Confidence intervals and p values rest on the same statistical theory, and with practice it is easy to translate one into the other by noting where the interval falls in relation to zero. That does not make them equally useful. The interval is much more informative because it indicates the extent of uncertainty, in addition to providing the best point estimate of what we want to know. The margin of error, the length of one arm of the interval, indicates precision: a large margin of error indicates low precision and an uninformative experiment, and a small margin of error is gold. In Cumming’s fictitious two-group experiment the difference is 54.0 with an interval of [7.2, 100.8], which suggests that the experiment has low precision and is perhaps of little value, and that reading is a much better approach than declaring the result statistically significant, p =.024.
Intervals dance too. In the long run, 95% of such intervals will include the true value, and an unidentified 5% will miss; a single interval is most likely one that includes it, but it might not, so Cumming’s guideline 13 reads: interpret your single confidence interval, but bear in mind the dance. Unlike a p value, a single interval is informative about the whole sequence, because its length indicates approximately the extent of the bouncing around, and a 95% interval is an 83% prediction interval for the estimate a replication would give, a 5-in-6 chance that the next result lands inside it. If an interval is short and close to zero, so that every value in it is negligible, a reader can conclude that the true effect is, for practical purposes, zero, which is the estimation way to think about what used to be called accepting the null hypothesis.
What no number can do
- Neither number replaces judgment. Wasserstein and Lazar close with no single index should substitute for scientific reasoning, and Cumming ends his guidelines where he began, with knowledgeable judgment in context.
- A tiny sample defeats the interval. A confidence interval is of little practical use when samples are very small, and an interval chosen for being the shortest, or an effect reported for being the largest of several, is not typical of the dance and is practically uninterpretable.
- A standardized number hides its ruler. Cohen’s d is measured on a rubber ruler, since the standard deviation it divides by changes from sample to sample, so whenever a value of d appears, a reader needs to know how it was calculated before interpreting it.
- Benchmarks are fallbacks. Cohen’s small, medium and large were offered with the warning that judgment in context should be preferred, and Cumming adds that no guideline will be universally applicable; for correlations in particular, any such benchmarks are likely to be inappropriate in many contexts.
- One study is rarely the answer. A single study is rarely, if ever, definitive; Cumming’s meta-analytic thinking treats any one study as most likely contributing rather than determining, and the ASA statement asks researchers to weigh the design of a study, the quality of the measurements and the external evidence for the phenomenon alongside any p value.
Nothing in this course says a p value is worthless. It says what each number can carry. When a result arrives as a p value alone, ask how large the effect is, in what units, and how wide the interval around it runs. A reader who asks those three questions has read the result; a reader who stopped at.05 has read a verdict.
Where to go next on the path
Reading a study, beside this unit, is about the choices behind a single p value, how many comparisons were available before the one in the abstract was chosen, and the two units are meant to be read together.
Knowing what you don’t know is about the gap between how sure a judgment feels and how often it is right, and Confirmation bias is about what a reader does with a result they wanted to be true.
Sources
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
- Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7-29.
- Wasserstein, R. L., & Lazar, N. A. (2016). The ASA’s statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133.
Practice it on Learn
Learn teaches effect sizes in short games and lessons, with the same research as this page.
Practice effect sizesSee what Learn is
Your first lesson is free and needs no account. For adults 18 and over.
More topics
- How to read a psychology study
A false positive is the finding of an effect that is only noise, a result that clears the line for statistical significance and then fails to hold up.
- The replication crisis: when a finding repeats
Replication is the process of running a study again, with new people and the original method, to see whether the same result appears.
- The Dunning-Kruger effect
The Dunning-Kruger effect is the tendency for people with the least skill in an area to overestimate how well they are doing.