In short
A p-value is the probability of getting a result at least as extreme as yours if the null hypothesis were exactly true and the rest of your model (sampling, assumptions, no hidden bias) were correct. It is a probability about data given a hypothesis, not a probability about a hypothesis given data: p = P(data | H0), never P(H0 | data). So p = .011 does not mean “there is a 1.1% chance the null is true” or “a 98.9% chance my hypothesis is true”; it means that in a world where the effect is exactly zero, only 1.1% of repetitions of your study would separate the groups as far as yours did or further. The four standard misreadings — that p gives the probability of the null, that 1 − p gives the probability of the alternative, that a small p implies a large effect, and that a large p proves there is no effect — all come from flipping that conditional round or forgetting that sample size sits inside the calculation.
You can have three published analyses behind you and still freeze if somebody stops you mid-sentence and asks what the number you just typed actually measures. That is not a gap in your training. The correct definition sounds so strange the first time you meet it that the brain quietly swaps it for a friendlier version, and the swap survives undergrad, the master's and often the whole PhD. The trouble is that the friendly versions license different conclusions, and all four of them turn up in discussion sections that reviewers read. Here is the real definition, piece by piece, and then the four errors with what your result genuinely supports in each case.
The definition, in full, with nothing shaved off
A p-value is the probability of obtaining a test statistic at least as extreme as the one you observed, if the null hypothesis were exactly true and the rest of the statistical model were correct. Three clauses, all load-bearing, and nearly every misreading consists of dropping one of them on the way. “If the null hypothesis were exactly true”. The p-value is computed inside an assumption you usually do not believe: the effect is zero, exactly zero, and everything you can see in your data is sampling noise. That hypothetical world is where the calculation starts, not where it ends. Asking a p-value whether the null is true is asking it about the thing it had to assume in order to exist.
“At least as extreme”. It is not the probability of your particular result — for a continuous variable that probability is zero — but of your result together with every result further from the null than yours. That is why a p-value is the area of a tail (or two) rather than a point.
“And the rest of the model were correct”. This is the clause almost everybody drops, and Greenland and colleagues (2016) put it front and center: a p-value measures how compatible your data are with the whole model, not with the null alone. A small p says something in that package does not fit, and that something might be the null — or independence of observations, or the absence of selection bias, or the fact that you ran five analyses and are reporting one.
One worked case to anchor it. A trial compares an intervention against a waiting list with 400 people per arm and finds d = 0.18, t(798) = 2.55, p = .011. Read literally: if the intervention did nothing whatsoever and the study were clean, only 1.1% of repetitions would separate the arms as far as this or further. That is the entire content. It says nothing about how much the intervention helps, nothing about the probability that the null is false, and nothing about whether the finding will hold up next time.
Why the correct definition refuses to stick
Because brains flip conditionals without announcing it. Nearly every Olympic rower is tall; hardly anyone tall rows for their country. Two wildly different probabilities built from the same two words, and once the sentence turns technical we stop noticing which one we just said.
A p-value answers “how unusual would these data be if there were no effect”. What a researcher wants to know is “how likely is it that there is no effect, given these data”. Cohen (1994) wrote a whole paper about this gap: significance testing does not deliver what we came for, it delivers the other thing, and what we came for needs information the test never had. The mind then files the answer it got under the question it asked.
One fact helps the definition settle. If the null is true and the test statistic is continuous, the p-value is uniformly distributed between 0 and 1. Getting p = .04 in a world with no effect is not a miracle; it happens 4% of the time by construction, and p < .05 happens exactly 5% of the time. A low p-value is not an impossible event under the null. It is an uncommon one, at precisely the rate the number itself states.
Error 1: “the p-value is the probability that the null hypothesis is true”
This is the most common one and it seeds the other three. p = .03 gets read as “a 3% chance there is no effect”, and from there everything else follows neatly. Reaching that figure would take two things the test does not have: a prior probability that the null is true, and Bayes' theorem to update it with your data. Without a prior there is no P(H0 | data), and the p-value contains no prior anywhere.
When the calculation is done properly, the number is not even close to what people think they are reading. Sellke, Bayarri and Berger (2001) worked out the bound most generous to the alternative within a broad family of priors, and with p = .05 the Bayes factor favoring the alternative cannot exceed roughly 2.5; starting from even odds, the posterior probability of the null stays around .29. Not 5%: closer to one in three. At p = .01 the same bound leaves the null around .11.
These are bounds under specific assumptions and they shift with the prior, so do not treat them as an official conversion table. What does not shift is the direction: a p-value overstates the evidence against the null relative to what almost everyone hears when they read it. A result at p = .05 is not strong proof of anything. It is a hint that asks for replication.
Error 2: “1 − p is the probability that my hypothesis is true”
The same mistake seen from the other side, and the one that shows up most in discussion sections and vivas: “p = .03, so I am 97% confident in my hypothesis”. It sounds reasonable because the subtraction is easy, and it fails for two entirely separate reasons.
The first is arithmetic. The complement of a probability about data is still a probability about data: 1 − p is the probability of observing a less extreme result than yours in a world with no effect. Perfectly well defined, perfectly useless, and not a statement about your hypothesis in any sense.
The second is logical. The alternative hypothesis is not your theory. H1 says “the effect is not exactly zero”, and inside that sentence live d = 0.01 and d = 3.00, the effect in the direction you predicted and the mechanism that never crossed your mind, including whatever bias your sampling procedure introduced. Ruling out zero does not choose among the infinitely many explanations compatible with not-zero, and your substantive hypothesis is only one of them.
The same flip infects confidence intervals: the 95% is a long-run property of the procedure, not the probability that the parameter sits inside your particular interval. If that distinction feels blurry, I unpack it in how to interpret a confidence interval.
Error 3: “a small p-value means a large effect”
A p-value fuses two things that cannot be separated afterwards. In the commonest test, two independent groups of n people each, the statistic is t = d · √(n/2): one factor is the phenomenon you are studying, the other is how many people you managed to recruit. The p-value comes out of the product, so the same value can arise from a tiny effect measured in hundreds of people or a huge effect measured in a dozen.
Small and significant. The trial above, d = 0.18 with 400 per arm: t(798) = 2.55, p = .011, 95% CI for d [0.04, 0.32]. It clears the threshold comfortably, and the entire interval sits below what Cohen's (1988) convention calls a medium effect (d = 0.50).
Large and non-significant. A pilot with 12 per group and d = 0.80: t(22) = 1.96, p = .063, 95% CI [-0.03, 1.63]. The effect is more than four times the size of the trial's and the p-value is nearly six times worse. Nothing paradoxical is happening: that is the square root of n doing its job.
Very large samples give the extreme version. With r = .06 and N = 2500 you get t(2498) = 3.00, p = .003, and that “highly significant” relationship accounts for 0.36% of the variance. Asterisks make the reading worse, because *** looks like “very large” and only says “very incompatible with zero”. If the question is whether your effect is big, it gets answered by the effect size and how to interpret it, not by the p-value.
Error 4: “a large p-value proves there is no effect”
This one is the most expensive because it looks like the cautious option. “p = .29, therefore there is no difference between groups” is a positive claim extracted from an absence of evidence, and the test was never built to supply it: the null is not accepted, it merely fails to be rejected.
Put numbers on it. A study with 25 per group observes d = 0.30: t(48) = 1.06, p = .294, 95% CI for d [-0.26, 0.86]. That interval is compatible with a small effect in the wrong direction, with zero, and with a large effect in the predicted one. The study has learned almost nothing about magnitude, and writing “no effect” reports a conclusion the data do not support in either direction.
Claiming absence needs a design built for absence: decide in advance the threshold below which you would not care about the effect, then test against that threshold. With 200 per group and an observed d = 0.05 you get t(398) = 0.50, p = .617, 90% CI [-0.11, 0.21]; that interval fits entirely inside equivalence bounds of ±0.30, so there you can legitimately write that the effect, if any, is smaller than your threshold of relevance. Same large p-value, opposite conclusion, and what changed was the design. The full procedure is in the TOST equivalence test.
Until you do that, the honest sentence is not “there was no effect” but “this study cannot distinguish between no effect and an effect as large as d = 0.86”. Less quotable, and true.
What to write when a p-value has to go in the sentence
Exact value, to two or three decimals, p = .011 rather than p < .05, with the one practical exception of p < .001 when the number drops below that. Never p = .000: it is not zero, your software is rounding. The full formatting rules for decimals, leading zeros and italics are in how to report p-values in APA 7.
Never on its own. The minimum sentence that survives review carries the test statistic, its degrees of freedom, the exact p, the effect size and the confidence interval around that effect size. With those five elements a reader can rebuild your conclusion even if they do not share your threshold; with the p-value alone they can only take your word for it.
Drop “marginally significant” and “approaching significance” for a p of .06. You are using the .05 boundary when it suits you and continuous evidence when it does not. If .06 is nearly there, then .049 is nearly not there, and you cannot switch logics paragraph by paragraph depending on which one flatters the result.
Watch the verb, too. “The data are poorly compatible with the absence of an effect” describes what happened; “this demonstrates that the intervention works” does not. In the methods, state which alpha you are using and why: .05 is a convention inherited from Fisher, not a constant of nature, and there are serious proposals to lower it to .005 for new discoveries (Benjamin and colleagues, 2018) and to justify it case by case instead (Lakens and colleagues, 2018).
Frequently asked questions
Is the p-value the probability that my result is due to chance?
No, however often it is phrased that way. The p-value already assumes everything is chance — that is the null hypothesis — and works out how unusual your data would be inside that assumption. What it cannot do is give the probability that the assumption is the real world: that is P(H0 | data), and it requires a prior the test never supplied.
What does p = .05 actually mean?
That if the effect were exactly zero and the model were correct, 5% of repetitions of your study would land as far from zero as yours or further. It does not mean 95% confidence in your hypothesis, and it does not mean a 5% chance the null is true: under the bound most generous to the alternative (Sellke, Bayarri and Berger, 2001) with even prior odds, the posterior probability of the null at p = .05 stays around .29.
If I get p = .001, is my effect more important than one with p = .04?
Not necessarily, because the p-value depends on the effect and the sample size together (t = d · √(n/2) for two groups of n people each). A d = 0.18 with 400 per group gives p = .011, while a d = 0.80 with 12 per group gives p = .063: the second effect is far larger and has the worse p. Importance is compared through effect sizes and their intervals, never through p-values.
Does p = .01 mean there is a 99% chance the finding will replicate?
No. The probability that a replication comes out significant depends on the true effect and on the power of the replication study, not on the original p-value. A first study with p = .01 and a small sample can have well under a 50% chance of clearing the threshold again, especially when the original estimate was inflated by the significance filter itself.
Can I call p = .06 “marginally significant”?
Better not, and a growing number of journals flag it on the way in. You are invoking the .05 boundary when it helps and continuous evidence when it does not. What you can do is report the exact value with the effect size and its interval, then discuss the magnitude and the precision of your estimate without pretending you crossed a line you did not cross.
Why .05 and not some other number?
Historical convention. Fisher offered it as a convenient criterion rather than a constant, and the American Statistical Association's own statement (Wasserstein and Lazar, 2016) reminds everyone that scientific decisions should not rest on whether a p-value crosses a threshold. Some fields now work at .005 for new claims and others use .10 in exploratory phases; what is expected today is that you justify yours in the methods.
If this rearranged something, carry on here
The definition does not settle by being read once. It settles by watching it fail on concrete cases and checking which sentence is left standing. Two natural continuations from here. One is what to do with a result that does not clear the threshold, which is where the fourth error does the most damage and where most people stall: my results are not significant, what now. The other is why the field has spent thirty years trying to remove the p-value from the role of sole judge, and what is taking its place.
Goodbye to the tyranny of the p-value →Keep the whole sentence, because it is the only part worth memorizing: a p-value is the probability of seeing data at least as extreme as yours in a world where the effect is exactly zero. Anything that sounds like “the probability that my hypothesis is true” is reading that conditional backwards. When a sentence you have written feels shaky, read it aloud starting with “if the effect were exactly zero...”; if it stops making sense, it was not saying what you thought it said.