What a Confidence Interval Actually Is (and What It Is Not)

The sentence turns up again and again in the theses that reach my desk: “there is a 95% probability that the true effect lies between 0.24 and 5.56”. It is readable, it sounds careful, and it is wrong. The awkward part is not the mistake itself but that the correct version looks like pedantry with no consequences — right up until an examiner or a reviewer asks what your interval licenses you to conclude. That is where it becomes obvious who understands the number and who copied the brackets out of the output. Here is what a confidence interval actually is, what that 95% is a statement about, why it tells you far more than a p value, and how to read its width, which is the information most Results sections throw away.

The 95% is not about your interval

In the classical framework, the thing you are estimating — the real difference between two treatments, the real correlation in the population — is a fixed unknown number. It is not a random variable: it does not move, it has no distribution, nothing happens to it with a probability attached. What moves from study to study is your interval, because it is built out of your sample, and a different sample would have given you a different one. So once you have computed [0.24, 5.56], the probability that the parameter is inside is not .95. It is 1 or it is 0, and you do not know which.

Neyman (1937) constructed intervals on exactly that logic: a rule which, applied systematically, yields intervals that cover the parameter 95% of the time. The 95% qualifies the rule, not the item that came off the production line today. It is the same distinction as knowing a diagnostic test is right in 95% of cases and knowing whether it was right about you: the first is knowable, the second is not.

This is not a beginner's slip, and it is worth saying so. Hoekstra et al. (2014) put six false statements about a confidence interval in front of undergraduates, PhD students and researchers with years of publications behind them. All three groups endorsed several of them, and experience offered no protection. If you have written the sentence, you are in very large and very well-published company.

What you can say without lying is duller and stronger: the values inside the interval are those your data do not rule out at that level, and the values outside are those they do. That is the version that survives a follow-up question.

What coverage actually promises, and when it breaks

Picture running your study 100 times, with 100 fresh samples from the same population and the same analysis. Each replication gives a different interval: some wider, some narrower, each centered somewhere slightly different. Roughly 95 of those 100 will contain the true value and roughly 5 will miss it completely. That is coverage, and it is the whole of what the confidence level promises. You hold one of the hundred, and there is no way to know whether yours is one of the good ones.

The promise comes with conditions, and they are the model's conditions: random sampling, the distributional assumptions of the test, measurement without systematic bias, and an analysis fixed before the data were seen. If you tried three outcome measures and reported the interval for whichever behaved best, your coverage is no longer 95%, whatever the software prints above the column. Greenland et al. (2016) hammer this point because it is the one that gets dropped: the interval models sampling error and nothing else. A sample that quietly became second-year undergraduates, 30% attrition at follow-up, or a questionnaire measuring something adjacent to your construct are all sources of error the interval cannot see and will never warn you about.

There is a promise it has never made either: where the next study's mean will land. Cumming and Maillardet (2006) worked it out — a 95% interval captures the mean of an exact replication about 83% of the time, not 95%. An interval is a claim about the parameter, not a fence around future results, and the next sample brings its own noise on top.

Why an interval tells you more than a p value

Start with the exact relationship between them, which rarely gets spelled out: a 95% interval contains precisely those null values you would fail to reject in a two-sided test at alpha = .05. Zero inside means p > .05; zero outside means p < .05. An interval is not a p value with decoration, it is an entire family of tests at once — it tells you what would happen to every null hypothesis you might have posed, not only the one at zero.

Put numbers on it. Two groups of 52 on PHQ-9 total scores, a mean difference of 2.90 points, SE = 1.34: t(102) = 2.16, p = .033, d = 0.42, 95% CI [0.24, 5.56]. The p value tells you one thing and one only: zero does not hold up against these data. The interval says that too, and then supplies the rest of the story — your data are as compatible with a 0.24-point difference, which no clinician or patient would ever detect, as with a 5.56-point difference, which on the PHQ-9 is wider than a whole severity band.

It changes the sentence you are allowed to write. With the p value alone, the available line is “there was a significant difference between groups”, which sounds like a finding and commits you to nothing. With the interval in view you have to state what you cannot rule out, and the reviewer gets to judge it. That is why the APA Publication Manual (2020) wants the interval next to the effect size rather than buried in a footnote, and why the reporting details matter: exact values, decimals and leading zeros are laid out in how to report p values in APA 7.

The width is the part nobody reads

Two non-significant results can mean opposite things, and only the width separates them. A pilot with 18 per group on the same scale: difference 3.80 points, SE = 2.5, t(34) = 1.52, p = .138, 95% CI [-1.28, 8.88]. The interval runs from “the intervention makes things slightly worse” to “the intervention produces a large improvement”. That is not evidence of no effect; it is evidence that this study could not see anything. The honest sentence is not “there was no effect”, it is “the design lacked the precision to decide”, and the brackets are what make it defensible.

Now the mirror image. A correlation of r = .05 between daily screen time and depressive symptoms with N = 3500: t(3498) = 2.96, p = .003, 95% CI [.02, .08]. Significant, and the entire interval sits below what Cohen (1988) labeled a small effect. Here the question genuinely is settled; the answer is just boring. Whatever is there, it is tiny. A narrow interval closes a question even when it brackets a dull value; a wide one closes nothing, whichever side of .05 it lands on. Those small, medium and large labels are conventions rather than laws, and the wider argument for judging a result by its magnitude instead of by which side of a threshold it fell on is made in the case against p value tyranny.

Width is not mysterious, it is arithmetic: the margin on each side is the critical value times the standard error, and the standard error of a mean is the standard deviation over the square root of n. With SD = 12, moving from n = 60 to n = 240 takes the margin from ±3.10 to ±1.53. Four times the sample to halve the interval — that is the real exchange rate, and it explains why recruiting another twenty participants at the end almost never changes the picture.

Which gives the most useful practical consequence here: the width of your intervals is decided at the design stage, not in the analysis. When you plan a sample size you are also deciding, without noticing, how much imprecision you will have to defend in the discussion. Powering a study to detect an effect and powering it to estimate that effect with a margin anybody could act on give different numbers, and the second is almost always larger.

Not every value inside the interval is equally plausible

The second common misreading treats the interval as a plateau where every value is equally credible and beyond which there is nothing. It does not work that way. The point estimate is the value most compatible with your data, and compatibility falls off smoothly as you move toward the ends — and keeps falling, rather than stopping, just past them. The 95% limits are not a cliff edge; they are a cut placed on a continuous curve.

Watch it happen with the same data. That 2.90-point difference with SE = 1.34 gives 90% CI [0.68, 5.12], 95% CI [0.24, 5.56] and 99% CI [-0.62, 6.42]. Three nested intervals, one study, identical numbers: the 99% includes zero and the 95% does not. Nothing changed in the world — you moved the cut. This is why Greenland et al. (2016) argue for calling them compatibility intervals, and why Amrhein, Greenland and McShane (2019) took the argument to Nature.

The immediate consequence is the one that rescues the most manuscripts: zero sitting inside your interval is not evidence that the effect is zero. Zero is one compatible value among many, usually flanked by values that would matter a great deal — in the pilot above it shared the interval with 8.88, which would have changed the entire conclusion. To claim absence of an effect you have to state a threshold of irrelevance in advance and show the whole interval fits inside it, which is an equivalence test and runs backwards from what you are used to: TOST equivalence testing walks through it.

Two overlapping intervals can still differ

The trap springs the moment you draw error bars. Two groups, the bars touch slightly, and somebody concludes the groups do not differ. The numbers disagree. Two groups of 90 on the emotional exhaustion subscale: group A M = 31.5, 95% CI [28.9, 34.1]; group B M = 27.2, 95% CI [24.7, 29.7]. The intervals overlap between 28.9 and 29.7. And the difference between them is 4.30 points, 95% CI [0.69, 7.91], t(178) = 2.35, p = .020.

The reason is that the interval for a difference is not assembled by stapling two intervals together. The standard error of a difference is the square root of the sum of the two squared standard errors, which is always smaller than their sum. Cumming and Finch (2005) turned this into a rule of eye: for independent groups of similar size, an overlap of up to about half a bar still corresponds to p ≈ .05, and two intervals that do not overlap at all are separated comfortably, closer to p = .01 than to .05.

Keep the operating rule: to compare two conditions, compute and report the interval for the difference. Eyeballing the gap between two separate intervals is the visual version of an analysis you have not run. The same applies to the narrative form of the error — “it was significant in women and not in men, therefore the effect differs” — because comparing significance is not comparing effects, and the difference between significant and non-significant is not itself significant (Gelman and Stern, 2006). If you want to claim the groups differ, test the interaction.

How to read an interval in three steps

Step 1. Say both ends out loud, in units that mean something. “From 0.24 to 5.56 PHQ-9 points”, not “the interval excludes zero”. If you cannot say what the lower end would mean and what the upper end would mean for the person who has to use your result, you do not yet know what you found. Correlations and odds ratios need the same translation, with the extra wrinkle that an odds ratio interval is asymmetric and a correlation interval travels through Fisher's z.

Step 2. Ask which decisions change between the two ends. If you would act the same way at both ends — roll the intervention out, or shelve it — your study has answered, whatever the p value did. If one end points one way and the other end the opposite way, your study has not answered, and the honest sentence names the imprecision instead of hiding it behind an asterisk. This step is what turns an interval into something a non-statistician can use.

Step 3. Ask what that coverage is assuming. That the analysis was settled beforehand, that the sample represents the population you are talking about, that the model is reasonable. A 95% interval computed on the best of five analyses you tried is not a 95% interval, and no formula will tell you so — only you and your lab notebook know.

Written down it costs almost nothing: 95% CI [0.24, 5.56], sitting next to the estimate, with the level stated and the same number of decimals as the statistic itself. And if the interval you are holding brackets zero and you now have to write the thing up, the same three steps become a whole Results section in what to do when your results are not significant.

Frequently asked questions

What does the 95% in a confidence interval actually mean?

That the procedure you used produces, over repeated samples in the long run, intervals containing the true value 95% of the time. It is a property of the rule, not of your particular result. Your interval is already computed and either contains the parameter or does not; the 95% describes how often the method is right, not how likely you are to be right today.

Can I say there is a 95% probability the effect lies in my interval?

No, and it is one of the most repeated errors in applied statistics (Morey et al., 2016). In the classical framework the parameter is a fixed number, so it has no probability of being inside: it is or it is not. The correct phrasing is that the values inside are compatible with your data at that level and the values outside are ruled out. If you genuinely wanted a probability statement about the parameter, that is a Bayesian credible interval — different machinery, and it needs a prior.

Does an interval containing zero mean there is no effect?

No. It means zero is one of the values compatible with your data, almost always alongside values that would matter. In the pilot above, 95% CI [-1.28, 8.88] includes zero and also includes 8.88, which would have changed the whole conclusion. To claim absence you have to set a threshold of irrelevance in advance and show the entire interval falls inside it, which is what an equivalence test does.

Why is my confidence interval so wide?

Because the margin is the critical value times the standard error, and the standard error depends on how variable your measure is and on the square root of your sample size. A wide interval usually means a small sample, a noisy measure, or both. The arithmetic is unforgiving: you need four times the sample to halve the interval, so adding a handful of participants at the end will not rescue it.

If two confidence intervals overlap, are the groups not different?

That does not follow. The interval for a difference is not the union of two intervals, because the standard error of a difference is smaller than the sum of the two. Means of 31.5 [28.9, 34.1] and 27.2 [24.7, 29.7] overlap and still differ: 4.30, 95% CI [0.69, 7.91], p = .020. Compare conditions using the interval for the difference, not by eye across two figures.

Should I report a confidence interval or a p value?

Both, with the interval first. They are the same mathematics — a 95% interval holds exactly the null values you would not reject at .05 — so the p value adds nothing the interval lacks, while the interval adds magnitude and precision. The APA Publication Manual (2020) asks for the estimate with its interval alongside the effect size, and the exact p value next to the test statistic.

Take two sentences away. The 95% describes the procedure, not your interval; and the width tells you whether your study answered anything, while zero only tells you which side of a convention you landed on. If both ends of your interval point toward opposite decisions, the problem is not how you worded it, it is precision — and precision is settled before you collect a single case: working out the sample size for your thesis covers how to plan for an interval you can defend.