Confidence intervals (CIs) are one of the most useful statistical tools in research, yet also one of the most poorly understood. If you only report the p-value in your articles, you are leaving very valuable information on the table. Let us clarify once and for all what they are, what they mean, and how to interpret them correctly.
What is a confidence interval, really?
A 95% CI gives you a range of plausible values for the population parameter you are estimating. The formal definition: which always takes a moment to grasp: is: if you were to repeat the study 100 times with different samples, approximately 95 of those intervals would contain the true population value.
What a 95% CI does NOT mean
This is important, because I see these misunderstandings even in published articles:
- It does NOT mean there is a 95% probability that the parameter lies within the interval you calculated. The parameter either is or is not inside: the probability refers to the procedure, not to the specific result
- It does NOT mean that 95% of the data fall within the range
- It is NOT the same as the range of the data distribution. A CI speaks to the estimation of the parameter, not to the data themselves
Why are they so important?
Confidence intervals tell you things that the p-value cannot:
- Precision of your estimate: a narrow CI means your estimate is precise; a very wide one tells you there is considerable uncertainty
- Practical significance: you can have a statistically significant effect (p < .05) yet with a CI so wide that it spans from trivial to enormous effects. That is not very informative
- Direction and magnitude of the effect: a CI for a mean difference of [0.2, 1.8] tells you the effect is positive and may range from small to large
How to interpret a CI in practice
Here are some concrete examples I use with my clients:
- CI for a mean difference: [0.5, 2.3]. Since it does not include 0, the difference is statistically significant. Moreover, even the lower bound (0.5) suggests a meaningful effect
- CI for a correlation: [-0.05, 0.42]. It includes 0, so it is not significant. However, note that the upper bound suggests there could be a moderate correlation: perhaps you need a larger sample
- CI for an odds ratio: [1.2, 3.8]. It does not include 1 (which would indicate no effect), so the effect is significant. The risk is between 1.2 and 3.8 times greater (compute it directly from your 2×2 table with our odds ratio calculator)
CIs and sample size
This is something fundamental that many overlook: the larger the sample, the narrower the CI. If your interval is too wide to be informative, the solution is usually to increase the sample size. In fact, planning your sample to obtain a CI with the desired precision is a more sophisticated approach than the classic power analysis based on p-values.
CIs in APA 7
APA 7 guidelines require reporting confidence intervals for the main statistics. The format is:
- Mean difference: Mdiff = 3.45, 95% CI [1.23, 5.67]
- Correlation: r = .34, 95% CI [.15, .51]
- Effect size: d = 0.67, 95% CI [0.32, 1.02]
Parametric CIs versus bootstrap CIs: when each one is appropriate
The "classic" CIs you see in SPSS or in most basic textbooks are parametric: they assume that the sampling distribution of the estimator is normal (or t-distributed), and they construct the interval as estimate plus or minus a critical value times the standard error. For means with large samples and reasonably symmetric distributions, this approximation works well. The problem arises with non-normal estimators, small samples, or estimators whose sampling distribution is asymmetric or unknown. The classic example is the correlation: the sampling distribution of r is skewed (especially for values far from zero), so the symmetric CI around r is biased. Fisher's z transformation corrects this, but few researchers apply it manually.
The bootstrap solves this problem in a general way. Instead of relying on a theoretical sampling distribution, it estimates it empirically by resampling your data with replacement thousands of times and computing the statistic in each resample. The resulting CI (especially the percentile or BCa-bias corrected and accelerated-version) does not assume normality and works well for almost any estimator: medians, robust correlations, regression coefficients, indirect effects in mediation models, omega reliability, and so on. In R, the boot package or, more modernly, the infer package make this implementation easy. For mediation models, the lavaan package allows you to request bootstrap CIs directly with se = "bootstrap". My general recommendation is: if your estimator is a simple mean with N greater than 30, parametric is fine; in any other case, especially for indirect effects, ratios, or measures of association, use bootstrap with at least 5000 resamples.
CIs for the estimators that matter most in psychology
The interpretation of a CI depends on the scale of the estimator, and that is where many readers stumble. For an odds ratio (OR), the value of no effect is 1, not 0. A CI of [0.85, 1.45] includes 1 and is not significant. Furthermore, ORs are constructed on the logarithmic scale, so the CI is asymmetric on the original scale. For more guidance on interpreting OR and relative risk together, see my guide on odds ratio and relative risk. For a standardized regression coefficient (beta), the value of no effect is 0, and the typical CI is constructed assuming approximate normality of the estimator, which works well with moderate samples (if you need to convert raw scores to z or T scales beforehand, our z-score converter handles it). For Cronbach's alpha or McDonald's omega, the CI must be calculated via bootstrap (the parametric formulas exist but are restrictive). In R, the MBESS package gives you precise CIs for alpha; semTools gives them for omega. For Cohen's d, the parametric CI uses the non-central t distribution, and not all software calculates it correctly, our effect size calculator handles this from means, t, or F, so avoid online calculators that simply use plus or minus 1.96 times the SE of d, because that underestimates the asymmetry.
Confidence interval versus credibility interval
If you read Bayesian articles, you will see "95% credibility intervals" (CrI). They are not the same as CIs, although they often coincide numerically. A 95% CrI says exactly what many believe a CI says: "given the data and the prior, there is a 95% probability that the parameter lies within this interval." This is a direct probabilistic statement about the parameter, which is impossible in the frequentist framework. With uninformative or weakly informative priors and moderate sample sizes, the numerical values of CI and CrI are usually very similar, but the philosophical and reporting interpretations are different. If you want to go deeper into this difference, my article on Bayesian statistics for beginners develops it in detail.
Planning sample size for precision: AIPE
The classic approach to sample size is power analysis: how many participants do I need to detect an effect of a given size with 80% power. The alternative, less well known but increasingly recommended, is AIPE (Accuracy In Parameter Estimation). Instead of asking "how many participants to detect an effect," you ask "how many participants to obtain a CI with a maximum width of X." AIPE shifts the focus from the binary decision (significant or not) to the precision of the estimate. The MBESS package in R implements AIPE for means, mean differences, correlations, multiple regression, and standardized effect sizes. A common scenario in consulting is a doctoral student who has used a power analysis to justify a sample of N equals 80 to detect a "medium" effect, but whose resulting CI is so wide that it barely allows for any meaningful interpretation. Planning for precision, when the sample budget allows it, is more honest and more useful for the cumulative literature.
Frequent errors interpreting CIs that I see in manuscripts
Reviewing manuscripts, I detect the same errors over and over. The first is interpreting two overlapping CIs as "not significantly different." Two CIs can overlap and yet the difference between the means be statistically significant, especially if the overlap is partial. The correct test is the CI of the difference, not the comparison of separate CIs. The second is reporting CIs that contain impossible values without commenting on it (for example, a CI for a proportion that includes negative values, indicating that the normal approximation is inappropriate and that the Wilson or Clopper-Pearson method should have been used). The third is calculating CIs separately for each subgroup in a moderation analysis and concluding that there is moderation because one is significant and the other is not, instead of formally testing the interaction. The fourth is reporting CIs from a model that did not converge or whose assumptions were violated: the CI will then be incorrect regardless of how it is interpreted.
My recommendation
Make it a habit to always look at confidence intervals, not just the p-value. They will give you a much more complete and nuanced picture of your results. And if you need help calculating or interpreting them, I can lend you a hand to ensure your statistical report is rigorous and complete.