Statistically Significant but Useless (and Non-significant but Large)

Two reviewer comments stop a manuscript equally dead. One says your effect is significant but never explains why anyone should care. The other says your group difference looks substantial but p = .075, so there is no finding here. Both point at the same crack: p values and effect sizes answer different questions, and treating them as one number leaves you either defending a result that cannot carry the claim or burying one that deserved a follow-up. This is about pulling the two apart, and about what to write in each case.

A p value does not measure size

Look at the arithmetic of the most common test, a two-sample t with n per group: the statistic is the standardized effect multiplied by the root of half the group size, t = d · √(n/2). Two factors multiplying each other. One is the phenomenon; the other is your recruitment budget. The p value comes out of that product, so it answers a mixed question: how surprising is a result at least as extreme as this one if the true effect were exactly zero, at the precision this sample gives me? It never answers whether the effect is large.

One uncomfortable consequence follows immediately. Any effect that is not exactly zero eventually becomes significant if you measure enough people, and in psychology almost nothing is exactly zero. Meehl (1990) called it the crud factor: in large samples any two questionnaire variables correlate a little, if only because they share a method, a moment and a respondent. At N = 10,000, correlations of r = .03 come out significant and mean nothing.

So significant is not a synonym for large, important or replicable. It means something narrow and modest: with this sample, the data sit badly with an effect of exactly zero. The American Statistical Association put it in writing (Wasserstein & Lazar, 2016): a p value measures neither the size of an effect nor the importance of a result. It is a statement about compatibility with a hypothesis, not about magnitude.

What follows is the same misunderstanding seen from both sides, with numbers attached. First, how p moves on its own when the only thing changing is N. Then the two cases that fill statistics forums every week: the significant result nobody can use, and the large result the design was never built to see.

One effect, three sample sizes

Take a correlation of r = .10, the size that turns up constantly in psychology whenever two self-report variables are related to each other. At N = 60: t(58) = 0.77, p = .447, 95% CI [-.16, .35]. At N = 400: t(398) = 2.01, p = .046, 95% CI [.002, .196], which barely clears zero. At N = 1200: t(1198) = 3.48, p < .001, 95% CI [.04, .16]. Identical correlation, moving from nothing to highly significant while the phenomenon stays exactly where it was.

You can compute the crossing point. With r = .10, the first sample size that returns p < .05 is N = 385. Below 385 people that effect is non-significant; from 385 upwards it is significant. That boundary is a property of your funding, not of the variables you are studying. Nobody would write in a discussion that the association appears once you pass 385 participants, yet that is literally what the sequence of p values says.

The intervals do tell three different stories. All are consistent with r = .10, but the first admits anything from a noticeable negative association to a substantial positive one, while the third pins the effect between .04 and .16. That is the information a p value throws away when it collapses everything into a yes or a no: how precisely you know the answer. If you suspect you have been reading intervals as significance tests in disguise, that habit is fixable and it is unpicked in how to interpret a confidence interval.

The working rule: before celebrating or mourning a p value, check the sample size behind it. A p = .001 from 3,000 people and a p = .001 from 40 describe completely different situations, and only one of them should send you looking for how big the thing you found actually is.

Case 1: significant and irrelevant at the same time

A workplace wellbeing program evaluated against a waitlist, 600 employees per arm, outcome measured with the PHQ-9. The group difference is 0.8 points with an SD of 5.5, so d = 0.15, 95% CI [0.03, 0.26]. The test returns t(1198) = 2.52, p = .012. In an abstract that becomes the program significantly reduced depressive symptoms, and it clears review at plenty of journals with nobody asking a follow-up question.

Here is the follow-up question. How much is 0.8 points of PHQ-9? With a reliability of alpha = .89 and an SD of 5.5, the standard error of measurement is SEM = 5.5 · √(1 − .89) = 1.82 points, so the difference between arms is under half the noise attached to a single person's score. The reliable change criterion (Jacobson & Truax, 1991) works out at 1.96 · √2 · 1.82 ≈ 5.1 points, which lands almost exactly on the five-point change conventionally treated as clinically meaningful on the PHQ-9. Your program moved the average employee about a sixth of that.

Said in language you could read aloud: with d = 0.15, pick one person at random from each arm and the treated one is better off 54% of the time, against the 50% you get from a coin; the two distributions overlap by 94%. And the interval, which is the part that settles it, runs from 0.03 to 0.26. Even the most optimistic value your data allow is an effect nobody would call meaningful. That is not an ambiguous result. It is clear evidence that the effect is real and trivial. To convert your own means and SDs into d with its interval, use the effect size calculator.

One turn of the screw stops this becoming the opposite error. Small does not automatically mean useless (Funder & Ozer, 2019): d = 0.15 from a free, five-minute, zero-risk intervention delivered to a million people can matter more in public health than d = 0.50 that requires twelve sessions with a clinician. The right question is never is it small, it is does it repay what it costs. That question is answered by the effect size placed next to the cost, never by the p value.

Case 2: non-significant and large at the same time

Now the mirror image. A pilot of a brief school-based program, 20 pupils per group, d = 0.58 favoring the intervention. The test gives t(38) = 1.83, p = .075. The two usual reactions are both bad: file the study in a drawer, or write that the effect approached significance, a phrase a Q1 reviewer deletes without arguing about it.

The interval says what actually happened: 95% CI [-0.05, 1.21]. Your data are compatible with nothing at all and with something substantial. With 20 per group, the smallest effect that design could detect at 80% power is d = 0.91, well above what you found, and its power to detect a real d = 0.58 was 43%, worse than tossing a coin. The study did not show the effect is absent. It showed it was built not to see the effect even if it was there.

What you write instead is that no significant difference was detected, followed by the full statistic, the effect size with its interval and a sensitivity analysis stating what the design could have found. If you want the small-sample correction, Hedges' g = 0.57 here. And if your claim is that the effect is negligible, a high p value will not buy it: you need a TOST equivalence test against a threshold you have defined. The full protocol for writing up a null result is in my results are not significant: what now.

The symmetrical warning, which almost nobody gives: that d = 0.58 does not prove the effect is large either. It is a point estimate from 40 people, and small-sample estimates are noisy by construction; when only the significant ones get published, the ones that reach the literature are inflated on top of that. What you have is not an effect of 0.58, it is a range from zero to 1.21, and the honest conclusion of a pilot is that this warrants an adequately powered trial, not that it works.

What makes an effect relevant, and where the threshold comes from

Relevance is not in the data. No calculation extracts it from your matrix, because it depends on what the result is for, and that lives outside the analysis. What you can do, and what separates an interpretable paper from one that only reports p values, is fix the threshold before you look and state it in the method: the smallest effect size that would be of interest to you.

Clinical anchor. How much the measure has to move for a patient to notice or for a decision to change: the minimal important difference for the instrument, a reliable change criterion, a response or remission threshold. The most defensible option when one exists. Field benchmark. What the interventions already available achieve: if standard care delivers d = 0.50, a new program delivering 0.15 does not change practice, significant or not. Cost and resources. What effect would justify the price, the training time or the risk. All three are arguments rather than calculations, which is why each one takes a sentence and a citation.

Cohen's benchmarks (0.20, 0.50, 0.80 for d) are useful as a last resort and dangerous as the only criterion. Cohen (1988) offered them for want of anything better and warned they were field-dependent. A d = 0.20 in a population-level intervention can be a success, and a d = 0.80 against a waitlist can be largely expectancy. How to use them without turning them into law is covered in interpreting Cohen's d.

With a threshold in place, you stop having two conclusions and start having four, which is what your data actually support. Significant with the whole interval above the threshold: a relevant effect. Significant with the whole interval below it: real and trivial, and you say so in those words (the PHQ-9 case above). Non-significant with the whole interval inside the irrelevance zone: practical equivalence, which is demonstrated with TOST rather than asserted. Non-significant with the interval straddling the threshold: undetermined, which is the pilot case and the most common honest conclusion in psychology.

How to write each case without getting it cut

Significant and small. “The program reduced PHQ-9 scores, t(1198) = 2.52, p = .012, d = 0.15, 95% CI [0.03, 0.26]. The magnitude is small: the 0.8-point difference falls below the instrument's standard error of measurement (1.82) and well below the five-point change usually regarded as clinically meaningful, so the effect, while consistent, is unlikely to be perceptible at the individual level.” A reviewer cannot attack that paragraph, because you have already said what they were going to say.

Non-significant and large. “The difference did not reach significance, t(38) = 1.83, p = .075, d = 0.58, 95% CI [-0.05, 1.21]. With 20 participants per group the study was powered to detect effects of d ≥ 0.91, so the interval remains compatible with effects of practical importance. These data neither rule the effect out nor establish it.” That last sentence is what turns an abandoned pilot into a justification for the next trial.

Four things that get cut in review without discussion. Highly significant, very significant. Significance has no degrees: you cross alpha or you do not. p = .000. No such value: write p < .001. Approached significance, marginally significant, trending toward. With alpha at .05, .075 is as non-significant as .612; what separates them is the interval, and that is where the story belongs. Calling an effect moderate on Cohen's scale alone, without translating it into the instrument's units or comparing it with anything.

The test for any sentence in a discussion is simple. Delete the p value and see whether the claim still stands with the interval in front of you. If the program is effective needs the .012 to survive and collapses the moment you look at [0.03, 0.26], the sentence was badly written. If we found no evidence of an effect holds up with [-0.05, 1.21] on the page, it was written well. That is the whole filter, and it is the same one a competent reviewer applies.

Frequently asked questions

Does a significant p value mean the effect is large?

No. It means your data sit badly with a true effect of zero, and nothing more. The p value depends on the effect size and the sample size together (t = d · √(n/2) for two groups), so a tiny effect turns significant once you measure enough people: with 600 per arm, d = 0.15 returns p = .012. Whether it is large is answered by the effect size and its confidence interval.

Why is my result significant in a large sample but not in a small one?

Because the only thing that changed is precision, not the phenomenon. The same correlation, r = .10, gives p = .447 at N = 60 and p < .001 at N = 1200. With r = .10 the .05 boundary sits at N = 385: below it the effect is non-significant, above it significant. That is a property of your sample, not of the effect you are studying.

Can a non-significant result have a large effect size?

Yes, and it is routine in pilots. With 20 per group, d = 0.58 gives t(38) = 1.83, p = .075, 95% CI [-0.05, 1.21]. That design could only detect effects of d ≥ 0.91 at 80% power. The correct reading is that the study could not see what it was looking for, not that the effect does not exist.

What is the difference between statistical significance and clinical significance?

Statistical significance says the result is unlikely if the true effect were zero. Clinical significance says the change is big enough to matter to a patient or to a decision, and it is judged against an external anchor: a minimal important difference, a reliable change criterion, a response threshold. A result can be statistically significant and clinically negligible, which is exactly what a 0.8-point PHQ-9 difference is.

How do I decide whether my effect is practically meaningful?

Compare it against a threshold you set before analyzing and justify in the method. Three defensible sources: a clinical anchor (minimal important difference, reliable change, response criterion), the effects achieved by interventions already available in your field, or the cost and risk your intervention demands. Cohen's benchmarks are a last resort, not a measure of importance.

How do I write up an effect that is significant but small?

Say it yourself before the reviewer does, with the full statistic, the effect size, its interval and a translation into the instrument's units: “t(1198) = 2.52, p = .012, d = 0.15, 95% CI [0.03, 0.26]; the 0.8-point difference falls below the questionnaire's standard error of measurement”. A real but trivial effect is publishable; dressing it up as an important finding is what gets desk-rejected.

Keep the decomposition in mind: a p value blends the effect with the sample size, which is why it can answer neither is it large nor does it matter on its own. Report the effect size with its interval every time, fix a relevance threshold in the method, and write conclusions that hold up with the interval in front of them. If they survive with the p deleted, you have a discussion; if they collapse the moment it goes, you have a sentence a reviewer is going to cut.