My Results Are Not Significant: What Now?

You ran the analysis you had planned, the number came back p = .217, and the whole discussion section you had in your head stopped making sense. This happens to most doctoral projects at least once, and the usual reaction — quietly refitting the model with a covariate that helps, or writing that the effect approached significance — costs far more than the null result ever would. There are three honest ways out and one dead end that everyone tries first. The formatting rules that apply to all of them are in how to report results in APA 7.

Non-significant does not mean the effect is zero

The p value answers one narrow question: if the effect were exactly zero, how surprising would data at least as extreme as mine be? A p = .217 says they would not be surprising. It does not say the effect is zero, that it is small, or that your hypothesis is wrong. Null hypothesis testing is asymmetric by construction: it can reject zero, never confirm it.

The information lives in the interval, not in the p. Picture a trial of a mindfulness app against a waitlist on rumination, 60 participants per group. The comparison gives t(118) = 1.24, p = .217, d = 0.23, 95% CI [-0.13, 0.59]. That interval includes d = 0.59, an effect that would change clinical decisions. Writing that the app does not work is a claim the upper half of your own interval contradicts.

From there comes the idea that organizes everything else: results come in three kinds, not two. There is evidence that an effect exists, there is evidence that it does not, and there is the undetermined case, where the study cannot tell the two apart. Most null results in psychology are the third kind, and what decides which one you have is not the p value but what you do after you see it.

This is not a wording technicality. The sentence you are entitled to write in the discussion changes completely across the three: 'we did not detect the effect' is always defensible, 'there is no effect' only becomes defensible after the extra work described below, and 'the groups are equivalent' requires a test almost nobody runs.

How to write a null result in APA 7

The three rules are the same as for any other result; they are just broken more often here. The p value goes in exact, to two or three decimals and with no leading zero, never as 'ns' and never as p > .05: p = .217 and p = .612 do not tell the same story. The test statistic goes in complete with its degrees of freedom. And the effect size with its confidence interval is more compulsory here than anywhere else, because it is the only thing that tells the reader which effects your data still allow.

Here is how the four commonest tests look. t test: 'The groups did not differ in rumination, t(118) = 1.24, p = .217, d = 0.23, 95% CI [-0.13, 0.59].' ANOVA: 'The effect of condition was not significant, F(2, 117) = 0.84, p = .434, partial eta² = .014.' Correlation: 'The association did not reach significance, r = .08, N = 140, p = .347, 95% CI [-.09, .24].' Chi-square: 'The distribution did not differ between groups, chi²(1, N = 180) = 1.42, p = .233.'

Four phrasings get struck out by reviewers without discussion. 'The null hypothesis was accepted.' The test cannot accept it; it can only fail to reject it, which is a different statement. 'There was no difference' or 'no relationship exists.' That claims more than you know: what you know is that you did not detect one. 'Approached significance', 'marginally significant', 'a trend toward'. With alpha at .05, .058 is exactly as non-significant as .612; the difference between them lives in the interval, and that is where it should be reported. And 'the effect was small but in the expected direction' when the interval crosses zero: the direction of an effect whose interval includes zero is not established.

The discussion paragraph that does work says three things in a row: what you did not detect, what remains compatible with your data, and what it would take to settle it. 'The 95% confidence interval ranged from d = -0.13 to 0.59, so effects of the size reported in earlier work remain compatible with these data; the study cannot rule them out.' That is honest, publishable, and closes no door that is not already closed.

Observed power is the dead end

The sequence is always the same: the p comes back high, G*Power opens, the observed d goes in, 23% comes out, and the manuscript says the study was underpowered. It does not work. Observed power, computed from the effect size you obtained, is a mathematical transformation of the p value itself: two studies running the same test with the same p have exactly the same observed power, and any p just above .05 returns power near 50%. Hoenig and Heisey (2001) called it the abuse of power for that reason. It is not a new piece of evidence; it is your p value in a different costume.

The circularity is easier to see backwards. If observed power is low because the effect did not come out, and the effect did not come out because power was low, you have learned nothing: you have described the same number twice. Fewer and fewer journals let that paragraph through, and where it still gets past review it does nothing about the question that matters, which is what effect your design could have seen.

What you do instead is a sensitivity analysis: fix the N you have and the power you want, then solve for the effect size. With 60 per group, two-tailed alpha of .05 and 80% power, the smallest detectable effect is d = 0.52. That statement has content: your design could not see anything below half a standard deviation, and if the literature reports effects around 0.30, it was never going to find them. The difference between the two calculations, and how each is written up, is in post hoc statistical power analysis.

The sensitivity analysis belongs in the results section, next to the test, in one sentence: 'A sensitivity analysis indicated that with n = 60 per group and power of .80 the study could detect effects of d = 0.52 or larger.' That tells the reviewer exactly what your sample bought, and it requires no apology.

Equivalence testing: the frequentist route to claiming no effect

Before you can say there is no effect, you have to say what counts as nothing. No formula gives you this: the smallest effect size of interest comes from clinical relevance (the smallest change a patient would notice), from previous work, from the effect the study was designed to detect, or from a reasoned agreement in the field. Cohen's (1988) benchmarks are conventions for interpretation, not a substantive threshold you can borrow for this job; interpreting Cohen's d explains why. For the app example we set the bound at d = 0.40, the change below which the clinical team would do nothing differently.

TOST (two one-sided tests) reverses the burden of proof: instead of testing against zero, you test against both bounds. The null becomes 'the effect is at least as large as the bound', and you need both tests significant to reject it. Our data give t(118) = 3.43, p < .001 against the lower bound and t(118) = -0.95, p = .172 against the upper one. You report the larger of the two, p = .172, and it does not get there: the study does not establish equivalence. The 90% confidence interval says the same thing, [-0.08, 0.53], which pokes out past 0.40 at the top.

So that study sits in the third case: it neither detected the effect nor supported its absence. Writing it that way, with both tests on the page, makes a better paper than any evasive phrasing, because it draws the boundary of what is known. When equivalence does come through, it looks different: with 200 per group and d = 0.05, both tests give p < .001 and the 90% interval sits inside the bounds, [-0.12, 0.22]. There you can write that the effect is statistically equivalent to zero within a margin of ±0.40 (Lakens, 2017).

Know the price before you promise anything. Establishing equivalence with a bound of 0.40 at 80% power takes roughly 108 participants per group, 216 in total, nearly double the 60-per-group study. Claiming an absence of effect is more expensive than detecting one, which is why so few papers do it properly. If you have the means and standard deviations and want to see where your effect falls relative to the bound you set, the effect size calculator returns d with its interval in one step.

Bayes factors: how far your data favor the null

A Bayes factor compares two models instead of testing one. The BF₁₀ says how many times more likely your data are under the effect hypothesis than under zero, and its reciprocal, the BF₀₁, answers the question a p value cannot: how many times more likely they are under no effect. A BF₀₁ of 8 means the data are eight times more expected if there is no effect than if there is one, under the effect model you specified. That is evidence for the null, not merely an absence of evidence.

For the 60-per-group example with the JZS Cauchy prior (Rouder et al., 2009) at the scale of 0.707 that JASP and BayesFactor use by default, the result is BF₁₀ = 0.39, that is BF₀₁ = 2.56. The conventional labels — 1 to 3 anecdotal, 3 to 10 moderate, above 10 strong — which are a working adaptation of Jeffreys's (1961) scale, leave that in the anecdotal band. Undetermined again, agreeing with what the interval and the TOST already said. The larger study gives BF₀₁ = 8.0, moderate evidence for the null.

The prior does move the answer, so you report a robustness check rather than hiding it. In the small study, sliding the scale from 0.5 to 1.41 moves BF₀₁ from 2.0 to 4.7: the conclusion crosses the 3 line, meaning it depends on the prior you picked, and that has to be stated. In the large study the range runs from 5.8 to 15.7, all of it moderate or strong, so the conclusion holds.

The sentence to write is this one: 'A Bayesian independent-samples t test (Cauchy prior, scale 0.707) gave BF₀₁ = 8.00, moderate evidence for the null model; a robustness check across scales of 0.5 to 1.41 returned values between 5.8 and 15.7.' JASP does this free, robustness plot included. One warning: a Bayes factor is not a p value under another name and has no threshold that gets you published. It is a continuous scale, and below 3 in either direction what you have is still a study that does not decide.

Before you write it up: what could have hidden a real effect

Measurement is the first suspect. An observed correlation is attenuated by the reliability of both instruments: if the true correlation is .30 and your scales have alphas of .70 and .65, what you should expect to see is .20. With N = 80 that gives r = .20, p = .075, 95% CI [-.02, .40], a textbook null sitting on top of a perfectly correct hypothesis. Before you conclude the absence of anything, look at the reliabilities: an effect can be spent entirely on measurement error.

Then come the usual suspects, and it is worth ruling them out in writing. Ceiling or floor effects on the scale (look at the distributions, not only at the means). Range restriction, standard when the whole sample is second-year psychology students. A failed manipulation check, which turns your comparison into a contrast between two conditions that never differed. Wrong dose or wrong timing: eight sessions where the effect needs sixteen. And the wrong test, which in small skewed samples can cost real power.

And then the design, where nearly everything is decided. Detecting d = 0.30 at 80% power takes 176 per group, 352 in total. A 60-per-group study has 37% power against that effect and 78% against d = 0.50: it was built to see medium effects, not the ones the literature typically reports. None of this can be fixed afterwards, but it can be explained, and the next study can be planned with the sample size calculation done before data collection.

With all of that in hand, publish it. A null that is well measured, adequately powered and bounded by an interval or a TOST is a contribution: it saves other teams a hypothesis that has already been tried. A vague null from a short sample with weak scales is not, and what separates the two is this final section rather than luck. There are journals that accept null results, and there is the registered report format, where the paper is accepted before the result is known; if your project rests on a single test, that is where it belongs.

Frequently asked questions

What do I do if my results are not significant?

First, do not touch the analysis. Report the exact p value, the full test statistic and the effect size with its confidence interval, then add a sensitivity analysis stating what effect your sample could detect. If you also want to claim there is no effect, you need a TOST equivalence test or a Bayes factor; without one of those, the correct wording is that you did not detect it.

Does a non-significant p value mean there is no effect?

No. It means you could not rule zero out, which is a different statement. The confidence interval usually shows it: in t(118) = 1.24, p = .217, d = 0.23, 95% CI [-0.13, 0.59], effects as large as 0.59 remain compatible with the data. Absence of evidence is not evidence of absence.

Should I report post hoc power for a non-significant result?

No. Power computed from the observed effect is a one-to-one function of the p value, so it carries no new information and the argument is circular (Hoenig & Heisey, 2001). What you do report is a sensitivity analysis: given your N and power of .80, the smallest effect you could have detected. With 60 per group that is d = 0.52.

How do I prove there is no difference between two groups?

With an equivalence test. Set a smallest effect size of interest before looking at the data, say d = 0.40, then run two one-sided tests against both bounds (TOST): if both are significant, and the 90% confidence interval sits inside the bounds, you can claim equivalence within that margin. It costs sample size: around 108 per group for a bound of 0.40 at 80% power.

Can I publish a non-significant result?

Yes, and increasingly so. What decides it is not the sign of the result but whether it is interpretable: a null with adequate power, reliable measures and a tight interval, or one backed by a TOST or a Bayes factor, tells the field something useful. Several journals accept null results outright, and registered reports are accepted before the result is known.

What is a Bayes factor and when should I use one?

It is the ratio of how likely your data are under two models. The BF₀₁ quantifies how many times more expected they are under no effect: BF₀₁ = 8.0 counts as moderate evidence for the null under the labels in common use, adapted from Jeffreys (1961), while anything between 1 and 3 means the study does not decide. Always report it with the prior you used and a robustness check.

A null result is not a failed study; an uninterpretable one is. Report the exact p, the effect size with its interval and a sensitivity analysis instead of observed power, then add a TOST or a Bayes factor if you intend to say anything about absence. If you need to decide which of the two fits your design, and what to write when the answer is undetermined, that is what my statistical consulting is for.