TOST Equivalence Testing: How to Show There Is No Difference

Your comparison came out at p = .438 and now you have to write the Discussion. The sentence that writes itself — “the two formats did not differ” — is the one a careful reviewer will strike out, because your test never asked that question. It asked how surprising your data would be if the true difference were exactly zero, and “not surprising” is what you get both when nothing is there and when your sample was too small to see what is there. Equivalence testing is the procedure that replaces the empty sentence with a claim you can defend, and the arithmetic is genuinely easy — what is hard is deciding, in advance, how small counts as nothing. The full menu of options when a result comes out null is in what to do when your result is not significant; this article is about the one option that ends in a positive claim.

A non-significant p is not evidence of absence

Null hypothesis testing was built to run in one direction. You set up a hypothesis you would like to knock down, you work out how surprising the data would be if it were true, and if they are surprising enough you reject it. There is no step in the procedure that lets you accept it. Failing to reject is what happens when the effect is genuinely nil, and it is also what happens when the effect is real and your sample was too small, your measure too noisy or your manipulation too weak. The three situations produce the same output. Altman and Bland made the point the title of a BMJ note back in 1995 and it has not aged: absence of evidence is not evidence of absence.

The confidence interval is where the difference becomes visible. Imagine two studies comparing the same two treatments on the GAD-7, both reporting a non-significant difference of −0.53 points. The first, with 154 participants, has a 90% interval of [−1.66, 0.60]: whatever is going on, a difference of two points has been ruled out. The second, with 52, has [−2.50, 1.44], which is compatible with the treatments being identical and equally compatible with one of them being two and a half points better. Same non-significant verdict, completely different evidence. A reader who only sees the p cannot tell which of the two studies you ran.

The reflex fix — computing observed power after the fact so you can report that the study was underpowered — does not help, because post hoc power is a rewriting of the p value and carries nothing the p did not already carry. Every non-significant result yields low observed power; that is arithmetic, not diagnosis. Why it fails and what to compute instead is in post hoc statistical power analysis.

What a reviewer actually wants is the answer to a different question, and that one is answerable: how large an effect can you rule out with the data you have? Equivalence testing asks exactly that, and answering it converts a limp non-result into a bounded claim. Not “we found nothing”, which nobody can argue with or check, but “we can exclude anything bigger than this”, which is a finding with edges.

TOST: two one-sided tests, one conclusion

The two one-sided tests procedure works by swapping which hypothesis has to be knocked down. Instead of a single null saying the difference is exactly zero, you write two nulls, each of them saying the difference is big. The first: the true difference is at or below the lower bound, −Δ. The second: the true difference is at or above the upper bound, +Δ. Each one is tested one-sidedly at α = .05. If you reject both, the only values left standing are the ones between the bounds, and that is the equivalence conclusion.

The arithmetic reuses everything the ordinary t test already gave you: same mean difference, same standard error, same degrees of freedom. All you do is shift the numerator by the bound. The lower test is the mean difference plus Δ divided by the standard error; the upper test is the mean difference minus Δ divided by the same standard error. The p you report is the LARGER of the two, because you need both rejections and the weaker one governs the conclusion. This is an intersection-union test, which is why — and people find this genuinely counterintuitive — you do not correct for running two tests. Two one-sided tests at .05 give you an equivalence test at .05, not at .10.

There is a shortcut that saves the arithmetic altogether. At α = .05, asking whether TOST is significant is exactly the same as asking whether the 90% confidence interval for the difference falls entirely inside your bounds. Ninety, not ninety-five: each one-sided test spends 5% in one tail, so the interval that corresponds to the pair leaves 5% in each tail and 90% in the middle. Checking a 95% interval instead is not wrong so much as conservative — you are quietly running the equivalence test at α = .025, and you will sometimes fail to conclude equivalence in a case your own preregistered alpha entitled you to.

None of this is new and none of it started in psychology. Schuirmann formalized the procedure in 1987 for bioequivalence, where the question is whether a generic drug delivers the same concentration as the branded one, and the regulatory version of the bound is the familiar 80–125% rule: the 90% interval for the ratio of geometric means has to sit inside those limits before the generic is approved. Psychology imported the machinery through Lakens (2017) and the tutorial by Lakens, Scheel and Isager (2018), which is where most current practice in the field comes from.

A worked example, with the numbers

A public mental health service wants to know whether the online version of its CBT protocol for generalized anxiety leaves patients no worse off than the face-to-face version. Post-treatment GAD-7 scores: online, n = 78, M = 8.42, SD = 4.10; face-to-face, n = 76, M = 8.95, SD = 4.36. The pooled standard deviation is 4.23 and the standard error of the difference is 0.682. The ordinary comparison gives t(152) = −0.78, p = .438, d = −0.13. On its own, that sentence means nothing at all.

The bound was fixed in the protocol before recruitment started: ±2 points on the GAD-7. The reasoning behind it was not statistical. The scale runs from 0 to 21, the team's decisions about stepping care up or down do not change for anything under two points, and the online format frees clinician time — so a difference smaller than two points would not alter what the service does with a patient. That justification, written in two lines, is what makes everything downstream defensible. Without it the bound is just a number that happened to work out.

Now the two one-sided tests. The lower one is (−0.53 + 2) / 0.682 = 2.16, which against 152 degrees of freedom gives p = .016. The upper one is (−0.53 − 2) / 0.682 = −3.71, p < .001. The larger of the two is .016, so both nulls are rejected and the difference is statistically equivalent to zero within ±2 points. The 90% confidence interval, [−1.66, 0.60], sits entirely inside the margin, which is the same conclusion reached by looking instead of computing.

Run the identical means and standard deviations through a smaller study — 26 per group instead of 78 and 76 — and the picture changes completely. The ordinary test gives t(50) = −0.45, p = .653, which looks like the same comfortable non-result. The equivalence test gives p = .108 and a 90% interval of [−2.50, 1.44], which spills past the lower bound. Nothing has been demonstrated in either direction, and the honest word for that is inconclusive. Two null results that read the same on the page: one supports a claim, the other supports nothing.

Choosing the SESOI is the hard part, and it is not a statistical decision

Everything above is arithmetic that a package will do for you. The bound is not, and it is where equivalence testing actually lives. A SESOI — smallest effect size of interest — is a claim about what would matter, and no amount of data will supply it, because the data cannot know what you are going to do with the answer. What a reviewer checks is not whether your number is the right one, which is unknowable, but whether you can say where it came from and prove you chose it before you looked.

Four routes survive review, and you should name the one you took. Clinical or practical relevance. The smallest change that alters what somebody does — a dose, a referral, a decision — which in clinical work usually means the minimal important difference established for that instrument. Cost-benefit. The effect the new option has to beat to be worth its extra cost, burden or training time; if the online format is cheaper, the bar it must clear is “not meaningfully worse”, and you can say how much worse you would tolerate for the money saved. Small telescopes. Simonsohn's (2015) proposal for replications: take the effect the original study had 33% power to detect, which works out at d = 0.50 for 20 participants per group and d = 0.31 for 50 per group. Feasibility. The smallest effect the sample you can actually recruit is able to bound, stated openly as a limitation rather than dressed up as a preference.

Cohen's 1988 benchmarks are conventions for describing effects, not thresholds for deciding anything, and reaching for d = 0.20 as an automatic bound is the laziest version of this decision — it can be far too narrow for an expensive trial and far too wide for a safety comparison. Where the scale means something to the people who use it, set the bound in raw units: two GAD-7 points is arguable in a clinical meeting, d = 0.47 is not. Standardized bounds are for scales nobody has an intuition for, and they inherit whatever standard deviation your particular sample happened to produce. How to read a d before you turn it into a bound is in interpreting Cohen's d, and the effect size calculator moves between raw and standardized units for you.

Bounds do not have to be symmetric. Often only one direction is a problem: the online format must not be more than two points worse, while being better is welcome news nobody needs protecting from. That is a non-inferiority test — one one-sided test, one bound — and its conclusion is “not meaningfully worse” rather than “equivalent”. Say which of the two you ran, because they answer different questions and a reviewer will notice if you claim the stronger one. And fix the number before you see the interval: picking a bound after looking at the data is the same researcher degree of freedom as picking an exclusion rule after looking at the p, and it is even easier to spot, because the bound always lands suspiciously just outside the interval.

Four things can happen when you run both tests

The difference test and the equivalence test answer different questions, so they can both be run on the same comparison, and there are four outcomes rather than two. Neither significant. Inconclusive: you cannot exclude zero and you cannot exclude effects bigger than your bound, which usually means the study was too small for the question. Difference test significant, equivalence test not. There is an effect, and it may well be large enough to matter. Difference test not significant, equivalence test significant. Evidence of absence in the only sense the phrase can honestly carry: whatever the effect is, it is too small to be interesting. Both significant. A real effect that is nevertheless trivially small.

That fourth cell is the one everyone forgets, and in large samples it is the common one. A registry study with N = 3,000 finds that hours of screen use correlate with depressive symptoms at r = .05, t(2998) = 2.74, p = .006. Significant, and in a press release it becomes a link between screens and depression. With a SESOI of ±.10 the equivalence test is also significant, p = .003, and the 90% interval, [.02, .08], sits entirely inside the bound. The correlation is real and it is smaller than anything the authors themselves said in advance would matter. Reporting only the p turns that into a headline the data cannot support.

This is the argument for building both tests into the analysis plan rather than reaching for equivalence only when a result disappoints. Running the pair on every primary comparison costs one extra line of code and removes the two worst sentences in a Results section: “there was no difference” when you never tested for one, and “there was a significant association” when the association is far too small to act on.

How to report TOST in APA 7

The Method carries the setup and needs four things: the bound with its units, the justification, the alpha, and the fact that all of it was fixed in advance. One sentence covers it. “Equivalence was assessed with two one-sided tests against a margin of ±2 points on the GAD-7, a bound fixed in the study protocol as the smallest change that would alter clinical decisions; both one-sided tests used α = .05.” If the bound is standardized rather than raw, say which standard deviation it is standardized on, because a reader cannot reconstruct that from the results table.

The Results paragraph reports both tests, in that order, and it looks like this. “The difference between formats was not statistically significant, t(152) = −0.78, p = .438, d = −0.13. The equivalence test against the prespecified margin of ±2 GAD-7 points was significant, t(152) = 2.16, p = .016, one-sided, and the 90% confidence interval for the difference, [−1.66, 0.60], fell entirely within the margin; the two formats are statistically equivalent within that bound.” The APA 7 conventions are the usual ones: t, p, M, SD, d, n and N in italics, CI and its bracketed limits in roman, degrees of freedom in parentheses attached to the t. Write “one-sided” next to that p, because every reader is entitled to assume two-tailed unless told otherwise. The rest of the t test conventions are in how to report a t test in APA 7.

Two phrasings get flagged, and both are avoidable. “There was no difference” claims something no test can deliver; the correct claim is bounded, and it should keep its bounds inside the sentence — equivalent within ±2 points, never equivalent full stop. “We accept the null hypothesis” is the other, and it is not what happened: you rejected two nulls of your own construction, which is a rejection like any other. In the Abstract, write the conclusion as a finding rather than an apology. An equivalence result is a positive result, and it should not read like a study that failed.

For software, R does this with the TOSTER package, which prints both one-sided tests, the larger p and the interval; jamovi runs the same package as a module, and JASP has an equivalence t test among its t test options. SPSS has no equivalence procedure, and for the simple designs it does not need one: the independent-samples output already prints the mean difference and its standard error, so the two one-sided t values are two divisions on a calculator, and setting the confidence level to 90% in the Options dialogue gives you the interval to check by eye.

Frequently asked questions

What is a TOST equivalence test?

TOST stands for two one-sided tests. Instead of testing whether an effect is exactly zero, you set a margin (the smallest effect size of interest) and test two hypotheses: that the true effect is at or below the lower bound, and that it is at or above the upper bound. Each is tested one-sidedly, normally at .05, and you report the larger of the two p values. If that larger p clears your alpha, both nulls are rejected and the effect is statistically equivalent to zero within your margin. It is the standard procedure for arguing that a difference is absent, and it comes from bioequivalence testing, where Schuirmann formalized it in 1987.

How do you choose the equivalence bound or SESOI?

It is a substantive decision, not a statistical one, and there are four defensible routes: the smallest change that alters a clinical or practical decision (the minimal important difference for the instrument), the effect a new option must beat to be worth its cost, the small telescopes rule of taking the effect the original study had 33% power to detect (d = 0.50 for 20 per group, d = 0.31 for 50 per group), or the smallest effect your feasible sample can bound, stated as a limitation. Set it in raw units when the scale is interpretable, justify it in one sentence, and fix it before you look at the data.

Why does TOST use a 90% confidence interval and not 95%?

Because the procedure is two one-sided tests, each spending 5% in a single tail. The interval that corresponds to that pair leaves 5% in each tail and 90% in the middle, so an equivalence test at alpha .05 is exactly the question of whether the 90% interval falls inside the bounds. Checking the 95% interval instead is conservative rather than wrong: it amounts to running the equivalence test at alpha .025, and it will sometimes deny you a conclusion your preregistered alpha allowed.

Can you run an equivalence test in SPSS?

SPSS has no built-in TOST procedure, but for a two-group comparison you do not need one. The independent-samples t test output prints the mean difference and its standard error, so each one-sided t is a single division: add the bound to the mean difference and divide by the standard error for the lower test, subtract it for the upper test, and look both up with the usual degrees of freedom. Setting the confidence level to 90% in the Options dialogue gives you the interval version of the same test. If you want it done for you, R has the TOSTER package, jamovi runs it as a module and JASP includes an equivalence t test.

What is the difference between a non-significant result and an equivalent one?

A non-significant result says your data are compatible with zero, and also with every other value inside the confidence interval, including effects large enough to matter. An equivalent result says the whole interval sits inside a margin you defined in advance as too small to care about. The same mean difference of 0.53 GAD-7 points is both things with 154 participants, where the 90% interval is [−1.66, 0.60], and only the first with 52, where it is [−2.50, 1.44] and reaches past a two-point bound.

How many participants do you need for an equivalence test?

For a two-group comparison at 80% power with alpha .05 and a true effect of zero, you need roughly 70 per group for a bound of d = 0.50, 108 per group for 0.40, 191 for 0.30 and 429 for 0.20. Narrowing the margin gets expensive fast: halving the bound roughly quadruples the sample. For comparison, the ordinary test needs 64 per group to detect d = 0.50, so equivalence at the same size costs only a little more, but that holds only when the true effect is exactly zero. If it is really d = 0.10 and your bound is 0.50, you need 82 per group.

With the bound fixed in advance and justified in one line, both tests reported, an exact one-sided p and the 90% interval beside it, a null result stops being an apology and becomes a finding. If the difficult part is the bound itself — an instrument with no established minimal important difference, a non-inferiority design, an equivalence test sitting inside a multilevel model or a meta-analysis — that is exactly the kind of decision we can work through together in my statistical consulting.