How to Report Post Hoc Comparisons in APA 7

You ran the ANOVA, it came out significant, and now you have four groups, six pairwise comparisons and a drop-down menu offering Bonferroni, Tukey, Šidák, Scheffé, Holm and half a dozen more, with not one line explaining how they differ. The temptation is obvious: try two or three and keep the one that lets your comparison of interest through. That move leaves no trace in the manuscript, and it is one of the most productive sources of false positives in the whole Results section. Here is what each correction controls, which one to choose for what you are actually doing, how the adjusted p is written, and why reporting only the comparisons that worked is not a saving of space but a bias. If the ANOVA line that goes immediately before this is still fuzzy, start at how to report an ANOVA in APA 7.

What goes into a post hoc comparison

Four numbers and a name. The mean difference with its sign, its confidence interval, the adjusted p value and an effect size; the name is the correction, and it travels with the number of comparisons you applied it to. In a four-arm trial with 30 people per group, the full sentence reads: "Tukey's test showed that the CBT group scored lower than the waiting list, M_diff = 4.70, 95% CI [1.33, 8.07], q(4, 116) = 5.15, p = .002, d = 0.94." Without the name of the method that p cannot be interpreted: the CBT versus psychoeducation comparison in the same study is .012 uncorrected, .057 under Tukey and .071 under Bonferroni, and all three are the same contrast on the same data.

The omnibus line goes in front, with both degrees of freedom and its effect size: F(3, 116) = 4.84, p = .003, η²_p = .11. That line says something differs across the four groups and does not say what; the post hoc comparisons say what, and say nothing new about the set as a whole. With more than two groups, a significant ANOVA with no comparisons underneath leaves the reader with half a result, and comparisons with no omnibus in front of them hide the context in which the error degrees of freedom were computed.

Italics and leading zeros split letter by letter, the same as everywhere else. Italicized: M_diff, q, t, F, d, p, n and N. Upright: α and η²_p, because they are Greek letters. The mean difference, the interval bounds, d and q keep the leading zero because they can exceed 1; p, α and η²_p drop it because they cannot. One detail about q: the two numbers in the brackets are the number of groups and the error degrees of freedom, not k − 1 and the error term as in an F. Four groups and 116 error degrees of freedom give q(4, 116), and that first 4 is the one most often copied across wrongly.

Once you go beyond three or four comparisons this is a table, not a paragraph: one row per comparison, columns for the difference, the interval, the adjusted p and the effect size, and a footnote naming the method and the number of comparisons. The effect size goes per comparison, not one global value for the whole ANOVA, and it is written with the usual Cohen's d convention: leading zero, two decimals, and read as a convention rather than a law.

The family: what you are correcting and how far it inflates

What a post hoc correction protects is the family-wise error rate: the probability of at least one false positive across the set of contrasts you ran, not within each one separately. That set grows fast, because k groups give k(k − 1)/2 pairwise comparisons. Three groups are 3 comparisons, four are 6, five are 10 and six are 15. That number, called m here, is what feeds almost every adjustment formula.

The figure usually quoted to dramatize the problem is the one for independent tests: if each is run at α = .05, the probability of at least one false positive is 1 − .95^m, which is .14 with three comparisons, .26 with six and .40 with ten. It is worth saying out loud that pairwise comparisons are not independent: they share groups and they share the error term, so those values are an upper reference point rather than the exact number for your design. The direction, though, is right, and with ten uncorrected comparisons that 40% stops being a methodological footnote: it is one false positive in two of every five studies like yours.

The uncomfortable part is that the family is not a fact about the data, it is a decision you make. If you measure two outcomes across four groups, you can treat that as two families of 6 comparisons or one family of 12, and both positions are defensible; what is not defensible is picking one after seeing the p values. This is the same problem preregistration solves, and one sentence in the Method handles it: "Holm's correction was applied separately within each dependent variable, across the six pairwise comparisons of each."

There is a legitimate argument for not throwing everything into one family, and there is an abused version of that same argument. The legitimate version is that two contrasts answering two distinct research questions, supporting two separate conclusions, do not form one family. The abused version is declaring that each of your fifteen comparisons answers a hypothesis of its own and leaving the section without correcting anything. An experienced reviewer separates the two in seconds, and the signal they look for is whether those hypotheses were written down beforehand or turned up when a p of .043 needed justifying.

Bonferroni, Šidák and Tukey

Bonferroni (Dunn, 1961) is the simplest and the most general: divide α by the number of comparisons, or equivalently multiply each p by that number. With six comparisons the threshold moves from .05 to .008; with fifteen, to .003. Its virtue is that it asks for nothing: it works for pairwise comparisons, complex contrasts, and tests of different kinds mixed together, and it still controls the family-wise rate even when the contrasts are correlated with one another. Its flaw is the price: it is conservative, increasingly so as m grows, to the point where in a large family the adjustment leaves you without the power to detect effects that are really there.

Šidák (1967) does the same job with the exact formula instead of the approximation: the threshold is 1 − (1 − α)^1/m. It is slightly less conservative than Bonferroni, and it is worth knowing how slightly before arguing about it with anyone. With six comparisons Bonferroni sets the bar at .0083 and Šidák at .0085; with ten, at .0050 and .0051. The gain lives in the fourth decimal, and Šidák's formula assumes the contrasts are independent, which is precisely what pairwise comparisons are not. If your decision changes between .0050 and .0051, the correction is not your problem.

Tukey is a different animal, and for the standard job it is the best of the three. Instead of splitting α blindly, it uses the studentised range distribution, which models the difference between the largest and smallest of k group means directly. It is built for one very specific family — all pairwise comparisons following an ANOVA — and within that family it controls the error rate exactly, with more power than Bonferroni. You can see it in the intervals: in our trial, Tukey's simultaneous margin is ±3.37 points against Bonferroni's ±3.47, and the CBT versus psychoeducation comparison lands at .057 under Tukey against .071 under Bonferroni. Neither lets it through, but Tukey loses less on the way.

The price of Tukey is its assumptions: homogeneous variances and roughly equal group sizes. When the variances genuinely differ, the alternative is Games-Howell (Games & Howell, 1976), which computes a separate standard error and its own degrees of freedom for each pair and does not require homogeneity. When the omnibus was not parametric, the post hoc is not either: a Kruskal-Wallis is followed by Dunn's test with its own adjustment, and that carries its own writing rules. Two more worth recognizing so that you do not reach for them by default: Scheffé protects every contrast imaginable, complex ones included, which makes it needlessly severe if all you are doing is pairs; and Fisher's LSD applies no correction beyond requiring a significant omnibus, which holds the family-wise rate with three groups and stops holding it with four or more.

Holm and the FDR: one dominates Bonferroni, the other controls something else

Holm (1979) is Bonferroni in steps. Sort your p values from smallest to largest and compare the first against α/m, the second against α/(m − 1), the third against α/(m − 2), and so on until one fails; from there you stop, and everything remaining is non-significant. With six comparisons and uncorrected values of .002, .009, .019, .031, .048 and .310, the first is compared against .0083 and passes, the second against .0100 and passes narrowly, the third against .0125 and fails. Two survivors where Bonferroni leaves one: Bonferroni demands .0083 of all six equally, and that .009 comparison misses by less than a thousandth.

What matters about Holm is that the gain is free. It offers exactly the same guarantee as Bonferroni on the family-wise error rate, under any dependence structure between the contrasts, and it is uniformly more powerful: it never rejects fewer hypotheses than Bonferroni on the same set of p values, and sometimes it rejects more. Put another way, there is no situation in which you would prefer Bonferroni to Holm if the family-wise rate is what you care about. R has known this for decades, which is why p.adjust() and pairwise.t.test() ship with Holm as the default method; Bonferroni still being the one you see most in manuscripts is inertia, not judgment.

The Benjamini-Hochberg FDR (1995) is not a softer version of the above: it controls a different quantity. Instead of the probability of making any false positive, it controls the expected proportion of false positives among the comparisons you declare significant. With q = .05 — this q is the false discovery level you accept, nothing to do with Tukey's studentised range q — the promise is that on average no more than one in twenty of the ones you announce will be wrong, not that your probability of being wrong even once is .05. It is a step-up procedure: each ordered p is compared against (i/m) × q, which here gives thresholds of .0083, .0167, .0250, .0333, .0417 and .0500, so the first four survive.

One, two or four comparisons out of the same six: that is the size of the decision. And it matters who makes it. The FDR is in its place when you are screening: a whole correlation matrix, item-by-item comparisons, dozens of outcomes in an exploratory study, where declaring a few false positives is acceptable if the trade is not missing the true ones. It is out of place in a confirmatory four-arm trial with six comparisons: there the question is whether this particular difference exists, the family-wise rate is the right guarantee, and switching to FDR to go from one significant comparison to four looks far too much like choosing the method by its answer.

How the adjusted p value is written

There are two conventions and both are correct, but only if you say which one you are using. The first, the one SPSS, R and jamovi hand you by default and the one that fits APA 7's rule of giving the exact value, is to report the already-adjusted p and compare it against .05: p = .057, not significant. The second is to report the uncorrected p and declare the corrected threshold: p = .012, above the Bonferroni-adjusted α of .008. What you cannot do is mix them, and the most frequent mix is reporting the adjusted p and comparing it against the adjusted threshold, which corrects twice and turns almost everything non-significant.

The adjustment breaks a few rounding habits. An uncorrected p of .0004 is a perfectly legitimate "p < .001", but multiplied by six it becomes .0024, and then it is written p = .002. A p value that sat below .001 almost never stays there after correction, and dragging the "< .001" across from the uncorrected output is an error you catch with one multiplication. At the other end, an adjusted p cannot exceed 1: .310 × 6 would give 1.86, so it is truncated, and SPSS prints 1.000, which is reported as it stands or as p > .999.

Label the column. If the original and the adjusted p both live in your table, use subscripts — p_Tukey, p_Holm, p_FDR — or a footnote naming the method and the number of comparisons. Two columns headed "p" with nothing to tell them apart is the commonest way this goes wrong, and in SPSS it happens almost by accident: the EMMEANS pairwise comparisons table calls "Sig." a value that already carries the adjustment, and the only thing that says so is the "Adjustment for multiple comparisons: Bonferroni" note underneath. The rest of the conventions for writing p are in how to report p values in APA 7.

The confidence interval has to carry the same correction as the p, and this is where inconsistencies become visible at a glance. The CBT versus psychoeducation comparison in our trial has an uncorrected 95% CI of [0.74, 5.86], which excludes zero; its Tukey simultaneous interval is [−0.07, 6.67] and its Bonferroni interval, computed at 99.17% because 1 − .05/6 is that level, is [−0.17, 6.77]. Both include zero, as they should alongside p = .057. An interval excluding zero sitting next to an adjusted p of .057 in the same row is a contradiction, and it is the first one to jump out at anyone reading the table carefully.

Planned or post hoc, and why the correction is declared first

The difference between a planned comparison and a post hoc one is not in the formula, it is in when you decided it. A planned comparison is written down before you see the data, comes out of the hypothesis, there are few of them, and it need not be a pair: "both active treatments combined against the waiting list" is a perfectly respectable planned contrast and appears in no post hoc table. A post hoc comparison is exploratory by definition, and its family is not the comparisons you ran but every comparison you could have run once you had seen the data. That is what the correction is paying for.

Two practical consequences follow. The first is that planned contrasts do not need a significant omnibus to justify themselves: if the hypothesis said group A differs from group C, that contrast is tested and reported whatever the overall F does. The second is that k groups admit k − 1 orthogonal contrasts, which carve up the between-groups variance without overlapping, and the classic argument for leaving them uncorrected — or lightly corrected — is precisely that they do not overlap; worth knowing that orthogonality carves up the between-groups variance without holding down the family-wise rate, which climbs with every contrast you add. The catch is that "planned" is a claim about your memory unless it is written down somewhere with a date. Preregistration is what makes it checkable, and since Simmons, Nelson and Simonsohn (2011) the field has a fairly good idea of how far a result moves when these decisions are taken with the data in view.

The same applies to the correction itself, and here the numbers in this article are the entire argument. On the same six p values, Bonferroni lets one comparison through, Holm two and the FDR four. If you choose the method after seeing the output, the method is a result of your data rather than a decision of your design; the p you publish no longer means what it claims to mean, and it makes no difference that the formula was applied correctly. The fix costs one sentence in the Method, written beforehand: "The six pairwise comparisons between the four conditions were planned, with Holm's correction applied to that family of six."

That leaves the other half, which is what you show afterwards. Reporting only the significant comparisons is selection bias, and it is not a matter of style. The adjustment depends on m, so if you run six comparisons and show three, the reader can recompute nothing: they see three rows of adjusted values built on a number that never appears, and those three were selected precisely for being small. All m comparisons go in, with their differences, their intervals and their adjusted p values, in a table if there are many; the ones that came out non-significant are what make the significant ones interpretable. If what is stopping you is the work of laying the table out in the right format, the APA results formatter puts the italics and the leading zeros in for you.

Already drafted it? Paste your Results section into Review one section: it reads it the way a reviewer would and returns the first objection, attached to the sentence that causes it.

Frequently asked questions

Which is better, Bonferroni or Tukey?

For the full set of pairwise comparisons after an ANOVA, Tukey, because it is built for that exact family and controls the error rate with more power. In the same trial, a comparison with a 3.30-point difference comes out at p = .057 under Tukey and p = .071 under Bonferroni, and the simultaneous interval margin is ±3.37 against ±3.47. Bonferroni wins when your set of contrasts is not all the pairs: complex contrasts, a subset chosen in advance, or tests of different kinds mixed together. And if you are going to use Bonferroni, use Holm, which gives the same guarantee and never rejects fewer.

How do you report a Bonferroni correction in APA 7?

By naming the method, saying how many comparisons it was applied to, and giving the already-adjusted p value: "The six pairwise comparisons were tested with a Bonferroni correction; the CBT group scored lower than the waiting list, M_diff = 4.70, 99.17% CI [1.23, 8.17], p = .002, d = 0.94." If you would rather report the uncorrected p, then declare the threshold (α = .05/6 = .008) and compare against it — but do not do both at once.

Do you report the adjusted p value or the original one?

The adjusted one, compared against .05: it is what SPSS, R and jamovi return by default and what fits APA 7's rule of giving the exact value. Label it so it cannot be confused with the original: p_Tukey, p_Holm, p_FDR, or a footnote with the method and the number of comparisons. Watch two rounding details: a p of .0004 multiplied by six is .002 and is no longer written "< .001", and an adjusted p truncates at 1.000 because it cannot exceed 1.

What is the difference between planned and post hoc comparisons?

When they were decided. Planned comparisons are written down before you see the data, come out of the hypothesis, are few, may be complex contrasts, and do not need the omnibus to be significant. Post hoc comparisons are decided afterwards, and their family is not the comparisons you ran but every comparison you could have run, which is why they pay a heavier correction. Without a preregistration or a dated protocol, "planned" is a claim nobody can check, and reviewers know it.

What is the FDR correction and when should you use it?

The Benjamini-Hochberg FDR (1995) controls the expected proportion of false positives among the comparisons you declare significant, not the probability of making any. With q = .05, on average no more than one in twenty of the ones you announce will be wrong. It fits when you are screening many contrasts at once — a correlation matrix, item-by-item comparisons, an exploratory study with dozens of outcomes — and it does not fit a confirmatory trial with six comparisons, where the family-wise rate is the appropriate guarantee. On six p values of .002, .009, .019, .031, .048 and .310, the FDR declares four significant and Bonferroni one.

Do you run post hoc tests if the ANOVA is not significant?

With tests that control the family-wise rate on their own, such as Tukey or Holm, technically you can: they do not depend on the overall F. With Fisher's LSD you cannot, because its only protection is precisely the requirement of a significant omnibus. In practice, starting to look at pairs after a non-significant F is exploration, and that is how it has to be reported, with every comparison visible and without turning one of them into the conclusion of the paper. What does not work is reporting the omnibus only when it lands and the post hoc tests only when they land.

With the mean difference, its interval, the adjusted p, the effect size and the name of the correction alongside its number of comparisons, your post hoc tests are reported the way APA 7 asks. When the design gets harder — variances that differ sharply between groups, unequal cell sizes, comparisons nested inside an interaction, or one dataset with several outcomes — deciding where the family ends stops being trivial, and that is what my statistical consulting is for.