Check Your Statistics Before You Submit (Without Reopening the Data)

The doubt always arrives at the same moment. The manuscript is finished, the cover letter is written, the submission portal is open in another tab, and then it hits you: are the analyses actually right? It is a fair question, and it almost never gets answered, because "checking the analyses" sounds like reopening the dataset, rerunning everything and losing a week you do not have.

Part of it does not work that way. A sizeable share of the statistical errors in a paper can be caught without the dataset, without the syntax and without the author: by reading the text that is already written. They are internal consistency failures, numbers that cannot all be true at once, and finding them takes a calculator and half an afternoon. Almost nobody does it before submitting, which is precisely why it is worth doing.

How large is that share? Nuijten, Hartgerink, van Assen, Epskamp and Wicherts (2016) recomputed roughly 250,000 results published in psychology journals between 1985 and 2013 and found that around half of the articles contained at least one inconsistency between the test statistic, its degrees of freedom and the reported p-value. Roughly one article in eight carried an inconsistency that changed the conclusion: the sentence said "significant" and the arithmetic said otherwise, or the reverse. These are published, peer-reviewed papers in respectable journals.

Two things follow. Peer review does not reliably catch this class of error. And if you catch it in your own manuscript before submission, you save yourself an awkward email from an editor or, at worst, a correction notice with your name on it.

Eight checks follow. All of them run on the PDF of your manuscript; none of them needs your data.

1. Recompute the p-value from the statistic and its degrees of freedom

This is the parent of all the others, and the easiest to state: given a test statistic and its degrees of freedom, the p-value is not a judgment call, it is a deterministic function. If you write t(126) = 2.03, the two-tailed p can only take one value. Any package returns it, and a spreadsheet does it in one line.

The formulas, if you want the shortcut:

• t: two-tailed, =T.DIST.2T(ABS(t), df). In R, 2 * pt(-abs(t), df).
• F: upper tail only, =F.DIST.RT(F, df1, df2). In R, pf(F, df1, df2, lower.tail = FALSE).
• χ²: upper tail only as well, =CHISQ.DIST.RT(chi, df). In R, pchisq(chi, df, lower.tail = FALSE).
• z: two-tailed, =2*(1-NORM.S.DIST(ABS(z), TRUE)). In R, 2 * pnorm(-abs(z)).
• Pearson's r: convert to t with t = r·√(df/(1−r²)) and use the first line.

The rounding rule, which is the part nearly everyone skips. When you write t(75) = 2.14 you are not claiming the statistic equals 2.14; you are claiming that, rounded to two decimals, it prints as 2.14. The true value lies between 2.135 and 2.145, so the compatible p-value is an interval, here .0352 to .0360. Only a reported p that falls outside that interval needs explaining. Skip this and you will scare yourself with third-decimal discrepancies that are nobody's error.

One more reading habit worth fixing: "p < .05" is a bound, not a value. It is consistent if the recomputed p can be smaller than .05, not if it equals it. Older literature reports bounds almost exclusively, and treating them as point values turns half a corpus into false alarms.

Three worked examples of how the reading goes:

What a mismatch means. In the overwhelming majority of cases, nothing dramatic: a transcription slip between the output window and the manuscript, a displaced decimal, a row of the SPSS table read one line too low. Two minutes and it is gone. The case that matters is the other one, where the recomputed and the reported p fall on opposite sides of .05. There you are not fixing a typo, you are fixing a claim, and probably the discussion sentence that hangs off it too.

The one-tailed trap. For t, r and z, the one-tailed p is half the two-tailed p, so a result that fails as a two-tailed test can be perfectly consistent as a one-tailed one. Take z = 2.24, p = .012. Two-tailed the p is .025 and it does not fit; one-tailed it is .0125 and it does. Is that an error? Nobody can tell, which is exactly why nobody should accuse you of one: it hinges on a directional test the text never mentions. The fix is yours and it is a single sentence in the analysis plan — all tests two-tailed, or which ones are one-tailed and why. That sentence turns an unresolvable ambiguity into a checkable claim, and it stops you looking like someone who chose the tails after seeing the result. F and χ² are exempt: they have an upper tail only, and halving their p is not "one-tailed", it is a mistake.

2. The p-value that the degrees of freedom cannot produce

Before recomputing anything, there is a coarse check you can do at a glance: with few degrees of freedom, a small p demands a very large statistic. If the statistic is not there, the p-value cannot be what you wrote.

The ceilings worth memorizing for p < .001:

• With 4 df (a paired t-test on five participants) you need |t| ≥ 8.61.
• With 9 df, |t| ≥ 4.78. With 18 df, |t| ≥ 3.92.
• A χ² on 1 df needs 10.83; on 2 df, 13.82.
• An F(1, 10) needs 21.04; an F(2, 20), 9.95.

An example that turns up in pilot studies more often than you would expect: "change over time was reliable, t(9) = 2.26, p < .001". The real p for that t is .050, sitting exactly on the fence. Nothing about nine participants can support a p below .001 at that effect magnitude, and that sentence is usually holding up a line in the abstract.

The rule: sweep your results for any p < .001 attached to a modest statistic on few degrees of freedom. If the statistic does not clear the ceiling for its row, the p-value is wrong.

3. The sample size that changes between method, table and text

This is the most human error of the eight and it shows up in otherwise immaculate manuscripts. The method says 132 participants were recruited. Table 1 adds up to 128. The results section refers to "the 124 participants who completed follow-up". And nowhere does the text say where the missing eight went.

It rarely means anything sinister. Usually the exclusions were decided at three different points in the analysis and the manuscript was written across four sittings. But to a reviewer it is an immediate flag, because there is no way to tell whether the exclusions followed a prior criterion or followed the results. And for an editor, three different sample sizes in the first two pages is one of the cheapest available reasons to sign a desk-rejection without sending the manuscript out for review.

The rule: list every N that appears anywhere in the manuscript — method, flow diagram, each table, each figure, each results sentence, the abstract — in a single column. There should only be three or four distinct numbers, and each one needs a sentence that owns it: N recruited, N analyzed, N for a specific analysis where attrition differed. A fifth number with no owner is work waiting for you.

4. Degrees of freedom that do not match the sample size

This is the natural sequel to the previous check, and it is sharper, because degrees of freedom reveal the N that actually entered each analysis whether or not the text counted it. The correspondences are fixed:

Two things this catches. First: the method reports 100 participants split 48 and 52, and the results show t(92). Those df should be 98, so six cases have gone missing without a sentence — probably listwise deletion on that particular variable, which is entirely legitimate, but it has to be written down. Second: a 3×3 contingency table reported as χ²(2). The degrees of freedom of a χ² test of independence do not depend on the sample at all, they come from the shape of the table, and there they can only be 4.

Two warnings so you do not flag yourself in error. Degrees of freedom with decimals, such as F(1.62, 87.30), are not a mistake: that is the Greenhouse-Geisser or Huynh-Feldt correction in repeated measures, and reporting them that way is correct. A t with decimal df, such as t(43.71), is Welch's correction for unequal variances, also correct.

5. Means and SDs that disagree between two tables

Descriptives usually appear twice in a manuscript: once in the sample characteristics table and again inside the main analysis table or in the text. They get typed at different moments, sometimes from different versions of the dataset, and that is where they drift apart. The same cell reads M = 24.31 (SD = 8.02) in Table 1 and M = 24.13 (SD = 8.20) in Table 3. A reviewer who notices does not think "typo", they think "which version of the data produced the analysis?".

Two extra checks that cost nothing:

The ceiling on a standard deviation. If a variable is bounded between a and b and has mean M, its SD cannot exceed √((b − M)(M − a)). On a 1-to-5 Likert item with a mean of 4.6, the maximum is √(0.4 × 3.6) = 1.20. An SD of 1.5 in that cell is impossible, not merely unlikely. Leave a sliver of slack for the n − 1 denominator and do not chase second-decimal differences, but a clear excess is a number that does not exist.

The GRIM test. When a mean comes from integer data — a count of correct responses, a summed questionnaire score, a single item — that mean has to be a multiple of 1/N. With N = 25 the possible means step in units of 0.04: 3.40, 3.44, 3.48. A reported mean of 3.45 is arithmetically impossible with 25 people, because it would require a total of 86.25. That is the GRIM test (Brown & Heathers, 2017); it works by hand, it only bites at small N, and it does not apply when the mean is an average of item averages, because the underlying values are no longer integers.

6. The effect size that is missing, or that contradicts its own statistic

APA 7 asks for an effect size alongside every primary result, and results sections that are simply a list of p-values still turn up constantly. But beyond being absent, an effect size can contradict the statistic it came from, and that too is visible without data:

• Cohen's d from an independent-samples t with roughly equal groups: d ≈ 2t/√df. For t(98) = 2.55 that gives d ≈ 0.52. If the table says d = 0.18, one of the two numbers is wrong. (With clearly unequal groups the right formula is d = t·√(1/n₁ + 1/n₂).)
• Partial eta squared from an F: η²p = df₁·F / (df₁·F + df₂). For F(3, 156) = 4.20 that is η²p = .075. A manuscript reporting .34 has a cell copied from the wrong row.
• Coherence with the p-value: a large effect in a large sample cannot come with p = .04, and a tiny effect with N = 60 cannot produce p < .001. When statistic, p-value and effect size do not form a coherent triangle, the effect size is usually the culprit — it is calculated last and pasted in worst.

For which index belongs to which design, and how to interpret it without leaning on textbook thresholds, see how to interpret Cohen's d.

7. Multiple comparisons with no correction (or none named)

This one is not arithmetic, it is counting, which makes it just as quick: count how many p-values are in your results section. If twenty of them belong to the same family and no sentence in the method says what was done about it, you have a problem that the reviewer will see before you do.

The numbers that make it obvious: with 10 independent tests at α = .05, the probability that at least one comes out significant by chance alone is 40%; with 20 tests it is 64%. Run twenty exploratory correlations with no correction and finding "something" is the expected outcome, not a discovery.

The rule: group your tests into families — the ones answering the same question — and for each family the method must say one of three things: which correction was applied (Bonferroni, Holm, Benjamini-Hochberg FDR), that the contrasts were planned and pre-specified, or that the block is explicitly exploratory and is read as such. With 20 comparisons, Bonferroni puts the threshold at .0025, which is usually too severe to be useful; Holm or FDR are almost always the defensible choice. Saying nothing is the only option that does not survive review.

A practical note if you are also running check 1: an already-corrected p-value will not match its statistic, and that is not an error. Label those columns (p-FDR, p adjusted) so that you, and everyone after you, can tell at a glance which p-values are supposed to be recomputable and which are not.

8. The analysis that is in the method and not in the results

The last one is the silliest and the most expensive. The method announced five things: descriptives, assumption checks, the mixed ANOVA, the post hoc tests and a moderation analysis by gender. The results section has four. The moderation vanished, and the likeliest reason it vanished is that it came out null.

To a reader, that absence is indistinguishable from selective reporting. It works the same way in reverse: an analysis that appears in the results without having been announced in the method reads as a decision made after seeing the data.

The rule: write every analysis you announce in the method in one column, one line each, and tick them off as you find them in the results. Then sweep in the opposite direction. Anything left over on either side needs a sentence: "the moderation analysis was conducted and was not significant (F < 1); it is reported in the supplementary materials". That is the whole fix, and it converts a hole into evidence of rigor. The same logic applies to the abstract: every claim in it needs a number in the results.

The full sweep, in the order it is done

Half an afternoon, with the PDF on one side and a spreadsheet on the other:

Step 1. Highlight every inferential result in the text and in the tables. Copy them into a sheet: test type, value, df, p, effect size.

Step 2. Add a column with the recomputed p and another with the difference. Sort by the difference and look only at the top rows.

Step 3. Check the degrees of freedom against the N of each analysis. Note every N that does not add up.

Step 4. Cross-check any descriptive that appears twice.

Step 5. Count the p-values, decide the families and write the correction sentence into the method.

Step 6. List method against results in both directions.

One process recommendation: do this before the final writing pass, not after. Many of these corrections change a sentence in the results, and changing a results sentence forces you back through the discussion and the abstract. Leave it to the end and you will either do it badly or do it too late.

What this kind of checking cannot tell you

Worth being clear about, so you do not over-trust it. Recomputation says nothing about whether the analysis was appropriate: only whether the p-value you report follows from the statistic you report. An immaculately transcribed ANOVA on a design that called for a mixed model passes the whole sweep.

It also does not cover non-parametric tests — Mann-Whitney U, Wilcoxon W, Kruskal-Wallis H — because their p-values depend on the approximation and the tie correction the software used, so recomputing from the text would produce discrepancies that belong to the recomputation method rather than to you. The same goes for the Q statistic in a meta-analysis.

And a practical detail that catches people out: this works on text. A result broken across the cells of a Word table arrives as loose figures and is no longer recognisable as "t(98) = 2.55". If your key results live only in tables, the table sweep is one you have to do by hand, column by column. It is one more argument for writing the central findings into the prose as well, which is what APA asks for anyway.

For the wider picture of what a methodological reviewer looks at on a first read, there is why Q1 journals reject you, and the anatomy of a well-built results section is laid out in the complete APA 7 results example.

Doing this sweep by hand works, but it is tedious and it competes with the urge to just submit. If you would rather have it done across the whole manuscript, Previum reads your PDF and recomputes the p-values from each statistic and its degrees of freedom, rounding interval included, inside a Reviewer 2 style pre-review that also looks at the design and the writing. It is the check that separates a typo fixed at home from an email from the editor, and it costs less than reading the paper through for the sixth time.

Frequently asked questions

Can you really detect statistical errors without seeing the data?

Some of them, yes, and not a small share. Anything that is internal consistency of the text itself — that the p follows from the statistic and its df, that the df match the N, that descriptives agree across tables, that the effect size is the one that F implies — is arithmetic on what is already written. What you cannot check without data is whether the analysis was the right one, whether the assumptions held, or whether the exclusions were reasonable.

My recomputed p differs in the third decimal. Is that an error?

Almost certainly not. "t(75) = 2.14" only guarantees the statistic lies between 2.135 and 2.145, so the compatible p-value is an interval, here .0352 to .0360. If your reported p falls inside it, it is consistent. The discrepancies that matter are order-of-magnitude ones (.04 against .005) or any that cross the significance threshold.

I ran a one-tailed test and the recomputation flags it. What now?

Write it down. For t, r and z the one-tailed p is exactly half, so a result that fails as a two-tailed test can be perfectly fine as a one-tailed one, and a reader has no way of knowing which you ran unless you say so. One sentence in the data analysis section — "tests of hypotheses 1 and 2 were one-tailed as they were directional; all others were two-tailed" — settles every line at once. F and χ² have no such option: upper tail only.

Can I write p = .000 if that is what SPSS prints?

No. SPSS prints three decimals, so ".000" means "smaller than .0005", not zero: a p-value of exactly zero does not exist. APA 7 asks for p < .001. Copy editors correct it without asking, and leaving it in signals that the number was copied off the screen rather than read.

What if I find an inconsistency in a paper I have already published?

It depends on whether it changes the conclusion. If it is a transcription slip that leaves the conclusion intact, contact the editor and it usually resolves with a short correction. If the recomputed and reported p-values fall on opposite sides of the threshold, work out which claims in the paper depended on that line before writing to anyone. Either way, reporting it yourself is far better than someone else finding it: these checks are already run automatically across entire journal corpora.

Does this replace having a statistician read the manuscript?

No, and it is not meant to. It is the lower filter: it removes the errors that should never reach a reviewer and that currently reach them in half of all papers. The decisions above it — whether the design supports what you conclude, whether the model fits your data structure, whether the sample was ever large enough for the question — still need someone who understands your study.

References

Brown, N. J. L., & Heathers, J. A. J. (2017). The GRIM test: A simple technique detects numerous anomalies in the reporting of results in psychology. Social Psychological and Personality Science, 8(4), 363-369.

Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016). The prevalence of statistical reporting errors in psychology (1985-2013). Behavior Research Methods, 48(4), 1205-1226.