How to Tell if a Paper's Statistics Are Made Up

You are reading the paper your whole thesis chapter leans on, and something will not sit still. The Method says N = 124 and the t test on the next page carries 118 degrees of freedom. The percentage of women in the abstract does not match Table 1. A p that looked far too large to you comes with an asterisk beside it. So you sit there, uneasy, unsure whether you have found a real error or simply misread something, because saying it out loud feels like accusing a stranger of something serious. The checks below run on the PDF and a calculator, they need no raw data, and they all answer the same narrow question: can this number be correct, exactly as printed? When it cannot, the cause is almost always a typo — and the useful part of this article is knowing when to let it go and when to stop trusting the conclusion hanging off it.

The p value that does not match its own degrees of freedom

Every published test carries everything you need to rebuild it: the test statistic, the degrees of freedom and the p value. Those three numbers are not independent. Fix any two and the third is determined, which means anyone can audit the third without touching the data. Read t(58) = 2.03, p = .02 and you can check it: the two-tailed p for that t on 58 df is .047. Nothing collapses — the conclusion survives — but the printed number is not the one those data produce.

The case that matters is the one that crosses the threshold. t(58) = 1.75, p = .04 reads as a significant result, and the two-tailed p for that t is .085. The usual explanation is not that anyone invented anything: it is an undeclared one-tailed test, because the one-tailed value is .043 and rounds to exactly that .04. That is still a serious reporting problem, but it is a different problem from the one you were imagining, and it calls for a different question to the authors. How the value should have been written in the first place is covered in how to report p values in APA 7.

This can be automated. statcheck reads the text of a paper, finds the results written in APA format and recomputes each p; there is an R package and a web app where you upload the PDF. Using it, Nuijten and colleagues (2016) recomputed more than 250,000 p values across eight psychology journals and nearly thirty years of issues: half the papers reporting null hypothesis tests contained at least one inconsistency, and one in eight contained one large enough to flip the conclusion about significance.

Its blind spot matters as much as its result. statcheck only sees results written in canonical APA style — t, F, r, chi-square and z with their degrees of freedom — in running text. It does not read regression tables, does not open figures, and knows nothing about mixed models or structural equation models. A clean statcheck report does not mean a clean paper; it means almost nothing was examined. It also flags anything that fails to round correctly, so fractional Welch or Greenhouse-Geisser degrees of freedom rounded off in the text get flagged when nobody has done anything wrong.

The N that changes between tables

Degrees of freedom are a head count in disguise, which is why they give things away. An independent-samples t test carries n₁ + n₂ − 2 degrees of freedom, so t(118) = 2.41 means 120 people. A three-group ANOVA reported as F(2, 117) also means 120. A bivariate correlation computed on 124 people carries 122 df. When the Method declares N = 124 and the paper never mentions a single missing value, you already have two different sample sizes living inside the same Results section.

There is nearly always a dull and entirely legitimate explanation. Listwise deletion in one analysis and pairwise in another. Four people who abandoned that questionnaire halfway. Two cases excluded under a criterion that is written down, three pages earlier. The problem is not that the N drops; it is that it drops silently, because then the reader cannot tell whether the 120 in the t test are the same 120 in the ANOVA. That one-line note under the table is among the cheapest things in a manuscript and among the most frequently missing, and I go through the whole list in how to check your statistics before submitting.

The variant worth a second look is when the N goes up. An analysis with more cases than the declared sample cannot be explained by missing data: it is explained by duplicated rows from merging two files, by an undeclared imputation, or by a later wave of recruitment that made it into some analyses and not others. None of the three is fraud by default, and all three change what the paper can support.

Percentages that will not fit their denominator

A percentage computed on n = 47 can take exactly 48 values: the multiples of 100/47, that is, steps of 2.13 points. A reported 34.0% works, because 16 out of 47 is 34.04%. A reported 35.0% matches nothing at all: 16 of 47 gives 34.0% and 17 gives 36.2%, and there is nothing in between. This is thirty seconds of arithmetic and it finds more than you would expect, especially in small clinical samples where one participant is worth two percentage points.

The commonest version is not an impossible percentage but a percentage computed over a different denominator. The abstract says 68% of the sample were women; Table 1 says 61 of 94, which is 64.9%. That usually means the abstract was written against an earlier version of the dataset, before the non-completers were dropped, and nobody revisited it. It is a typo, and it is the kind of typo a careful reader silently translates into “these authors have not reread their own paper”.

Do not flag as an error what is only rounding. A five-category column can legitimately sum to 99.9% or 100.1%, and in a multiple-response item the total exceeds 100 by design. The signal is a mismatch rounding cannot explain: a column summing to 104%, a percentage that is no multiple of 100/n for any plausible n, or two figures in the same row that demand incompatible denominators.

Means and SDs the scale does not permit

If each person's score is a whole number — a single item, or the sum of several integer-scored items — and the mean is taken over n people, that mean has to be a multiple of 1/n. With n = 28, the possible means around 3.44 are 3.43 (96/28) and 3.46 (97/28). A mean of 3.44 does not exist: no combination of integer responses from 28 people produces it. The check is called the GRIM test, Brown and Heathers formalized it in 2017, and it works while the sample is small — above roughly a hundred cases the grid of possible values becomes finer than two-decimal rounding and stops discriminating anything.

The standard deviation has a ceiling of its own, and that one keeps working in large samples. A variable bounded between 1 and 5 with a mean of 4.50 cannot have an SD above the square root of (4.50 − 1) × (5 − 4.50), which is 1.32; with the n − 1 correction and 60 participants the ceiling rises to 1.33. What sets it is the most extreme distribution the scale allows, everyone answering 1 or 5 and nobody in between: with a given n it is not always reached exactly, but nothing can go past it. A table reporting M = 4.50 and SD = 1.55 on a 1-to-5 scale is describing something that cannot have happened, and the check fits in one spreadsheet cell.

When the mean, SD, n and range are compatible but only barely, SPRITE takes it further by reconstructing which integer samples would produce exactly those values. Sometimes the only compatible distribution is absurd for what the paper claims to have measured: a comfortable mean wellbeing score that requires half the sample pinned at the floor. That is a second-pass tool for someone who already knows what they are looking at, because a strange distribution is strange, not impossible.

GRIM breaks easily and you need to know when. It does not apply when the individual score is not a whole number: weighted items, standardized scores, scales with half-point options, or a score that is already the average of several items, where the grid becomes 1 divided by n times the number of items and turns far too fine to catch anything. It also fails when the n behind that variable is not the study N, which is exactly what happens with missing data. Before you take an incompatibility seriously, work out how the score was built — that is what reverse items and scale scores covers, and it is where half of a beginner's findings evaporate.

Terminal digits, and why this signal is the weakest

Across a large set of measured numbers, the final decimal digit should be spread roughly evenly between 0 and 9. People inventing numbers by hand do not manage it: zeros and fives pile up, repeated digits go missing, and sequences appear that feel random to a human and do not to a counter. Applied to a whole table of means and standard deviations, the imbalance becomes visible.

The most developed version of this idea in clinical research is the Carlisle method, which looks not at digits but at the baseline characteristics table of a randomized trial. If allocation really was random, the p values comparing groups at baseline must be uniformly distributed between 0 and 1. Too many p values close to 1 — groups suspiciously similar to each other — is exactly as improbable as too many small ones. Carlisle (2017) ran the procedure over 5,087 trials published in anesthesia and general medical journals and flagged a small set whose baseline distributions were very hard to explain by chance.

It comes last here because it behaves worst on real data. An instrument that rounds to 0.5, a scale with six possible values, a set of thirty numbers, a figure computed in two steps with intermediate rounding: all of that produces non-uniform terminal digits with nobody touching anything. The four earlier checks say “this number cannot be what they say it is”. This one says “this set of numbers surprises me”, and being surprised is not evidence. If it is all you have, you have nothing.

It is almost always a typo, and that needs saying out loud

The base rate settles it. If half the psychology papers reporting tests contain at least one p that disagrees with its own degrees of freedom, nobody seriously claims half of psychology is fabricated. What exists is a chain: the software output, an intermediate spreadsheet, the manuscript table, a reviewer who forces one analysis to be rerun, the table half-updated because four cells were changed and the fifth was missed. Every link in that chain loses a decimal, and none of them involves bad faith.

An inconsistency is a much smaller claim than it looks. It says these numbers, together, are not compatible. It does not say which of the two is wrong. It does not say why. It does not say whether the substantive result changes. Treating “does not add up” as “was made up” is the same logical jump we criticize in a reviewer who reads p > .05 as “there is no effect”, and it lives in the same family as the rest of the common errors in interpreting results.

The distance between the two sentences is not a matter of tone. One is a two-line email to a corresponding author; the other is a public allegation with career consequences against someone who most likely dragged a cell wrong at eleven at night. The cost lands on you too: publish the accusation, watch it turn out to be a typo, and you have burned the only person who could have given you the data, in front of everyone who was watching.

What to do when you find one

Rule yourself out first, because that is where most first-time findings die. Go back to the Method and check whether the test was one-tailed, whether the analysis uses a subsample, whether deletion was pairwise, whether those degrees of freedom come from a Welch or Greenhouse-Geisser correction, whether the scale has half-point options, whether that table describes the full sample or only the completers. Read it twice before you write to anyone.

If the inconsistency survives that reading, email the corresponding author with no adjectives in the message. “I am using your Table 2 for a meta-analysis and I cannot reconstruct the p value in row 3 from the t and the degrees of freedom — am I misreading it?” That sentence gets answered. The one that opens with “I have detected an error” does not, and it leaves you nowhere to stand if the error was yours. If nothing comes back in a fortnight and the number affects what you plan to do with the paper, the next step is the journal editor, who has a COPE-based procedure for exactly this, and then PubPeer if you want it on the record.

When the paper matters because you are going to reuse its numbers — powering a study, extracting for a meta-analysis, benchmarking your effect against theirs — do not copy the value, rebuild it. Paste the published sentence into the effect size from text calculator and see whether the statistic, the degrees of freedom and the p hold together before it enters your extraction sheet. A number that fails to reconstruct contaminates your meta-analysis, not theirs.

One check pays better than all of the above, and it is not about someone else's paper. It is about yours, before you submit it. A methods reviewer who recomputes your p from your degrees of freedom and lands on a .04 that should have been .085 does not ask for a minor correction: they return the manuscript and wonder aloud what else went unchecked. The difference between those two situations is twenty minutes of arithmetic done at the right moment.

Frequently asked questions

How can I tell if the data in a paper are fabricated?

From the outside you cannot, and that is the honest answer: without the raw data nobody can assert it. What you can establish is whether the published numbers are compatible with each other, which is a different claim and a checkable one. Recompute every p from its statistic and degrees of freedom, cross-check the N those df imply in each table, divide every percentage by its denominator, and compare each mean and SD against what the scale allows. If something fails, you have shown that the number cannot be right as printed — you have not shown why.

What is statcheck and how do I use it?

It is a program that reads the text of a paper, locates tests written in APA format and recomputes the p value from the test statistic and its degrees of freedom. It exists as an R package and as a web app where you upload a PDF and get back a table of what matches and what does not. It only sees canonical APA reporting — t, F, r, chi-square and z with their df — in running text, so it never inspects regression tables, figures or mixed models. A clean report means very little was examined, not that the paper is clean.

What does it mean when a p value is inconsistent with its degrees of freedom?

It means the published statistic and df imply a different p from the one printed. For example, t(58) = 1.75 corresponds to p = .085 two-tailed, so a p = .04 sitting beside it cannot come from those two numbers. The usual causes are a copying error, an undeclared one-tailed test, degrees of freedom left over from an earlier version of the analysis, or fractional Welch df rounded in the text. It is called a gross inconsistency when correcting it moves the result across the .05 threshold, because that changes what the paper concludes.

What is the GRIM test?

It checks whether a reported mean is arithmetically possible. If each person's score is a whole number — a single item, or the sum of several integer-scored items — and the mean is taken over n people, that mean must be a multiple of 1/n: with n = 28, a mean of 3.43 is possible (96/28) and 3.44 is not. Brown and Heathers formalized it in 2017 and it works on small samples, because past roughly a hundred cases the grid becomes finer than two-decimal rounding. It stops applying when the individual score is not a whole number, or when that variable has fewer cases than the full study.

Can I report a paper for statistical errors?

Before reporting anything, email the corresponding author with a question rather than an accusation: most inconsistencies are typos and are resolved by a corrigendum or by an answer that explains what you had missed. If nothing arrives within a fortnight and the number matters, the next step is the journal editor, who normally follows COPE guidance and has a defined process. PubPeer lets you put the comment on the public record, named or anonymously, and is where the cases journals ignore end up. Always describe what does not add up, never why you think it does not.

How many published papers contain statistical errors?

Psychology has a direct measurement: Nuijten and colleagues used statcheck to recompute more than 250,000 p values across eight journals between 1985 and 2013, and found that half the papers reporting tests carried at least one inconsistency, with one in eight carrying one large enough to change the conclusion about significance. The figure says far less about honesty than about the process of copying numbers from software output into a table by hand. It is also the reason to run the same checks over your own manuscript before you submit it.

The five checks take an afternoon to learn and then stay with you for good: they change how you read a paper and, more usefully, how you audit your own before it goes out. If what brought you here was a rejection carrying a line like “the reported statistics are not internally consistent”, the translation of that sentence and the reply it deserves are in why Q1 journals reject you.