Six Checks You Can Run on a Paper Without the Data

You have agreed to review a manuscript, or you are about to cite a paper that half your theoretical framework leans on, and all you have is a PDF. No data, no syntax, no output. The usual assumption is that without those you can only comment on the argument and take the numbers on trust. That is not true. A results section is a system of equations that contradicts itself whenever something is wrong, and a good share of those contradictions are visible with a calculator. These are the six that pay off most, in the order worth running them, each with the hand method that makes it work.

1. Recompute every p value from its statistic and degrees of freedom

A p value is not a datum: it is a deterministic function of the test statistic and its degrees of freedom. Given t(58) = 2.10 there is exactly one possible two-tailed p, and it is .040. That turns every fully reported result into a closed check: the text hands you three pieces, and if the third does not follow from the first two, somebody copied wrong, rounded wrong, or changed the analysis without changing the number.

The hand method fits in one cell. In a spreadsheet: =T.DIST.2T(2.10, 58) for a two-tailed t, =F.DIST.RT(3.84, 1, 118) for an F, and =CHISQ.DIST.RT(2.86, 1) for a chi-square. In R they are 2 * pt(-abs(2.10), 58), pf(3.84, 1, 118, lower.tail = FALSE) and pchisq(2.86, 1, lower.tail = FALSE). For a correlation, convert to t first with t = r · sqrt(N − 2) / sqrt(1 − r²): an r = .24 with N = 140 gives t(138) = 2.90 and p = .004, so that one passes.

Two grades of severity, and they should not be lumped together. A plain inconsistency: «t(58) = 2.10, p = .002», where the true value is .040 and the conclusion does not move. And a decision error, which is the one that matters: «F(1, 118) = 3.84, p = .04» in a paper whose method section declares two-tailed tests at alpha = .05, when that statistic on those degrees of freedom returns p = .052. There the effect crosses from significant to non-significant, and the abstract sentence usually crosses with it. Same story with «chi²(1, N = 200) = 2.86, p = .04»: the real value is .091.

Before you write anything, rule out the two legitimate explanations. One is a one-tailed test, which halves the value for a t; if the method declares it, there is no error, and if it does not declare it and half the p values only reconcile once you assume it, that is a finding of its own. The other is a correction for multiple comparisons: a Bonferroni or FDR adjusted p has no reason to match the raw one. Rule out rounding too, because a statistic printed to two decimals is really an interval, so a mismatch in the third decimal is nothing. This check has been automated for years in statcheck, which has an R package and a web version, and when Nuijten and colleagues (2016) ran it across nearly three decades of psychology, they found at least one inconsistent p value in roughly half of the articles that reported statistical tests, and a conclusion-changing error in about one in eight. The reporting rules that make the audit possible at all are in how to report p values in APA 7: a paper that writes «p < .05» and hides the statistic cannot be checked, and that is information too.

2. Reconcile the degrees of freedom with the N the paper claims

Degrees of freedom carry the sample size inside them. You do not need the authors to tell you how many people entered each analysis: the parentheses already say so, and they often say something different from the abstract.

The identities that cover nearly everything you will meet. Independent-samples t test. df = n₁ + n₂ − 2, so t(118) means 120 people. Paired-samples t test. df = N − 1, so t(41) means 42 pairs. One-way ANOVA. F(k − 1, N − k), so F(2, 117) means three groups and 120 people. Multiple regression. F(k, N − k − 1), so F(4, 145) means four predictors and 150 cases. Correlation. df = N − 2. Chi-square. df = (rows − 1)(columns − 1), which does not depend on N at all, and that is precisely why APA makes you put the N inside the parentheses: it is the only place it appears.

The typical use is a subtraction. The abstract says N = 132 and the main analysis is F(2, 117), which is 120. Twelve people are missing and the method never mentions them. There may be a perfectly good reason (dropouts, an exclusion criterion, a group that does not enter that particular model), but the reason has to be written down, because twelve cases vanishing between the abstract and the headline analysis is exactly the shape selective exclusion takes.

Three warnings so you do not raise false alarms. Non-integer degrees of freedom are not an error, they are a correction: usually Greenhouse-Geisser in repeated measures, sometimes Welch in a t test with unequal variances. An F(1.62, 95.62) is written exactly as it should be. In multilevel and mixed models the degrees of freedom are approximated (Satterthwaite, Kenward-Roger) and this check does not apply as it stands. And within one paper it is normal for different analyses to run on different Ns because of missing data; the anomaly is not that they vary, it is that they vary with nobody saying so.

3. Reconcile the Ns across abstract, method and tables

The previous check compares the parentheses against the prose. This one compares the prose against itself. Take the three sources of N every paper has (the abstract, the participants section, and the first column of Table 1) and add or subtract until they contradict each other.

A textbook case: the abstract announces N = 212; Table 1 shows two groups of 104 and 100, which sum to 204; and the main contrast is t(202), which is also 204. The 212 never enters an analysis. Eight people left somewhere between recruitment and results, and the number that reaches the abstract, the one everybody will cite, is the larger of the two. This is not fraud: it is an abstract written before the exclusions were finished and never revised afterwards. But if those eight left for a reason connected to the outcome, the effect you are reading belongs to a different sample.

Inside Table 1 the check is cheaper still: the categories of each variable have to sum to the column total. Sex 145 + 67 = 212; education 38 + 96 + 74 = 208, with four undeclared missing. When a row sums to less than the column there are silent dropouts; when it sums to more, somebody is counted twice, or a non-exclusive category is being presented as though it were exclusive.

What you are after in the end is the full chain: how many were approached, how many consented, how many completed, how many entered each analysis. A paper that reports the whole chain can be audited in two minutes and usually survives it; one that gives a single number and repeats it everywhere is hiding the interesting part without meaning to. How that chain gets declared, and what has to be said about the mechanism behind the loss, is in how to report missing data in APA 7.

4. Divide every percentage by its denominator

A percentage of people cannot be just any number. If the sample is 212, each person is worth 0.47 percentage points and the achievable percentages form a grid with 212 rungs. Multiply the percentage by N and see whether you get a whole person: 68.5% × 212 = 145.22, which is nobody. The two reachable values on either side are 145 people (68.4%) and 146 (68.9%). That 68.5% cannot have come out of that sample.

The second use is more common than the first: the denominator that moves. The paper declares N = 212 and later writes «34 participants (19.1%) met clinical criteria». Out of 212, thirty-four people are 16.0%. The 19.1% comes from dividing by 178, the completers. Neither number is wrong; what is wrong is that the reader cannot tell which base is in play without doing the division, and the one that reaches the abstract tends to be the flattering one.

The same arithmetic applies to means, and there it has a name: the GRIM test (Brown & Heathers, 2017). If a variable can only take whole-number values per person (a Likert item, a count of episodes, a sum of items), its mean over N cases has to be a multiple of 1/N. With N = 47, an M = 3.42 is impossible: 3.42 × 47 = 160.74, and the nearest achievable means are 3.40 and 3.43. The condition is strict and has to be respected: it does not hold for means of means (a scale score built by averaging items is no longer an integer) nor when the N for that particular variable is not the total N.

And one last cheap one: the percentages of a single variable have to sum to 100. Summing to 100.1 or 99.9 is rounding and is fine. Summing to 106 means the categories overlap and nobody said so, or a multiple-response item is being presented as if it were exclusive.

5. Recompute the effect size from the F or the t

An effect size is rarely extra information: in most designs it follows from the statistic and the degrees of freedom, so it can be inverted. Three formulas cover almost everything. Partial eta squared. partial eta² = (F × df₁) / (F × df₁ + df₂). With F(2, 117) = 4.86 that is .077. Between-groups Cohen's d. d = t × sqrt(1/n₁ + 1/n₂). With t(118) = 2.10 and 60 per group that is d = 0.38. R squared and the model F. F = (R²/k) / ((1 − R²)/(N − k − 1)). With R² = .18, four predictors and N = 150 that is F(4, 145) = 7.96.

The useful move is reading the mismatch backwards. If a paper reports F(2, 117) = 4.86 and then partial eta² = .18, the question is not whether the effect is large: it is that for partial eta squared to be .18 the F would have to be 12.84. If it reports t(118) = 2.10 with 60 per group and d = 0.62, the t would have to be 3.40. One of the two numbers comes from another analysis, another subsample, or a calculator fed the wrong input, and which of them is the real one changes the whole interpretation.

A large share of the mismatches you will find here are not errors, they are labels. Partial eta squared and eta squared are different quantities, and the giveaway is the sum: classical eta squared partitions the total variance of the design, so the effects in a factorial cannot add up to more than 1, whereas the partials can. If a paper calls eta² three effects that sum to 1.18, they are mislabeled partials, and that has to be asked, because it changes what each one means. The d in repeated measures behaves the same way: if a paper reports d = 0.36 alongside t(41) = 2.31, that number is t/sqrt(N), which makes it a d_z computed on the difference scores, and it is not comparable with the between-groups d in a meta-analysis. Inverting tells you which formula they used even when the paper never says.

If you are doing this across several papers, keeping it in a spreadsheet gets tedious fast. The effect size from text calculator performs exactly this inversion: you paste the results sentence as it stands in the PDF and it returns the effect size implied by it, which is the number you want to hold up against the one the authors report.

6. Check that the references exist and are the ones the text claims

This check has three levels and they cost wildly different amounts. The first is whether the reference exists. The second is whether the record —authors, year, journal, volume, pages— matches the real work. The third is whether the work says what the sentence citing it claims it says.

The first level is mechanical. If the entry carries a DOI, paste it after https://doi.org/ and see where it lands; if it does not, paste the exact title into search.crossref.org. Two results are suspicious: nothing comes back under that title, and the DOI resolves to an article other than the one cited. That second case is the fingerprint of a reference that was composed rather than copied, and it is what chatbot-generated reference lists produce: real authors, a real journal, a plausible title, and a DOI belonging to somebody else's paper. A block of fifteen references takes ten minutes.

The second level is where the good-faith errors hide. Rebuilding a citation from a record adds co-authors who never signed, swaps the online-first year for the issue year, or turns a preprint into the published article, which sometimes does not even carry the same results. It also brings in a check almost nobody runs: that nothing you cite has been retracted. The Retraction Watch database is integrated into Crossref and is free to query; a retracted paper cited as though nothing had happened, especially one holding up the central argument, is a serious and very quiet problem.

The third level cannot be automated and is the one that warps literatures most: citation drift. Somebody cites a paper for a claim that is close but stronger, the next author cites the citer, and three hops later a moderate correlation in an undergraduate sample has become an established fact. The only way to check it is to open the source and look for the sentence. Do it for at least the three references that hold up the main hypothesis. The full procedure, applied to your own bibliography before submission, is in checking your references before you submit.

What order to run them in, and how to write up what you find

Order matters, because the first checks give you the map. Start with 2 and 3, which are subtractions and tell you how many people are really in each analysis; everything else is interpreted against that. Move to 1, which is where decision errors show up. Then 4 and 5, which lean on the Ns you have already pinned down. Leave 6 for last, because it eats the most time. On an empirical paper of ordinary length, the first five fit in one sitting with a spreadsheet open beside the PDF.

Finding something does not license you to conclude anything about anyone's intentions, and writing it as though it did is the fastest way to get an editor to ignore you. An inconsistency is written up as arithmetic and as a question, with the page and the number you get: «On p. 12 the paper reports F(1, 118) = 3.84, p = .04. Recomputed from the statistic and its degrees of freedom, that test returns p = .052. Could the authors verify the value, or indicate whether a correction was applied?» That is impossible to argue with and very easy to answer, which is precisely what you want.

If the paper is already published and the error is consequential, the sensible order is authors first, editor second if nobody replies, and PubPeer only once those have gone nowhere. Most of the inconsistencies you will find are transcription slips between the software output and the manuscript, and an email fixes them.

The most profitable use of the six, though, is not on somebody else's paper: it is on your own, the day before you submit. Every one of them catches something your eye stopped seeing six months into the manuscript, and every one of them will be run by a careful reviewer, or by the editor's desk before it even reaches review. The longer list, with everything worth checking beyond these six, is in check your analysis before you submit.

Frequently asked questions

Can you review a paper without the raw data?

Yes, and much further than people assume. A results section is full of redundancy: the p follows from the statistic and its degrees of freedom, the degrees of freedom contain the N, percentages contain their denominator, and effect sizes follow from the F or the t. Every redundancy is a check that needs only the PDF. What you cannot check without data is whether the analyses were run on the variables described, or how many were run before this one.

How do I recompute a p value by hand?

In one spreadsheet cell. For a two-tailed t, =T.DIST.2T(2.10, 58) returns .040; for an F, =F.DIST.RT(3.84, 1, 118) returns .052; for a chi-square, =CHISQ.DIST.RT(2.86, 1) returns .091. In R they are 2 * pt(-abs(t), df), pf(F, df1, df2, lower.tail = FALSE) and pchisq(x, df, lower.tail = FALSE). For a correlation, convert to t first with t = r · sqrt(N − 2) / sqrt(1 − r²).

What does it mean when the degrees of freedom do not match the N?

That the analysis ran on fewer people than the paper advertises. An F(2, 117) in a three-group ANOVA is 120 cases; if the abstract says 132, twelve left somewhere. Usually it is missing data with listwise deletion, which is not a problem unless it goes undeclared. Watch for false alarms: non-integer degrees of freedom are the Greenhouse-Geisser or Welch correction, and in mixed models they are approximated, so the arithmetic does not apply.

How can I tell whether a reference is fabricated?

Paste the DOI after https://doi.org/ and see where it goes; with no DOI, search the exact title on search.crossref.org. The two signals are that nothing exists under that title, and that the DOI resolves to a different article from the one cited. That second combination —real authors, real journal, plausible title, somebody else's DOI— is the signature of a chatbot-generated bibliography. While you are there, check that nothing cited has been retracted; the Retraction Watch database is queryable through Crossref.

What is the GRIM test and what is it for?

It checks whether a mean is arithmetically possible. If a variable takes whole-number values per person, its mean over N cases must be a multiple of 1/N: with N = 47, an M = 3.42 cannot exist, because 3.42 × 47 = 160.74 and the nearest achievable means are 3.40 and 3.43 (Brown & Heathers, 2017). It applies only to integer variables (a single item, a count, a sum) and with the N for that variable, not the study total.

What should I do if I find an error in a published paper?

Email the authors first with the page, the number they report and the number you get, and no adjectives. Most are slips made copying software output into the manuscript and end there. If nothing comes back in a few weeks and the error touches the conclusion, take it to the journal editor. PubPeer comes after that, never before, and always with the arithmetic in front instead of the suspicion.

None of these six checks proves a paper is wrong: they prove that two numbers in the same paper cannot both be true, and that is enough to ask a precise question, which is what a review is for. If running them over your own manuscript leaves you unsure which of the two numbers was the right one, or if the mismatch turns out to come from an analysis run on a sample you no longer describe, that is what my statistical consulting is for.