The same Results section, before and after

The Results section is finished. You ran the analyses four months ago, a co-author reran part of them after cleaning the file, and the document is on its seventh version. Now you have one narrow decision to make: send it as it stands, or spend two afternoons going back over numbers that, as far as you remember, were fine. This article is that decision with the figures on the table: three paragraphs from a Results section, what was wrong with them, and what they say once repaired. The study is invented; the errors are the ones that turn up every week, and every number recomputes from what the paragraph itself publishes, so you can check them as you read.

The study, and the rules of the game

The manuscript describes a three-arm trial: an intervention, an active control and a waiting list. One hundred and eight people were randomized, 36 per arm, and 90 completed the post-treatment measure: n = 28 in the intervention arm, n = 30 in active control and n = 32 on the waiting list. The variables in the three paragraphs are a 21-item anxiety scale ranging from 0 to 63 and a 10-item rumination scale ranging from 10 to 40.

The rule is that nothing below needs the raw data. The degrees of freedom already tell you the N: 87 is 90 minus 3 groups, 58 is 28 plus 32 minus 2, and 106 is 90 plus 18 minus 2. If those numbers do not match the N in the Method section, there is a problem before you look at anything else. And if they do match, the rest of the check is arithmetic: a statistic, its degrees of freedom and a known distribution.

The three errors belong to the three families that account for almost everything a reviewer flags in Results. Arithmetic. The number does not follow from its own ingredients. Precision. The number is there, but without the uncertainty that goes with it, so it claims more than it can. Justification. The analysis may be executed perfectly well and still be indefensible, because nobody has said why that test and not another.

Paragraph 1: the p that does not follow from its degrees of freedom

Before. “Analysis of variance showed significant differences between the three groups in post-treatment anxiety symptoms, F(2, 87) = 4.31, p = .003. Means were 12.4 (SD = 4.1) in the intervention group, 13.5 (SD = 4.3) in active control and 15.6 (SD = 4.5) on the waiting list.”

An F of 4.31 on 2 and 87 degrees of freedom leaves 1.64% of the distribution above it. The p value is .016, not .003. Put the other way round, which is how a reviewer reads it: for p to be .003 on those same degrees of freedom, the F would have had to be 6.21. There is no room for interpretation here and no need for a language model to have an opinion; it is a cumulative distribution function and it comes out the same on any computer in the world. The interesting part is that the F itself is right: take the three means, the three standard deviations and the three group sizes from the paragraph, build the between-group and within-group variance, and you get 4.31. So the analysis was done properly and what traveled badly was the p value, copied from another row of the output or from an earlier version of the analysis that was rerun after two cases were excluded and from which only some numbers were updated. It is the dullest error there is, and it is also the most common.

It is worth saying how serious it is, because overstating it here would be a con: .003 and .016 fall on the same side of .05, so the paragraph's conclusion does not change. In the terminology of this field that is not a gross inconsistency, it is an inconsistency. The problem is not what it does to that result, it is what it does to the rest: a reviewer who finds a p that does not follow from its degrees of freedom in the second paragraph of Results stops reading the manuscript and starts auditing it. If you want the detail of how each p value is written, it is in how to report p values in APA 7.

After. “Analysis of variance showed differences between the three groups in post-treatment anxiety symptoms, F(2, 87) = 4.31, p = .016, η_p² = .090, 90% CI [.010, .182]. Means were 12.4 (SD = 4.1; n = 28) in the intervention group, 13.5 (SD = 4.3; n = 30) in active control and 15.6 (SD = 4.5; n = 32) on the waiting list. The effect size interval spans values from very small to moderate, so the magnitude of the group effect is estimated with limited precision in this sample.”

The effect size also comes out of the paragraph itself: η_p² is the numerator degrees of freedom times the F, divided by that same quantity plus the denominator degrees of freedom. With 2, 4.31 and 87 it gives .090. The 90% interval is the convention for η² rather than an oversight, because the F is a one-tailed test and the 90% interval is the one that corresponds to a 5% contrast.

Paragraph 2: the effect size with no interval

Before. “The intervention group scored lower on rumination than the waiting list, t(58) = 2.14, p = .037, d = 0.55, representing a medium effect size.”

There is no arithmetic error here, and it is worth saying so rather than selling smoke: the d squares with its t. With 28 and 32 people, d is the t multiplied by the square root of the sum of the reciprocals of the two group sizes, that is 2.14 times 0.259, which gives 0.55. All consistent. What fails is something else: the paragraph presents a point estimate as if it were a finding and labels it with an adjective, “medium”, that lends it a firmness it does not have. “Medium” is not a property of the intervention, it is a conventional label stuck onto a number this study was never going to pin down. The writing rules, down to the italics and the leading zero, are in how to report Cohen's d in APA 7.

The 95% interval for that d, computed from the noncentral t distribution, runs from 0.03 to 1.07. Read that slowly: on these data, the effect of the intervention on rumination is compatible with something close to nothing and equally with something large. The half-width is 0.52, wider than the estimated effect itself. For that interval to be ±0.20 you would need roughly 200 people per group, and the study has 28 and 32. At those sizes, power to detect an effect of 0.55 was 55%: barely better than a coin toss. None of this invalidates the result; it places it, which is why in the corrected version the adjective disappears and the interval replaces it, saying the same thing without promising anything.

After. “The intervention group scored lower on rumination than the waiting list at post-treatment (M = 21.0, SD = 5.6, n = 28 versus M = 24.1, SD = 5.6, n = 32), t(58) = 2.14, p = .037, d = 0.55, 95% CI [0.03, 1.07]. The mean difference was 3.1 points, 95% CI [0.20, 6.00]. The lower bound of the interval approaches zero, so these data are compatible with both a trivial and a large effect, and the estimate of magnitude should be treated as preliminary.”

Notice that the corrected version adds the mean difference in scale points, 3.1 points with its own interval. That is the number a clinician can judge without translating anything: on a scale running from 10 to 40, three points are three points. The d standardizes so that studies can be compared; the raw difference is what lets anyone argue about whether it matters.

Paragraph 3: the test nobody justified

Before. “Participants who dropped out of the study (n = 18, M = 22.3, SD = 8.3) had higher baseline anxiety scores than those who completed it (n = 90, M = 19.4, SD = 4.6), t(106) = 2.09, p = .039. This was taken into account when interpreting the treatment effects.”

The arithmetic is again correct: with those means, those standard deviations and those sizes, Student's t comes out at 2.09 and its p is .039. The problem is that Student's t assumes the two variances are equal, and the paragraph itself publishes two standard deviations, 8.3 and 4.6, whose variance ratio is 3.3. And it is not just any inequality: the smaller group, the one with 18 people, is the more variable one. When that happens Student's t becomes liberal, meaning it hands out more significant results than it should. That p = .039 is biased downwards, and the paragraph says not one word about why that test was chosen.

With Welch's t, which does not assume homogeneity of variance and costs exactly the same to compute, the result is t(19.14) = 1.44, p = .166. The conclusion turns over: what was announced as a difference between those who drop out and those who stay no longer holds. Delacre, Lakens and Leys (2017) argue for exactly this, using Welch by default, and advise against the two-step procedure of testing equality of variances first in order to decide which t to run afterwards. So the correction is not “you are missing Levene's test”: it is “use the test that does not need the assumption, and say so”.

There is a second layer a real reviewer will flag. With 90 and 18 people and those two standard deviations, the Welch comparison only reaches 80% power for differences of around 6 scale points, and the observed one is 2.9. In other words, the analysis was never in a position to settle the question in either direction. A non-significant result here would not demonstrate that attrition was random either, and that is the claim half of all manuscripts slip into the Discussion with nobody looking.

After. “Baseline anxiety was compared between those who completed the study (n = 90, M = 19.4, SD = 4.6) and those who dropped out (n = 18, M = 22.3, SD = 8.3). Because the variances differ appreciably and the sample sizes are unequal, Welch's t test was used, which does not assume homogeneity of variance: t(19.14) = 1.44, p = .166. The difference was 2.9 points, 95% CI [−1.32, 7.12]. At these sample sizes and this dispersion, the comparison would only reach 80% power for differences of around 6 points, so it can neither rule out nor establish an association between dropout and baseline severity.”

What actually changed

The data are the same. Nobody else was recruited, no new analysis was run that would require reopening the file, and not one design decision was touched. The first result is still significant, because .016 sits below .05 exactly as .003 did. So is the second. The only one that falls is the third, and it falls because it was never standing: that p of .039 was the by-product of a test that did not apply.

What changes is what the text claims. The three corrected paragraphs come to 280 words where the originals came to 123: 157 words more, about a third of a column in almost any journal's layout. In exchange there are three reviewer comments you will not receive, and they are three of the most expensive kind, because they arrive in the first round and force you to rewrite Results and Discussion at the same time.

It also changes what the Discussion can say. If the attrition comparison is not significant under the right test and did not have the power to be significant anyway, the sentence “groups were comparable at baseline” leaves the manuscript and an honest limitation takes its place. That is worse for the story and better for the paper: a declared limitation is a limitation the reviewer does not have to discover, and a reviewer who starts discovering things rarely stops at the first one.

Why all three make it into the journal

Because they always do. Nuijten and colleagues (2016) ran an automatic recomputation over more than 250,000 p values published in eight psychology journals between 1985 and 2013: half of the articles using null hypothesis significance testing contained at least one p inconsistent with its test statistic and degrees of freedom, and one in eight contained a gross inconsistency, the kind that may have changed the statistical conclusion. Gross inconsistencies were also more prevalent among p values reported as significant than among non-significant ones, which points to a systematic bias in favor of significant results.

That recomputation, the one in the first paragraph, is already solved and it is free. The statcheck package, by Michèle Nuijten and colleagues, is on CRAN under a GPL-3 license and has a web version at statcheck.io: you upload a pdf, an html or a docx and it recalculates the p values. It recognizes correlations and t, F, χ², z and Q tests as long as they are reported completely and in APA style. Run it. It is the best ratio of cost to saved trouble available in this trade today, and saying otherwise in order to sell something else would be a lie.

Its limits are declared by the tool itself and are worth knowing: it usually cannot find statistics inside tables, in psychology journals it detects roughly 60% of the significance tests, and a small formatting deviation, square brackets instead of parentheses or a semicolon instead of a comma, is enough for a result to slip past it. Checked in September 2026, because these things change.

And above all: recomputing p values does not touch the other two paragraphs. A missing interval is not an arithmetic inconsistency, there is nothing to recalculate. Nor is a badly chosen test: the numbers are computed correctly, what is wrong is which ones were computed. Those two still depend on somebody reading the section knowing what to look for, and peer review is not that filter: a reviewer reads, a reviewer does not recompute. Which is why all three paragraphs in this article would have gone through.

How to put your own section through this

By hand is possible, and it takes an afternoon. Recompute every p value from its statistic and its degrees of freedom. Check every effect size against the t or the F it shares a sentence with. And write, for each test, a single sentence saying why that one and not another; if the sentence will not come, you have found your problem. The full sweep, in order, with the eight checks you can run without opening the data, is in how to check your statistics before submitting.

The automated route does the same pass over the whole manuscript. Previum recomputes p values from the degrees of freedom in code, not with a model; it checks effect sizes against the author's own F or t; and it verifies references one by one against eight databases: Crossref, OpenAlex, Semantic Scholar, PubMed, DOAJ, arXiv and OpenLibrary. The prose of the report is written by a model, but around numbers it cannot change. If you want to see the exact shape of the output before uploading anything, the full example report is there.

What it does not do, so there are no surprises: it does not decide which test you should use. It flags that the justification is missing and shows you what your own numbers imply, as in the third paragraph above, but choosing between Welch, a mixed model or a resampling procedure is a judgment call, and you make it, or you hand it to somebody. When the problem is design rather than wording, that is consultancy, which is human work and is billed by the hour.

And it is not peer review. It is author self-assessment of your own manuscript before you submit it: the same reading you would give it yourself if you had the distance and the time. The manuscript is not stored, the data processing agreement is published, and the sub-processors are named one by one.

Frequently asked questions

If statcheck already recomputes p values and it is free, why would I want anything else?

For the first paragraph, you do not want anything else: run statcheck and fix whatever comes out. What it does not cover are the other two cases in this article. An effect size without an interval is not an arithmetic inconsistency and there is nothing to recalculate; a badly chosen test is not one either, because the numbers are computed properly and the error lies in which ones were computed. Add its declared limits: it does not read tables, and in psychology it detects roughly 60% of the tests. The part it solves it solves very well and for nothing; the rest is still reading.

I fixed the p and it is still significant. Do I have to mention that anywhere?

If the manuscript has not been submitted yet, no: you correct the number and move on. There is nothing to declare about a draft. If the article is already published the answer changes, and it depends on the size of the mismatch: when the corrected p falls on the same side of .05 as the published one, a minor correction usually suffices; when it crosses over, the editor will want to revisit the result itself. In both cases you write to the editor rather than let it pass.

My d interval runs from almost zero to a lot. Is it not better to leave it out?

It reads better to your supervisor and worse to the reviewer, who will compute it in thirty seconds from your t and your ns. A wide interval is not a defect of the analysis, it is information about the size of the sample, and presenting it yourself lets you frame it: preliminary, hypothesis-generating, to be replicated with more people. Hiding it turns an ordinary limitation into a finding that does not survive the first check.

A co-author ran the analyses and I no longer have the original output. Can I check anything?

Almost everything in this article, yes, and it is the most common situation. Recomputation needs neither the SPSS output nor the syntax: it needs the statistic, its degrees of freedom and the N, and all three are already written in your manuscript. The three errors above are detectable from the paper alone. What you cannot do without data is change the analysis, and for that you do have to call the co-author back.

Will Previum tell me which test I should have used?

No, and be wary of anything that promises it will. It flags that the justification is absent and shows you what your own numbers imply: that your two standard deviations differ this much, that the small group is the more variable one, that at that n the power reaches only so far. With that on the table, the choice between Welch, a mixed model or a resampling procedure is yours, because you know the design, or it goes to a statistician.

Does this replace peer review?

No. It is not peer review: it is author self-assessment of your own manuscript before you submit it. Nobody external passes judgment on your work and there is no verdict that carries any weight with a journal. What it does is bring the most mechanical comments forward, which are precisely the ones you can fix on your own and the ones that sting most when they arrive signed by an anonymous reviewer four months later.

Keep the part that does not expire: in a Results section, every number has to follow from other numbers on the same page. The p follows from the statistic and its degrees of freedom, the effect size follows from the t or the F, and the test follows from a sentence explaining why that one. When all three fit, the section can be audited and nothing happens. When one does not, the doubt does not stay put: it spreads to everything else you have written, and that is the expensive part.