What an automated review catches that your supervisor misses (and the other way round)

Your draft is ready and there are two places to send it before it goes out: your supervisor's inbox, which will take between a week and two months, and a tool that hands you a report in a few minutes. The temptation is to treat that as a choice —either I trust the person or I trust the software— and then settle it out of loyalty, which is the worst way to settle anything. It is not a choice: they are two readings of the same file, looking for different things and missing different things. What follows is the split: what each one finds, what neither of them finds, and the order to put them in so that the most expensive attention in your project does not get spent on decimal points.

Two different readings of the same file

Your supervisor opens the PDF and reads it as a scientist. She goes for the question: whether it holds up, whether the design can answer it, whether the introduction promises something the results do not deliver, whether the discussion claims more than a cross-sectional design can carry, whether a reader in your field will buy it. While she reads, the numbers go past like furniture: she registers "it came out", "medium sample", "an ANOVA", and moves on toward the meaning, which is what she came for.

The machine opens the same file and reads it as an auditor. It does not understand the question and does not need to: it takes each test and checks that the p value is the one that statistic yields with those degrees of freedom, it takes each effect size and checks that it comes out of the F or the t printed two lines above, it takes each reference and looks it up in the databases to see whether it exists. It has no opinion about anything; it checks that the document is consistent with itself.

The overlap between the two readings is smaller than it looks. Both will see an N that changes between the abstract and the method, because that jumps out while reading. From there they separate so completely that what one of them misses is close to a full catalog of what the other does well. Which is why "which of the two is better?" has no answer, and "what do I spend each one on?" does.

Why the numbers slip past an expert reader

It is not carelessness, it is how expert reading works. Someone who knows a field reads for meaning, and the digits turn into labels. Faced with F(2, 87) = 3.41, p = .038, what registers is "it came out, three groups, ninety people", and on to the next sentence. Checking that .038 really is what an F of 3.41 with 2 and 87 degrees of freedom gives you means stopping the reading, opening something else, coming back, and then doing it forty more times. Nobody reads like that, and anyone who tries stops understanding what they are reading.

The size of the hole has been measured. Nuijten, Hartgerink, van Assen, Epskamp and Wicherts ran statcheck over more than 250,000 p values from 30,717 articles in eight psychology journals published between 1985 and 2013: half of the papers using null hypothesis significance testing contained at least one p inconsistent with its own test statistic and degrees of freedom, and one in eight contained a gross inconsistency, the kind that may have changed the statistical conclusion (Behavior Research Methods, 2016, 48, 1205-1226).

Every one of those papers had a supervisor, co-authors, an editor and peer reviewers. All of them read it; almost nobody recomputed. Hence the uncomfortable consequence: the error most likely to reach print is not the deep conceptual one —that one has several expert readers hunting it at once— but the arithmetic one, which has none. It is exactly the kind of failure you catch with a mechanical pass before submitting, and no other way.

What the machine finds in a single pass

Part of this work is free, and that is worth saying first. statcheck is an R package released under a GPL-3 license and downloadable from CRAN, with a web version that accepts PDF, HTML or DOCX; it describes itself as a spellchecker for statistics, and it locates tests written in APA format —r, t, F, χ², Z and Q— in order to recompute the p. Its own documentation sets the limits: it usually cannot find statistics inside tables, a small deviation from the format (square brackets instead of parentheses, a semicolon instead of a comma) makes a test invisible to it, and in typical psychology journals it detects around 60% of the tests (checked in September 2026). If your results live in tables, which is where half of psychology lives, it will tell you very little.

What the Previum report adds is coverage, and the whole mechanism is visible in that example: the p values are recomputed from the test statistic and the degrees of freedom with code, not with a model; the effect sizes are checked against the F or the t you wrote yourself, which is where the d that does not come out of its own t shows up; and every reference is looked up one by one in eight databases (Crossref, OpenAlex, Semantic Scholar, PubMed, DOAJ, arXiv and OpenLibrary). The prose of the report is written by a model, but around numbers it cannot change.

References deserve their own line, because that is where your supervisor has less of a chance than anywhere else. Checking sixty references means sixty searches. Nobody does that while reading a draft: you recognize the surnames you know and take the rest on trust. And the broken ones are not visibly different —a changed volume number, a DOI that leads to another paper, a citation a language model composed for you in very neat handwriting— because they have exactly the same shape as the good ones. Checking them one by one is machine work in the literal sense: repetitive, dull, and with a binary criterion for being right.

None of those three checks has an opinion, and you will need that distinction the day you show this to somebody: what they flag comes out of code, and it comes out the same whoever reads it. The report also carries objections phrased the way a reviewer would phrase them, and those are a debatable point of view: read them the way you read an opinion. The three numbers underneath are not up for debate, and that is the difference worth defending.

What your supervisor sees and the machine never will

Whether the question is worth asking. No amount of internal consistency rescues a study that answers something nobody is asking any more. A machine can certify that the manuscript is consistent with itself; it does not know that the debate closed in your field four years ago, or that what you are presenting as novel is the variable everyone already puts in as a covariate. That is known by someone who has spent fifteen years reading that literature, and there is no way to deduce it from the file.

Whether the design answers the question. Whether the comparison group is the one you needed or the one you had, whether the measure you used is the one your field accepts or the one that came in the battery, whether the causal verb in your discussion survives a cross-sectional design, whether your sample supports the population you are generalizing to. None of this has a closed answer: it depends on the literature, on what you are going to claim, and on what you are prepared to hold under questioning. The machine audits the manuscript against itself; your supervisor audits it against the field.

And then there is what is not in the file. Your supervisor knows the project: which analysis you tried before this one, why fourteen participants dropped out, which variable disappeared between the pre-registration and the draft, what you promised the ethics committee. A machine only reads what you wrote, so a silence is invisible to it; to someone who was there, it sounds hollow. No tool is going to run that quality check, and it is the underlying reason supervision exists at all.

On top of that sits the political part, which is not a minor one: which journal to send it to, with what cover letter, in what authorship order, which editor rejects this design on sight and which one sends it out for review. That is not verification, it is the accumulated experience of surviving the process many times, and it weighs heavily in why Q1 journals reject you.

The blind spot they share

It is worth naming what escapes both of them, because nobody else will warn you. If the number you copied into the table is not the number your analysis produced, the report will find it perfectly consistent: it recomputes the p from the degrees of freedom you wrote, not from your data. Consistency is not truth. A perfectly coherent table can be a perfectly coherent transcription error.

Neither of them sees how many times you ran the analysis before this one, or that the file you analyzed was an old export, or that the final model was chosen by looking at the result. That only comes out with the data and the syntax open, which is a different job: the one a statistician does when you pay them to redo the analysis, not to read the manuscript.

And neither the machine nor your supervisor is the third reader who decides: that is peer review, and it comes later. What you are doing here is author self-assessment of your own manuscript before submission. The sentence sounds defensive and it is not: it is exactly the scope this has, and calling it anything else buys you a problem the day an editor asks.

Order matters: the machine first, your supervisor second

With the order reversed, here is what happens. Your supervisor sends the manuscript back with fifteen comments, you fix them, you call the text closed, and then the arithmetic pass finds seven p values that do not add up and three references that do not exist. You reopen a document that was already closed, and you go back to her with corrections to something she had already blessed. The good version of that scenario is that part of her attention went on digits; the bad one is that it did not, and the digits are still there.

In the other order it costs you a few minutes and an afternoon of fixes. You send her a manuscript where the p values match their degrees of freedom, the effect sizes come out of their own test statistics and all sixty references exist. Everything she marks from that point on is science, because there is nothing else left to mark. It is the cheapest way I know of making sure her hour goes entirely on the thing only she can do.

The same logic works sideways: the co-author who has sent you their section, the chapter your committee will read, the version you show the department methodologist before taking up their whole afternoon. And note what this does not do: it does not shorten how long your supervisor takes to reply. It changes what comes back.

How to say it without sounding like you outsourced your thesis

Say it, and say it precisely. "I ran the manuscript through an AI" invites the worst available interpretation, which is that a model wrote or judged your paper. "I recomputed every p value from the degrees of freedom and checked the references against the databases; these seven do not add up" is a different sentence, and it happens to be the true one. The first opens a discussion about AI; the second opens a discussion about your seven numbers, which is the one you wanted.

Bring the output as a list of discrepancies, not as a verdict. Each line is a question you have to settle yourself in your own software output: a p that does not match may be a typing slip, a pair of transposed degrees of freedom, or a value from a different model you ran that never made it into the table. The report says the two numbers do not agree; which of the two is wrong is for you to decide with the file open.

And if the report queries an analysis decision she made, take it as a question and not as a sentence. The machine does not know your field and she does: she may have a three-sentence answer that closes the matter, and she may not, and on that day you will have done your job. Either way the conversation goes better with the printout in front of you than with a hunch.

If your group has an explicit position against AI in manuscripts, the useful line is between generating text and checking numbers. Nothing comes back here for you to paste: what comes back is recomputed arithmetic and references checked against public databases, and the prose wrapped around them is a report for you, not a paragraph for your paper. If the worry is the data, the data processing agreement is published, the sub-processors are named and the manuscript is not stored.

Frequently asked questions

Do I have to tell my supervisor I ran the manuscript through a tool like this?

Yes, and it is in your own interest: if it surfaces later, it looks like you were hiding it. Describe it by what it does rather than by the label —recomputing p values from the degrees of freedom, checking effect sizes against the paper's own test statistics, verifying references in the databases— and show her the list of discrepancies. Most objections to "AI" collapse on their own once the sentence stops being abstract.

My group does not accept AI use in manuscripts. Does this fall under that?

It depends where you draw the line, and the reasonable line runs between generating text and checking numbers. Recomputing a p from its degrees of freedom is arithmetic with code; looking a reference up in Crossref is a search. The prose of the report is written by a model, but it is prose for you, not material that goes into the paper. Show the example report to whoever has to decide, and let them decide about the thing rather than about the word.

The report flags a p value my supervisor had already signed off. Who do I believe?

The software output, which is the only thing that settles it. In arithmetic there are no two opinions: either that .038 comes out of that F with those degrees of freedom or it does not. Open the output, look at the three numbers and fix whichever is wrong, which is often a mistyped digit. On design decisions the split runs the other way and there is no tie there either: your supervisor decides.

What if the report questions an analysis my supervisor chose?

Take it in as a question. "The report says that with nested data a one-way ANOVA ignores the classroom structure, can we talk about it?" is a sentence that can be answered; "the AI says your analysis is wrong" is a sentence that can only be won or lost. There are two legitimate endings: she explains why your field does it that way, or the two of you find a problem it was much better to find now.

My supervisor takes two months to reply. Does this replace her?

No, and no tool does: nobody else is going to judge whether your thesis question is worth asking and whether the design answers it. What you do get is that those two months are not spent on a round trip about typos. If what is missing is scientific criticism and it is not arriving, that gets fixed by finding more readers —a co-author, someone in the department, a conference— and, for psychology or health sciences analyses, with paid human help; but the decisions in your thesis stay on her ground.

Can I write in the manuscript that it has been reviewed?

Not in the sense an editor will read it. This is not peer review: it is author self-assessment of your own manuscript before submission, so it does not license writing that the paper "has been reviewed". The AI use declaration many journals now ask for is a different matter: there, describe what you did —automated consistency and reference checking on your own draft— and err on the short side.

Keep the split: arithmetic and references are machine work; the question, the design and the journal are your supervisor's work; and what is not in the file is invisible to both. Run the machine first, walk into the meeting with the list already resolved, and spend that hour on the one thing nobody else can do. If opening the list leaves you with an analysis decision you cannot defend, that is no longer arithmetic: it is what my statistical consulting is for.