ReviewBench (200 eLife articles, their judge and their metrics) and the Walters & Wilder reference benchmark (636 citations, by type): how it is measured, what comes out and what cannot be concluded.
Two public benchmarks, their code, their judge and their metrics. Every step, what comes out, and what cannot be concluded.
ReviewBench is the benchmark Reviewer3 publishes (open source, MIT license, public database): 1,000 eLife articles with their human reviews and the comments of three tools —Reviewer3, GPT-5.2 and Gemini 3 Pro— on each. The central claims of each article are extracted, and a judge (a model, Gemini 3 Pro, with three prompts) decides, comment by comment, which claim it points to, with which stance, whether it says why it matters, whether it proposes a fix and whether it is anchored in the text. The metrics come from comparing all of that with what the humans said.
We did the same with 200 of those 1,000 articles, drawn at random with a fixed seed. Our engine —Previum’s prompt, in exhaustive mode: every critique, no cap— writes the review in English on the whole text with grok-4.6. Each critique enters their database as a comment with the same shape as Reviewer3’s: the problem, why it matters, the fix and the quote. Then their judge runs with its three prompts, untouched, and their metrics with their code. The other columns are their database: recomputed with that same code, their published figures come out to the decimal.
One caveat that cannot be hidden: they judged with gemini-3-pro-preview and Google has retired it, so ours is judged by gemini-3.1-pro-preview. To measure that drift we re-judged Reviewer3 and the humans on 100 articles with the new judge: Reviewer3’s stance-matched overlap goes from 65.4 % to 64.3 % and its consequential rate from 89.6 % to 87 %; no metric moves more than three points. Details in the JSON.
| Metric (mean per article) | Human reviewers | Reviewer3 | Gemini 3 Pro | GPT-5.2 | Reviewer (us) |
|---|---|---|---|---|---|
| Comments per article | 10.6 | 16.1 | 6.5 | 24.1 | 16.3 |
| Points to a claim | 63 % | 77.1 % | 77.1 % | 67.4 % | 73.1 % |
| Could overturn it | 60.8 % | 89.7 % | 68.7 % | 78.3 % | 97.2 % |
| Claims of the article addressed | 60.7 % | 88.3 % | 72.3 % | 95.7 % | 89 % |
| Addresses what humans address | — | 95.6 % | 81.1 % | 97.6 % | 93.4 % |
| Agrees with humans, same stance | — | 70.4 % | 53.2 % | 70.5 % | 67.9 % |
| Says why it matters | 63.3 % | 97.4 % | 88.9 % | 82.8 % | 100 % |
| Proposes a fix | 60.6 % | 99.9 % | 89.2 % | 98.6 % | 100 % |
| Anchored in the text | 48.2 % | 83 % | 85.7 % | 70.9 % | 98.1 % |
200 articles, paired: every source is compared on the same articles. The 95 % intervals (bootstrap over per-article means), the n per metric and the paired tests against each source (Wilcoxon signed-rank, Holm-corrected) are in the JSON.
Walters & Wilder (2023, Scientific Reports) asked GPT-3.5 and GPT-4 for 84 academic texts and checked their 636 citations by hand: 195 are fabricated and 441 exist. It is the benchmark Reviewer3 uses for its reference checker, on a subset of 476 citations it does not publish. We ran all 636 from the original file, books included.
The checker is the one that runs in Previum: it transcribes the citation (it does not recall it), asks eight databases —Crossref, DOAJ, OpenAlex, PubMed, Semantic Scholar, OpenLibrary, Google Books and arXiv— and only calls a citation real when two independent databases agree on title and authorship. What it cannot find it marks “no trace”, never “fabricated”: in the benchmark, “no trace” counts as a prediction of fabricated.
Three passes were run and all three are in the JSON: the first with eight citations in parallel (OpenLibrary answered no book at all, rate-limited), the second slowly and without authors (recall went up and it accepted fabricated books carrying a real book’s title), and the third, the one published, with the authors taken from the citation itself.
| Type | Citations | Fabricated | Fabricated caught | Real recognized | Accuracy |
|---|---|---|---|---|---|
| Articles | 383 | 140 | 93.6 % | 94.7 % | 94.3 % |
| Books | 204 | 28 | 89.3 % | 79.5 % | 80.9 % |
| Chapters | 33 | 23 | 91.3 % | 80 % | 87.9 % |
| Web and other | 16 | 4 | 75 % | 25 % | 37.5 % |
| All | 636 | 195 | 92.3 % | 86.4 % | 88.2 % |
Reviewer3, on 476 citations it does not publish: 97.2 % of fabricated caught, 99.3 % of real recognized, 98.5 % accuracy.
How to read it: on articles we are close to what Reviewer3 publishes; on books we are not, and the reason is located: bibliographic databases cover books poorly and a fabricated book usually reuses a real book’s title. It is the pending improvement of the checker; when it is done, the benchmark is re-run and this table changes.