How to Check Your Reference List Before You Submit

Most of a manuscript cannot be checked quickly. An editor cannot rerun your models, cannot audit your coding scheme and cannot tell from the page whether your sample was recruited the way you say it was. The reference list is the exception. It is the one section that anybody can test in five minutes without knowing your field: pick an entry, paste the DOI, see whether it opens what it claims to open.

That asymmetry is the whole point. A bad reference does not invalidate an analysis, but it is cheap to catch and expensive to survive, and it arrives as a desk-rejection with two lines of explanation that tell you nothing — one more entry on the list of reasons a Q1 journal turns a manuscript away before it ever reaches review. Since drafts started being written with a language model open in the next window, this has stopped being an occasional slip and become a structural risk.

Why models produce references that do not exist

A language model does not look anything up when it writes you a citation. It assembles one. It has seen hundreds of thousands of APA references and knows exactly what one looks like: a surname, an initial, a year in brackets, a title with a colon and a methodological subtitle, the name of a real journal in the right subfield, a volume, a page range and a DOI carrying the right publisher prefix. Every component is plausible because every component is real. The combination is what does not exist.

This is the awkward part: a fabricated reference does not look like an error. A badly formatted citation announces itself. A fabricated one is immaculate. Bhattacharyya et al. (2023) measured this in medical content generated with ChatGPT and found high rates of fabricated or inaccurate references precisely in the material that read best.

And it is not confined to the drafting stage. Reports generated with AI have been seen to ship bibliographies containing dozens of fabricated entries: asked to transcribe the reference list of a manuscript that had no reference list, the model reconstructed one from memory and handed over an entire bibliography nobody had ever written. The practical rule is the same whether you drafted the text yourself or a tool helped: a reference is only checked once an external database has recognized it. Asking a model whether its own reference exists checks nothing at all.

The failure nobody looks for: a DOI that resolves to something else

Start here, because it is the one that survives every other filter. A broken DOI is caught by any link checker and by you, the first time you click it. A DOI that resolves passes the spell-check, passes the submission system, passes your co-authors, and is only caught by a human being opening it and reading what appears.

No bad faith is required to produce one. The reference manager imported the wrong record. You reordered the bibliography and a DOI slipped a row. You copied from a PDF and the identifier from the entry above came with it. The outcome is always the same: a reviewer clicks, and lands on an unrelated paper.

How to check it. Pasting the DOI into doi.org and seeing a page load is not the check. The check is reading that page and confirming its title and authors match your entry. If you would rather stay in a terminal, ask Crossref for the record:

Two cautions, so that you do not convict yourself unfairly. First, databases sometimes hold a truncated title: there are articles registered in Crossref with only the part before the colon, so a poor title match does not prove the DOI is wrong. Check the authorship before you change anything. Second, arXiv DOIs (the ones beginning 10.48550) are registered with DataCite rather than Crossref, so Crossref and OpenAlex will both return a perfectly legitimate 404 for a perfectly legitimate reference. That case is settled on arXiv itself.

The reference that does not exist at all

Plausible authors, a real journal, a coherent year, and an article that was never published. If the editor catches this one, the conversation about the rest of your manuscript is over before it starts.

How to check it. Search the full title, in quotation marks, across these catalogs. All of them have public APIs and none of them asks for a key:

• Crossref (api.crossref.org) — the DOI registry behind most publishers, and the first stop for any journal article.
• OpenAlex (api.openalex.org) — a very large open catalog, successor to Microsoft Academic Graph, good at what Crossref misses.
• PubMed (eutils.ncbi.nlm.nih.gov) — biomedicine and health only, but authoritative there. Appending [Title] to the search term removes most false positives.
• Semantic Scholar (api.semanticscholar.org) — its match endpoint exists for exactly this: pairing one reference title against a corpus rather than returning a list of vaguely similar work.
• DOAJ (doaj.org) — the open access journal directory, and the piece that rescues Spanish-language and Latin American bibliographies. Journals such as Psicothema, Anales de Psicología or Liberabit sit there and often nowhere else.
• OpenLibrary (openlibrary.org) — the five above are article databases. Books, manuals and reference works (Cohen, Nunnally, the APA manual) appear in none of them, and they are half the bibliography of a thesis.
• arXiv (export.arxiv.org) — preprints in physics, mathematics, computing, statistics and economics.

Two habits that save you false alarms. If the title carries a subtitle after a colon, search the part before it first; catalogs do not store subtitles consistently. And if it is a book with an edition in brackets, strip the brackets: searching OpenLibrary for "Publication manual of the American Psychological Association (7th ed.)" returns nothing at all, whereas the same title without "(7th ed.)" returns it immediately.

Retracted work that is still being cited

A retraction does not remove the article. It leaves it published with a notice attached. The PDF you downloaded three years ago and kept in a folder does not carry that notice, and your reference manager will not volunteer the information unless you have configured it to. So retracted work gets cited in good faith, years later, sometimes as supporting evidence.

It lands badly in review because the editor cannot distinguish carelessness from oversight. Either way it means you never went back to the source.

How to check it. Three routes, in descending order of effort:

• OpenAlex carries an is_retracted field on every record, so asking for the work by DOI answers the question in one line. Trust the field rather than the title: Wakefield et al. (1998) is stored with "RETRACTED:" prefixed to the title, but LaCour and Green (2014) in Science, retracted in 2015, keeps a clean title and is given away only by the flag.
• The Retraction Watch Database (retractiondatabase.org) is the specialist source, with tens of thousands of records and the stated reason for each retraction. Since Crossref acquired it, the full dump is downloadable free as a CSV from api.labs.crossref.org/data/retractionwatch, which is what you want if you intend to screen a whole bibliography in one pass.
• PubMed tags these with the publication type "Retracted Publication" and links the corresponding retraction notice. For biomedicine it is the fastest check available.

If you use Zotero, switch on the warning: it flags retracted items in your library using Retraction Watch data, which is as close as this gets to checking itself.

One nuance worth holding on to: citing retracted work is not automatically wrong. If your paper is about that case, or discusses the literature built on top of it, the citation is legitimate. What cannot happen is for it to appear unmarked, in the middle of a list of studies supporting your hypothesis. The COPE retraction guidelines (Barbour et al., 2019) set out why the notice has to travel with the work.

A year or a volume that has drifted

One digit. The work exists, the authors are right, the title is right, and the year says 2019 for an article that appeared in 2018, or the volume reads 34 instead of 43. Nobody is rejecting a manuscript over this, but it reaches the proofs, and there it costs a round of correspondence.

The usual cause is two legitimate dates colliding: online-first publication and the print issue, which can be more than a year apart. When the reference manager imports one and the running text uses the other, the in-text citation and the list entry stop matching, which is the visible half of the problem.

How to check it. The Crossref record separates published-online from published-print. APA 7 asks for the year of the version you are citing, so if you cite the version of record — the final one, with volume and pages — that is the issue year. Note the trap: confirming a reference by consensus across databases does not validate the year, because the matching is done on title and authorship. The year is on you.

Authors missing, or in the wrong order

APA 7 asks for up to twenty authors in the reference list, and from twenty-one onwards the first nineteen, an ellipsis and the final author. The "et al." cut belongs to the in-text citation, not to the list. What turns up regularly is a list truncated to three or six names because that is how another journal's style left it.

Order is the more serious version. Authorship order is scientific information, not formatting: it says who led the work. Inverting it, or slipping in a co-author who did not sign, is the kind of thing a reviewer who knows that group notices instantly. It happens more often than you would expect when a reference is typed up from a database record instead of exported.

How to check it. Pull the record from Crossref or OpenAlex and compare the full list, in order. Automated consensus matching is deliberately tolerant about authorship — a shared surname is enough, because lists get truncated and abbreviated differently across catalogs — so it is good for ruling out that the record found belongs to a different paper, not for auditing who signed. That part stays manual.

Citations in the text that are not in the list, and the reverse

By far the commonest failure, and the only one that needs no internet connection. You cite Fernández and Ortiz (2021) in the discussion and the list holds only Fernández (2019). Or you cut a paragraph in the final pass and its reference stayed behind, orphaned. APA 7 is explicit about it: every in-text citation needs an entry, and every entry needs to be cited at least once.

How to check it. If you work in Word with a reference manager (Zotero, Mendeley, EndNote), the list is generated from the citations and the problem disappears by construction — the price being that citations must be inserted as fields rather than typed. If you maintain the bibliography by hand, the check is mechanical: pull the surnames and years out of the running text, sort them, and compare against the list. In LaTeX, BibTeX raises the warning for you.

One place worth a second look: citations inside tables and figure captions. Many managers do not index those, so they survive the clean-up and then have no entry waiting for them.

Preprints wearing a journal's clothes

You cite something from PsyArXiv or bioRxiv with a journal name and a volume that do not exist yet. Sometimes it is optimism — the paper was under review when you read it. Sometimes the citation arrived pre-assembled that way. And sometimes a model filled the gap with the most probable journal.

The problem underneath is not formatting. A preprint has not been through peer review, and presenting it as though it had changes the weight a reader gives your argument. A reviewer who knows that preprint will say so.

How to check it. Look the DOI up in Crossref or OpenAlex: if the work really was published, the record carries the journal name and volume, and arXiv itself often stores the DOI of the published version once the author declares it. If it appears nowhere but the repository, cite it as a preprint — in APA 7 the repository takes the journal's place, with no volume and no pages. It is also worth revisiting preprints you cited a year ago: they are frequently published by now, and the version of record may have changed the very analyses you are citing.

Why a resolving DOI is not proof, and two sources are

This is the fine point, and it separates a real check from one that only feels reassuring.

A DOI resolving means one thing: somebody registered that identifier and put a URL behind it. It says nothing about whether it identifies the work you are citing. In the other direction, a database failing to find a reference says nothing either. PubMed covers biomedicine only, DOAJ covers open access only, and none of the article databases index books, chapters, theses, technical reports or conference proceedings. A flawless reference can come back empty from three catalogs in a row with nothing wrong with it.

Hence the rule that makes any of this usable: a reference counts as verified when at least two independent sources agree on the title and on the authorship. Two rather than one, because a single catalog can return a different paper with a near-identical title — search for a title as generic as "Social media addiction and personality in adolescents" and you will find several, by unrelated groups. And title and authorship, not title alone, for the same reason.

The corollary is the one that matters most, and the hardest to hold on to when you are anxious the week before submission: "not found" does not mean "fabricated". It is the normal outcome for books, chapters, theses, reports and local or non-English journals. Confusing the two and rewriting a sound bibliography out of fear does more damage than the failure you were trying to avoid. There is exactly one case where silence is informative: the reference declares a DOI and no database recognizes that DOI. That deserves a careful manual look, because an identifier nobody registered does not arise from carelessness.

The whole procedure, in twenty minutes

The order is about economy. Start with the entries that carry a DOI: they are the fast ones, and they are where the serious failure lives. Move on to the ones without, which have to be searched by title. Leave books and chapters until last, and do not panic about them. Then save the final five minutes for the text-against-list comparison, which will almost certainly turn something up.

If the manuscript came out of your thesis, this is the moment to do it properly. Turning a thesis into an article means the bibliography gets cut and reordered several times, and reordering is exactly when identifiers drift. The same goes for most of the problems in the errors that recur most in psychology manuscripts: they nearly all appear in that same phase.

Frequently asked questions

How do I know whether a reference is fabricated?

Search the full title in at least two independent catalogs (Crossref, OpenAlex, PubMed, Semantic Scholar, DOAJ, OpenLibrary, Google Books or arXiv) and confirm they agree on title and authorship. If two agree, the work exists. If none of them finds it and the reference also declares a DOI that no database recognizes, that is a serious signal and needs a manual look. If there is no DOI and the entry is a book, a chapter, a thesis or an article in a local journal, the likeliest explanation is simply that it is not indexed.

Is it enough that the DOI opens a page?

No. A DOI resolving only proves the identifier is registered, not that it belongs to the work you cite. The hardest failure to detect is exactly that: a legitimate DOI attached to the wrong article after a bibliography was reordered. Open the link and confirm the title and authors on the page are the ones in your entry.

How do I check whether something I cite has been retracted?

Three ways. Search the Retraction Watch Database at retractiondatabase.org, or download the full CSV dump from api.labs.crossref.org/data/retractionwatch if you want to screen a whole bibliography at once. Ask OpenAlex for the work by DOI and read the is_retracted field. In biomedicine, check whether PubMed tags it with the publication type "Retracted Publication". Zotero will also flag retracted items in your library if you have the warning enabled.

A reference appears in no database. Should I drop it?

Not if you know it exists. Bibliographic databases cover books, chapters, theses, technical reports, conference proceedings and local or non-English journals poorly. Absence is not evidence of anything on its own. What is worth doing is keeping the PDF or the physical source to hand, in case the editor asks.

Can I cite a preprint?

Yes, as long as you cite it as one. In APA 7 the repository takes the place of the journal, with no volume and no pages. What you cannot do is give a journal name and issue number to something that has not been through peer review. And before you submit, search again for the preprints you cited some time ago: if they have since been published, the version of record may have changed the analyses you are relying on.

Does my reference manager not handle all this?

Only partly. Zotero, Mendeley and EndNote keep the in-text citations and the list consistent and apply the style correctly, which is a great deal. They do not verify that the record you imported belongs to the work you cite, they do not detect a crossed DOI, and they do not check the year against the version of record. Whatever entered your library wrong comes out beautifully formatted and still wrong.

How many bad references does it take to sink a submission?

One is enough to change the conversation. Not because a single citation matters that much, but because of what it implies: if an editor checks a reference and it does not exist, they stop trusting the things they cannot check, which is nearly everything else. That is what makes this failure so expensive relative to how little it costs to prevent.

References

American Psychological Association. (2020). Publication manual of the American Psychological Association (7th ed.). https://doi.org/10.1037/0000165-000

Barbour, V., Kleinert, S., Wager, E., & Yentis, S. (2019). Guidelines for retracting articles. Committee on Publication Ethics. https://doi.org/10.24318/cope.2019.1.4

Bhattacharyya, M., Miller, V. M., Bhattacharyya, D., & Miller, L. E. (2023). High rates of fabricated and inaccurate references in ChatGPT-generated medical content. Cureus, 15(5), e39238. https://doi.org/10.7759/cureus.39238

Before you submit: the bibliography is the last thing you revise and the first thing anyone checks, and by then you have been inside the manuscript for so long that you no longer see what an outside reader sees on the first pass. If yours is already written, run it through Previum: a free Reviewer 2 style pre-review that tells you what a reviewer would object to before the journal does, and whose Professional report also checks the reference list against the databases described here, holding to the same two-source rule. Cheaper to hear it now than in the editor's letter three months from now.