Can I trust ChatGPT with my statistics?

You have been using it for months and you still cannot decide what you think. It explained an entire concept in ten minutes that a whole textbook chapter had failed to make clear, and last week it handed you a reference that does not exist. Half your department uses it daily and says nothing; the other half tells you it is cheating, having never opened it. The public conversation is no better: either it is the future of science or it makes everything up. Neither sentence helps you decide whether the analysis sitting open on your screen right now can go through it. The useful question is narrower, and it does have an answer: which parts of your work this tool can do, which it cannot, and why the line falls where it falls.

Where it genuinely helps, and that is not a small list

Explaining a concept comes first, and it is where most people underrate it. You can ask for the same explanation again, and again, with your own variables and at your own level, without the embarrassment of asking a supervisor the same thing for the fourth time. A supervisor does not have that patience three Tuesdays in a row, and a textbook does not rephrase: it repeats the paragraph you already failed to follow. For getting from I do not know what a random effect is to I think I see it now, nothing is faster. You verify what you learned later, once you have something to verify it against.

Writing code comes second, and here it is genuinely good. Turning anxiety score as the outcome, session as a fixed effect and patient as a random effect into lmer(anxiety ~ session + (1 | patient)) takes it a second, and decoding the R error message that has had you stuck since yesterday takes another. What it saves you is the syntax, not the decision: you choose the model, you run the code, you read the output.

Drafting comes third, and it is the tool's home ground. Turning four bullet points into a methods paragraph, cutting two hundred words from a discussion without losing the argument, tightening the English of a paper written by a non-native speaker at two in the morning. It also works as a sparring partner: asking what a reviewer would object to in a paragraph gives you a list that will not be right about everything, but that puts on the table the two obvious objections you have failed to see for three weeks.

Notice what those three have in common: in all three you can check the answer immediately. You know you understood the concept when you can explain it to someone else. You know the code is right when it runs and the output makes sense. You know the paragraph works when you read it. That is the pattern — and it also predicts the failures, because the tool falls down exactly where you would have to trust it without being able to check.

Generating is not calculating

Almost everything follows from this. You type: I have t = 2.31 with 58 degrees of freedom, what is p? and it answers with a number. That number does not come from evaluating the t distribution on 58 degrees of freedom. It comes from having read millions of sentences of that shape and producing the most likely continuation. The correct two-tailed answer is p = .024. A model can write .024 and it can write .03, and both cost it exactly the same, because in both cases it is doing the only thing it does, which is choosing the next piece of text.

This is worse than an obvious error. A wrong but plausible p value survives every human reading: nobody redoes a colleague's arithmetic, and you least of all on an analysis you already closed. Where it does fall over is later, at the worst possible moment: a growing number of psychology journals run submissions through an automated checker that recomputes p from the test statistic and its degrees of freedom. Inconsistency between those three numbers is one of the most common errors in the published literature: in the most cited audit of this, covering more than 250,000 p values from eight psychology journals between 1985 and 2013, half of the papers reporting significance tests contained at least one inconsistency, and roughly one in eight had one large enough to have changed the statistical conclusion (Nuijten et al., 2016). That was before there was a chat window in the room.

The honest qualification matters here. When the chat does not merely write code but actually runs it, it is calculating. The number comes out of a library that evaluates the distribution properly, exactly as R or jamovi would. The risk moves, it does not vanish: the question becomes whether the code did what you think it did — whether it dropped rows with missing values, whether it picked the variable you meant, whether the test was one- or two-tailed — and that is something you can check by reading it. Telling those two situations apart is also what separates a chat from a system that verifies, which is a different category of tool: what an AI paper reviewer actually is.

Hence the single most useful rule in this article: no number that ends up in your manuscript may come out of a conversation. It comes out of software you ran, or out of a calculator that calculates. If you want a value from a chat, ask for it with the code beside it and run the code.

It does not remember references, it reconstructs them

Same mechanism, different face. It has seen millions of references and learned their shape: surname, year, journal, volume, pages, DOI. When you ask for one that supports your sentence, it produces something with that shape and with the content your sentence needs. Sometimes it exists. Sometimes it does not. Internally there is no difference between the two.

The failure is not random, which is precisely why it fools people: what comes out is the most plausible reference that could exist. A surname that really does publish on the topic, a journal that really does publish that kind of work, a year consistent with the literature, and a correctly formatted DOI that resolves to nothing. A fabricated citation with absurd authors is visible on the page. This one is invisible to you and to your reviewer.

The mixed case is more uncomfortable still. A paper that does exist, cited with the wrong year, in the wrong journal, or credited to a co-author who was not on it. It survives the casual check — the name is there, the topic fits — and fails the real one, which is what happens when the journal's submission system tries to resolve your DOIs.

The fix is not to stop using it, it is to change the request. Do not ask for five references that support your paragraph; ask for five search terms and go to the database yourself. Before submission, every DOI has to resolve and every entry has to appear where it claims to be: checking your references before you submit is a half-hour procedure that spares you the single most humiliating email an editor sends.

Choosing the test without seeing the data

This is the quiet failure, because nothing about it looks wrong. You describe your study in three lines and get back a test with a well-written justification. It is not a random answer: it is the modal answer to that description, the one any textbook would give to that sentence. The trouble is that your study is not the sentence.

I want to compare two groups produces a t test. But if the two groups are the same patients measured twice, it is a paired t test; if every participant contributed four sessions, it is not a t test at all, it is a mixed model; if the outcome is a count with 60% zeros, it is neither. The chat cannot see any of that. It sees your description, and the description is exactly the part you have not mastered. That is why you are asking. And uploading the file does not fix it either: nothing in the matrix says that rows 1 and 2 are the same person measured twice, or that sampling ran by classroom.

There is a cruel asymmetry on top. The stranger your design, the harder the answer pulls toward the ordinary one, because the model leans toward whatever is most frequent in the literature, and your awkward case is precisely what departs from it. It is right when you did not need it and wrong when you did.

The useful version of that conversation runs the other way. Instead of which test should I use, ask what do you need to know about my data to decide the test, and then answer it: how many groups, whether they are the same people, how many measurements per person, what kind of outcome variable, what happened to the missing values. That list, once answered, almost decides itself, and the full decision tree is in how to choose a statistical test.

How to ask so that it is actually useful

Ask for the procedure, not the result. Give me the R code for this instead of give me the result of this. You run the code, and the number comes from a library. It is a five-word difference in your prompt, and it completely changes where the value in your table came from.

Ask for the explanation, and ask for the counter-argument. Make it explain the code line by line and state what it is assuming; if you cannot follow the explanation, do not run that code on your data. And rather than asking it to validate your analysis, ask it to argue why your analysis does not answer your question. That yields three times as much, because the other prompt is too easy: a model trained to be helpful will agree with you with remarkable elegance.

Leave the formatting to something deterministic. Putting M, SD, t, degrees of freedom and p into APA 7 is not a matter of judgment, it is four rules applied without a single slip: italics for Latin symbols, roman for Greek, two decimals, a leading zero only where the value can exceed 1. A chat gets it right almost every time, and almost every time across a hundred results means several errors scattered through your section. The APA 7 results formatter applies the same rule a hundred times out of a hundred.

If what you want is the full map of where this fits inside a thesis and where it does not, chapter by chapter, that is in using ChatGPT and AI for your thesis statistics. What is left to say here is not about what the tool can do, but about which documents should never go into it.

Four cases where uploading the document really is a problem

Your own manuscript, with your own data and your own words, pasted into a chat so it can tighten a paragraph, is not a scandal in most settings. Two things are worth checking and then you can move on: whether your account is used to train models, and whether your institution has a licensed instance of its own, which is increasingly common. The four cases below are different, and what makes them different is not technical. In all four, the material is not yours to share.

Someone else's manuscript that you are peer reviewing. Review is confidential: the authors handed unpublished work to the journal, not to you plus a third party of your choosing. Several publishers now say so explicitly in their reviewer policies, and the NIH has banned it outright for grant review. If the policy does not persuade you, the plain version will: an unpublished paper on a topic close to your own, passing through a system you do not control. You can still use a chat to tighten the writing of your report once you have written it, without pasting in the paper you are judging.

An invention you have not yet filed. Novelty is a patentability requirement, and disclosure before filing can destroy it. Whether a chat conversation counts as public disclosure is arguable, and that is exactly the problem: the argument will be settled years after your window has closed. It also depends on where you file — the UK and Europe apply absolute novelty, while the US and Australia offer a twelve-month grace period, which is a safety net and not a plan. An algorithm, a device, a protocol you intend to protect, a measurement method with a commercial route: that part waits for the filing. The rest of the paper can go in.

The other two are data, and you are holding them for someone

Participant-level personal data. Not the aggregated results — the file. Dates, diagnoses, free text, interview transcripts, or a combination of variables narrow enough to identify someone in a small sample: in a service with eleven patients, age plus sex plus discharge date is already a name. Your ethics approval — IRB, HREC or your university research ethics committee — came with a data management plan naming who would see those data and where they would live, and a chat window was not on that list. The responsibility for that sits with the researcher, not with the tool.

Data held under a confidentiality agreement. A data sharing agreement with a hospital or an NHS trust, a registry that released the file with conditions of use, an industry-funded trial, a consortium clause. Here it does not matter whether the data are de-identified and it does not matter what you personally think of the real risk: what you signed describes where those data may go, and that sentence does not accept creative interpretation at eleven at night before a deadline.

Notice that none of the four cases says the tool is dangerous. They say you are holding that material for someone else: for the journal, for your future patent, for your participants, for whoever released the file. The same rule would stop you forwarding it by email to a colleague for a quick look, and that comparison is the best test I know. If you would not forward it, do not paste it.

Frequently asked questions

Can I use ChatGPT to do my thesis analysis?

For writing the code and understanding what each line does, yes, and it will save you weeks. For producing the results, no: the numbers have to come from running that code on your data in R, jamovi, JASP or SPSS. The line is that a chat generates text while a statistics package evaluates a distribution. If your supervisor asks where a value came from and the honest answer is the chat told me, you do not have that value yet.

Does ChatGPT calculate p values correctly?

It does not calculate them, it writes them. Given t = 2.31 with 58 degrees of freedom, the correct two-tailed answer is p = .024, and a model can produce that number or something near it with nothing in its workings distinguishing exact from approximate. The exception is when it genuinely runs code: then the number comes from a library and is correct, provided the code did what you think. Before copying anything, check whether code was actually executed in that conversation or merely discussed.

Does ChatGPT make up references?

It reconstructs them, which is more misleading than inventing them outright. It produces a reference with the right shape and the content your sentence needs: a plausible author, a real journal, a consistent year, a well-formed DOI. Some of what it produces exists and some does not, and internally there is no difference. No reference enters your bibliography until you have resolved its DOI or seen it on the journal's own site.

Is it safe to upload my manuscript to ChatGPT?

Your own manuscript, with your own data and your own text, is not a problem in most settings; check whether your account is used for training and whether your institution has a licensed instance. There are four cases where it is: a manuscript you are peer reviewing, an invention you have not yet filed, files containing participant-level personal data, and data received under a confidentiality agreement. In all four the issue is not the technology, it is that the material is not yours.

Can I use ChatGPT to help me peer review a paper?

Not with the manuscript inside it. Peer review is confidential: it is unpublished work the authors gave to the journal. Several publishers prohibit it explicitly in their reviewer policies and the NIH has banned it for grant review. What you can do is use a chat to tighten the wording of your report after you have written it yourself, without pasting in text from the paper you are assessing.

Do I have to declare that I used AI in my paper?

At almost every journal, yes, and an AI cannot be listed as an author because it cannot take responsibility for the work or sign the conflict-of-interest statement (the ICMJE criterion). The usual form is one sentence in the methods or the acknowledgments saying which tool you used and for what: drafting, translation, code. Check your target journal's policy before submitting, because they range from declare it to generated text is not accepted. What never appears in any declaration is a result that came out of a conversation, because that is simply not done.

Keep the boundary: ask for the procedure and not the number, the search term and not the reference, the questions that decide a test and not the test. And keep the four closed doors: another author's manuscript, the unfiled invention, your participants' data, and whatever you signed with the people who released a dataset to you. If what you have in front of you is a paper about to go out and what you want is someone to read it with a reviewer's eye, that is what Previum does: it recomputes what can be recomputed and tells you, before the editor does, where they are going to come at you.