When Reviewer 2 Asks for an Analysis You Don't Master: What to Do

The first time it happened to me I had a sample of one hundred people and a model I had been polishing for months. I had collected the data, run the analyses, written the manuscript myself, and submitted it to a Q1 journal with real hope. The editor had even written to me personally saying he was interested in the topic and looking forward to the manuscript. It came back with two reviewer rounds. In the second one, reviewer 2 asked for something seemingly simple: had we checked for outliers?

I had not. When I did, I found seven. I recomputed the model without them. The significant effects disappeared. What I had wasn't a paper, it was an illusion. I had to withdraw the manuscript, and it was never published in that journal.

That story is a few years old. I tell it because I haven't stopped thinking about it since, and because the outside version of my career: "he became obsessed with statistics, did a methodology master's, set up a consultancy": starts with that row of outliers I never looked at. That rejection pushed me deeper into methodology, made me focus my PhD on it, and led me to start helping researchers who showed up with the same scenario I had been in.

If you are reading this, something like that: not exactly that: is probably happening to you right now. Reviewer 2 has asked for an analysis you don't know how to run. The major revision deadline is approaching. Between your ego, your supervisor, and the doubt about whether the journal will give you another chance if you take longer, you don't know where to start. What follows isn't an abstract guide. It's the exact steps I apply today with researchers who write to me in panic, the reviewer letter open in one tab and the manuscript in another.

First, before anything else: the reviewer is sometimes right

There's a very human reflex when you read the reviewer's comment: thinking they're wrong, that they don't understand your design, that it's unfair. Sometimes that's true. But before defending yourself, ask yourself an honest question: is what they are asking reasonable?

In my case, it was. Checking outliers in a sample of one hundred wasn't a methodological whim: it was a basic step I had not taken. The reviewer wasn't asking me for anything exotic, he was pointing at a real hole in my reporting. Accepting that from the start completely changed how I drafted the response.

Before writing anything, spend twenty minutes reading the comment three times and asking yourself: if I were reviewing this manuscript, would this request seem reasonable? If yes, you are on the right track. If no, you'll have to defend: but only if you can do it with literature in hand.

Sort the request into one of three categories

Once you accept that the request exists and needs an answer, the next thing is to classify it. In my experience working with manuscripts in psychology, medicine, nursing and physiotherapy, almost everything a methodological reviewer asks falls into one of three categories. Knowing which one you are in determines what you do next.

The most expensive mistake is confusing the first two. There are researchers who try to run, in four weeks, a structural equation model with categorical indicators when they have never done a CFA in their life. The result is an analysis that the next reviewer or a methodologist on the thesis committee will spot at first glance. Better to deliver nothing than to deliver something done poorly.

Category 1: what you can learn in 48-72 hours

Many real day-to-day requests sit here: checking assumptions (normality, homoscedasticity, sphericity), computing and reporting effect sizes, adding 95% confidence intervals, running a non-parametric equivalent, doing a sensitivity analysis with and without outliers, justifying sample size with a post-hoc power analysis.

The outlier check from my anecdote was exactly this. The technical skill was googleable. What I lacked was the discipline to do it systematically, without anybody asking. If you are the one in this scenario, you have two things to do and one piece of advice:

What you have to do: run the additional analysis with the best reference at hand (a solid tutorial from a methodologist with authority, not the first Stack Overflow result), report results honestly even if they hurt you, and document every methodological decision so you can justify it in the response letter.

The piece of advice: before touching the main model, check all the basic assumptions at once. If you forgot outliers, you probably also forgot normality or linearity. The guide to verifying assumptions on the blog is the first place I send people in this scenario, and you can lean on tools like the normality test interpreter to cover the most basic block of checks without having to remember cut-offs by heart.

Category 2: when what they ask isn't realistic to learn now

This is where the plan changes. If the reviewer asks for a SEM with robust bootstrap and categorical indicators, a Cox model with competing risks, a multilevel model with three nested levels, or a network analysis with group comparison, you are not in a "weekend marathon" situation. You are in a situation where a poor analysis is worse than no analysis: the reviewer saw it once, they will see it again.

You have two honest paths. The first is to contact a methodologist in your department and offer co-authorship in exchange for the analysis and the writing of that section of the methods. It is the orthodox option and probably the best one if you know somebody and there is time. The second is to hire an external statistical consultant who runs the analysis and helps you draft the response: without co-authorship, in service format, declared in acknowledgements if your institution requires it. That's the scenario I see most in my own consulting work: a researcher with a good study who has hit an analysis outside their zone and needs fast, careful support.

What is not an option is to deliver something improvised. The journal will catch it in the second round and you will have spent the credit of the major revision on something that didn't convince.

Category 3: when the reviewer is wrong

It happens. A review published in BMJ years ago estimated that around half of statistical comments from topic-area reviewers contained some sort of error or overinterpretation. It's not that reviewers are clumsy: it's that most of them are experts in the topic, not in advanced methodology, and when the editor doesn't assign a methodological reviewer, this happens.

Patterns where the reviewer is wrong:

• They ask for a test that doesn't apply to your design. For example, a normality test on a sample of N>500, where any test will reject normality due to power and not substance.
• They ask you to increase N because "it's small," with no justification, when your N was justified a priori with a power analysis adequate to the expected effect size in your field.
• They ask for a more demanding alternative without reason. "Why didn't you do SEM?" when your hypothesis doesn't require latent variables and a Path Analysis with observed variables is defensible.
• They confuse two methods. They ask for "a Bayesian analysis" as if it were one thing, when whether a Bayes factor, a Bayesian regression, or an exploratory analysis with priors is appropriate depends on the context.

When you are here, defending is legitimate. The important thing is doing it respectfully and with literature in hand. Cite a methodological reference paper that backs your decision, explain why your approach is defensible for your specific design, and acknowledge the reviewer's point ("we understand the concern; however, in this design the standard practice is…"). Never with a confrontational tone.

How to draft the response: point-by-point pattern

Regardless of the category, the response letter structure is always the same. Number every reviewer comment, and for each one write four elements in this order:

1. Acknowledgement. Start by thanking the comment and briefly reformulating it so the reviewer knows you understood. One short sentence is enough.
2. What you did (or why you didn't). Describe the concrete action. If you ran a new analysis, say which one and with what software. If you decided not to, justify it here.
3. The result or the justification. Report the new numbers (or the bibliographic citation if you are defending). In the first case, do not hide unfavourable results: transparency is what keeps your credibility with the editor.
4. Where it is in the revised manuscript. Cite the page and line numbers where the change appears (e.g., "Section 2.3, page 7, lines 12-19"). It saves the reviewer the search and proves the change really exists.

For category 1, a typical response looks like: "We thank the reviewer for the observation. We have checked outliers using Mahalanobis distance (χ² threshold at p<.001) and identified 7 cases. We have reanalysed the model without these cases as a sensitivity analysis and report both results (Table 3). The main pattern of findings holds/does not hold. This section is now in the revised manuscript, page 9, lines 14-22."

For category 2: "We thank the reviewer for the suggestion. Given the complexity of the requested model and to ensure analytical rigor, we collaborated with [name/methodological co-author] who ran the SEM with robust bootstrap. Results are reported in Table 4 and Figure 2. The methods section incorporates a detailed justification of the approach (page 6, lines 5-21)."

For category 3: "We thank the reviewer for the observation and understand the concern. In designs like ours (cross-sectional, N=120, three groups), the standard practice in the literature is to prioritise [method X] over [method Y] for [specific reason], as argued by [author, year] and [author, year]. We have added a brief justification of this choice to section 2.4 (page 8, lines 9-16) so the reader has the methodological reasoning made explicit."

What I learned from the outlier that cost me a Q1

The first lesson seems the obvious one: check outliers. But that lesson is too small. The real lesson was a different one: methodological rigor isn't something you do because someone is going to catch you, it's something you do because your own work deserves it. When somebody writes thinking "if they don't ask, I don't report," they are building a fragile manuscript even if it ends up published.

The second lesson: and this one I only got years later: is that having a methodologist on call is cheap insurance. The difference between the first and second Q1 manuscripts you submit isn't luck: it's knowing whom to call when a comment lands that you don't know how to answer. My PhD students today call before submitting, not after receiving a major revision. The reviewer letter becomes much less terrifying when your analysis plan has already been read by another methodological pair of eyes before submission.

If you are in the middle of a major revision right now and the reviewer's request has blocked you, that's exactly the conversation I have every week through my reviewer 2 response service. Typical turnaround: 48-72 hours from receiving the manuscript and the letter. Better before withdrawing the manuscript than after.

And if you have not submitted yet, the other useful conversation: the one most avoided and most useful: is reviewing the analysis plan before submission. It is what separates the PhD students who get a desk reject from those who get a major revision: in many cases, the first group made in the plan exactly the mistake I made with my hundred participants and my seven outliers. That, today, is perfectly preventable. Read the most common mistakes I see before you submit, or register your plan on OSF before you start: both reduce the cost of learning what I learned late.