When Your Committee Attacks Your Analysis: What to Say

You are forty minutes in, the results slides are behind you, and someone looks up and says that analysis was not the right one for your data. Your mouth goes dry, and in the two seconds of silence the two bad exits appear: defend it as though there were nothing to discuss, or fold and agree with everything. Neither is what the room expects from you, and there is a third route you can write down and rehearse before you walk in. If you do not yet know where they are going to come at you, start with the twelve questions almost every committee asks.

What the room is actually testing when it attacks an analysis

Your examiners have read the thesis. The decision they are pointing at was not discovered at minute forty: it has been flagged in their copy for days, and they waited for the session to see what you would do with it. That changes the question in front of you. It is not «is this fatal?», it is «does this candidate know why they did it and what it cost them?». A doctorate does not certify that you took the best available turn at every fork in the analysis. It certifies that you know which turns you took, on what criterion, and how far your conclusions travel as a result.

Objections come in three kinds, and they do not get the same answer. The factual error: degrees of freedom that do not add up, a test read backwards, a table that contradicts the text. They are right, and you concede it whole and fast, with no ornamentation. The defensible choice: you went one way and the person asking would have gone another. This is by far the most common, and it is settled by showing the criterion, not by agreeing or disagreeing. The request for something you did not do: an analysis that is not in the thesis and that they now want to see. That is reanalysis or future work, and it is answered as such.

Almost all the damage in a defense comes from confusing the three. Conceding a perfectly defensible choice as though it were an error leaves you with no thesis halfway through the session, and forces the panel to decide what to do with a piece of work its own author has just disowned. Arguing about wrong degrees of freedom as though they were a matter of taste costs you credibility for everything that follows, including the parts you can actually defend. Classify first: are you telling me I got it wrong, that you would have done it differently, or that you want to see more?

One thing worth holding on to while your voice shakes: an outright fail is rare, and a debatable analytic choice almost never causes one. What does happen is that a badly handled answer turns into major corrections, a resubmission, a lower outcome or an awkward paragraph in the report. The gap between those outcomes is decided in the sixty seconds after the question.

Concede, explain, offer: the three beats of an answer

Before the three beats there is a pause almost nobody takes. Drink some water, then put the question back in your own words: «If I have understood you, you are asking why I ran a repeated-measures ANOVA rather than a mixed model». It buys you four seconds of thinking that do not look like a freeze, and it prevents the most common disaster of all, which is answering brilliantly a question nobody asked.

Concede. One sentence, no preamble, no «well, actually»: «You are right that complete-case analysis requires a strong assumption about the missingness». That concession is what buys credit for the two beats that follow, because someone who has just named a limit out loud has far more standing to explain the rest than someone who has spent three minutes haggling. What you do not do is concede more than was asked: admit the specific limit, not the whole chapter.

Explain. The second beat is not an apology, it is a reconstruction of the decision: what you knew, what you had in front of you, and what rule you applied, in the past tense and with its location in the document. «When I made that call I had this, I looked at that, and I applied this criterion, which is on page 112.» If, while rehearsing, you find you cannot name the criterion, do not improvise one in the room. That is not a defense problem, it is a decision taken without a criterion, and you still have days to find one or to write it up as a limitation.

Offer. The third beat closes with something concrete: the alternative analysis by name, its estimation method, and the criterion that will tell you whether the conclusion moves. If you have already run it, say the number. Never promise what you cannot deliver in the final version, because examiners remember exactly what you promised and they write it into the outcome.

Script 1: «that analysis was not appropriate for your data»

The scene: a three-wave intervention study, 112 participants at baseline, 16% missing at follow-up, so the mixed ANOVA was run on the 94 complete cases and returned a group by time interaction, F(2, 184) = 5.12, p = .007, partial eta² = .053. The internal examiner tells you that listwise deletion throws away eighteen people and assumes the data are missing completely at random, and that this is what mixed models are for.

The full script: «You are right that complete-case analysis requires the missingness to be completely at random, and that is a strong assumption. When I made that decision I had two things: Little's test was not significant, chi²(21) = 24.6, p = .265, which does not prove the missingness is completely at random but was what I had, and those who dropped out did not differ from completers on any baseline variable, which is the comparison on page 112. On that basis I took complete cases as the primary analysis and declared it as such. That said, the mixed model is the more defensible option, because it uses all 112 cases and only requires missingness to be random conditional on what was observed. I have since fitted it with restricted maximum likelihood and the interaction holds, F(2, 201.7) = 4.44, p = .013. In the final version I can report it as the primary analysis and keep complete cases as a robustness check.»

Look at what that paragraph does. It concedes the technical point outright, with no haggling. It shows that the decision had a criterion and where that criterion is written down. And it closes by offering something that improves the thesis instead of defending it. At no point does it tell the examiner they are wrong, and it does not need to: the room has already seen that you know why you did what you did.

If you have not run the alternative, the third beat changes shape but not substance: «I have not fitted it, and I am not going to guess in this room whether the result holds. I will bring it into the final version as the primary analysis and report the interaction under both approaches, including if it goes against me.» That last clause, including if it goes against me, is what separates a credible promise from a procedural one. How the write-up looks afterwards, with the original and the alternative side by side, is in writing results when assumptions are violated.

Script 2: «you ran fifteen correlations and corrected for none of them»

The scene: a correlation matrix with 15 tests, six of them below .05, and no correction anywhere. It is one of the easiest objections to raise and one of the worst answered, because the reflex is to say the analysis was exploratory. Sometimes that is true. Sometimes it is a label you have just invented in the room to rescue six results.

The full script: «That is fair, and the arithmetic is visible: with 15 tests at 5% you expect 0.75 false positives by chance alone even if nothing were related. Four of those correlations correspond to the hypotheses stated in section 3.2, and the remaining eleven are exploratory and labeled that way in Table 5. What is missing, and should be there, is control of the false discovery rate. I have since applied the Benjamini and Hochberg (1995) procedure across all 15: the ones at p = .002, .004 and .009 survive, and the ones at .021, .038 and .046 do not. Those three I would rewrite as exploratory with no directional claim, and none of the chapter's conclusions depends on them.»

What holds that answer up is not the correction, it is the last sentence: knowing which of your conclusions survive and which do not. A committee will happily accept that three results fall away. What it will not accept is a candidate who cannot say which ones. Rehearse that part with your own table, because it is precisely the follow-up question that comes next.

The trap to avoid: if those 15 tests are presented in the thesis as hypothesis tests, you cannot relabel them as exploratory out loud. The document is open in front of them and they will check. The honest move is what the script does — separate the ones that genuinely were prior hypotheses, treat the rest as exploratory from here on, and say out loud what falls.

Script 3: «why compare the groups without testing measurement invariance?»

This is the genuinely hard case: they ask about an analysis you did not run and do not command. The scene: you compare two groups of 56 on observed questionnaire scores, and an examiner asks whether you established measurement invariance first. The temptation to produce something that sounds technical is enormous, and it is exactly what will end you, because the follow-up question takes it apart.

The full script: «I did not test it, and that is a real limitation: by comparing observed scores I am assuming the instrument behaves the same way in both groups, and I never put that to the test. I know how it is done — a sequence of multi-group models, configural, metric and scalar, checking whether fit deteriorates as each constraint is imposed, with the usual criterion of a drop in CFI smaller than .010 (Cheung & Rensvold, 2002). What I cannot tell you today is how it would behave in my sample, because with 56 per group multi-group estimation is tight and I am not going to guess at a result. It is the first thing I would do before sending this chapter to a journal, and if scalar invariance did not hold, the mean comparison I present would have to be reframed in terms of observed scores rather than the latent construct.»

Two rules sit underneath that script. First: never say «I don't know» and stop there. Say what you do know, mark where your limit is, and say how you would resolve it, in that order. Second: never invent a figure or a citation. A number thrown out in the room and contradicted thirty seconds later does far more damage than the gap you were trying to cover, and they already knew the gap was there. When the request is for a whole method you do not command, the logic is the same as with a reviewer: what to do when you are asked for an analysis you do not master.

A separate case: when the person asking is simply wrong. It happens, and you do not settle it by winning the argument, you settle it by handing them the page. «I may not have made that clear in the text: the correction is applied and it is in Table 4, on page 118. If you still read it differently, I will go through it again and make it explicit in the final version.» Never «that is false», never a third round. You have shown the evidence and left them a way out, and that is everything available to you there.

Four answers that do sink a defense

Inventing a number. «I think alpha was .82.» If you cannot remember, do not say it: «I don't have that by heart, it is in Table 3, let me check». A figure misquoted in the room goes into the record as a false claim about your own data, and from that moment every other number of yours gets verified.

Handing the decision to someone else. «My supervisor decided that.» The thesis is yours and your name is on it. You can say the decision was discussed with the team, or that you implemented the analysis with advice, but the criterion has to come out of your mouth. Acknowledging collaboration is normal; using it as a shield is what gets punished.

Total capitulation, or arguing the third round. Two ends of the same mistake. «You are right, the whole analysis is wrong» turns one objection into a thesis-level problem you can no longer walk back. Pressing the same argument a third time turns one objection into a problem about you. Two exchanges, then close it: «I take the point, I will work through it and argue it properly in the final version».

Rescuing a result with elastic language. «Approached significance», «a marginal trend», «in the expected direction». It is the fastest way to lose a methodologically trained examiner, because it is not a claim about an analysis: it suggests you would stretch the data if anyone let you. A result that did not come out is reported with its interval and with what remains compatible with your data, which is the point of what to do when results are not significant.

How to prepare this in the week before

List your three or four most attackable decisions. You already know them: they are the ones that kept you up at one in the morning, the ones you asked about twice, the ones you wrote up with more adjectives than the rest of the thesis. For each, write the three beats on a card: the concession sentence, the criterion with its page, the alternative with its name. If the card goes blank at beat two, you have found something to fix while there are still days to fix it.

Then say them out loud against a clock. A good answer runs between forty and ninety seconds. Written down they all look reasonable; spoken, you discover you spend twenty seconds circling before you name the limit, and that circling is exactly what a panel reads as something being hidden. Record yourself once. It is unpleasant, and it works.

Rehearse with someone who asks in bad faith, not with someone who likes you. If nobody around you has the methodological profile of your panel, the thesis defense simulator generates the questions from your own abstract and aims, which is the only way to practice on your decisions rather than on a generic list from the internet.

One memory rule for the day itself: memorize the location, not the figure. Table 3, page 112. Table 4, page 118. Take the thesis in with tabs on it and open it without embarrassment when you need to. «It is in Table 4 on page 118, let me read it to you» is a better answer than a half-remembered number, and it projects the opposite of what you are afraid it projects.

Frequently asked questions

What do I do if my committee attacks my analysis in the defense?

Answer in three beats. Concede the limit in one sentence with no hedging; explain the criterion you used when you made the decision, in the past tense and saying which page it is on; then offer the alternative analysis by name and estimation method, with the result if you have already run it. Before you start, classify the objection: a factual error is conceded whole, a defensible choice is answered with the criterion, and something you never did is reanalysis or future work.

What should I say if I do not know the answer in my viva?

Never a bare «I don't know», and never an invented answer. Say what you do know about the issue, mark clearly where your limit is, and say how you would resolve it: «I have not estimated it and I am not going to improvise the specification here; I know the sequence is configural, metric and scalar, and it is the first thing I would do before submitting». Examiners tell the difference between an acknowledged gap and a bluff, and only the bluff is punished.

Can you fail a viva because of a statistical mistake?

An outright fail is rare, and a debatable analytic choice almost never causes one. A serious analysis problem usually converts into major corrections, a resubmission, a lower outcome or a report with reservations. What can genuinely sink a defense is not the analysis but defending it with something untrue: an invented figure, an assumption you claim to have checked and did not, or an «exploratory» label applied on the spot.

Can I say my supervisor or a statistician did the analysis?

You can say who did what, and acknowledging real collaboration is the correct thing to do. What you cannot do is use it as a shield: you are the one defending the thesis, and the criterion behind each decision has to come from you. If there is an analysis you can neither explain nor defend, that is the work of the week before, not the argument on the day.

What if the examiner is wrong?

Do not argue, hand them the page. «I may not have made that clear in the text: it is in Table 4 on page 118; if you still read it differently, I will go through it again and make it explicit in the final version.» Two exchanges at most, then close. Winning an argument in front of a panel is not an available outcome; showing the evidence and leaving a way out is.

How do I prepare for statistics questions in a thesis defense?

List your three or four most attackable decisions, write the three-beat answer for each on a card, and say them out loud against a clock: forty to ninety seconds each. Rehearse with someone who asks in bad faith, or with a simulator that generates questions from your own abstract and aims. And memorize where each number lives, table and page, instead of the numbers themselves.

A panel does not attack an analysis to watch you fail: it does it to see whether you know where the limits of your work are. Concede, give the criterion, offer the reanalysis, and none of the three requires you to lie or to fold. If what keeps you awake is not knowing whether your debatable decision is a real problem or one examiner's preference, that second opinion is what my statistical consulting is for, and it works far better before the defense than after it.