Defense Questions About Your Statistics (and How to Answer Them)

There is a thread on r/GradSchool titled «Just failed my defense» with 778 upvotes and 177 comments. A post pulling that kind of response says something about the fear people carry into the room, and the fear is almost never about the literature chapter. It is about the ten minutes when someone on the committee adjusts their glasses, looks at your results section and asks why you did that. You can run the analysis, you did, it worked, you wrote it up. Justifying it out loud in front of three people who have been publishing for twenty years is a different skill, and nobody trained you in it. This is only that block: nine statistics questions that come up defense after defense, in a viva or a committee, with the answer that defuses each one.

The panel is not checking whether you know statistics. It is checking whether the decision was yours

Almost no statistical question from an examiner has a single correct answer. They ask «why a regression and not a multilevel model?» knowing perfectly well that both are defensible. What they want to find out is something else: whether you chose, or whether you followed the tutorial you found at two in the morning. A candidate who says «I applied Bonferroni» has executed. One who says «I applied Bonferroni because my six contrasts were confirmatory and I would rather lose power than inflate type I error» has decided. The difference is audible within three seconds.

That is why all nine take the same shape, and the shape is worth memorizing. One: what I did. The decision, in one sentence, no hedging. Two: on what criterion. The real reason, tied to your question and your data, not to what the field happens to do. Three: what it costs. What your choice leaves out, said by you before the panel says it. That third beat is what turns an answer into a good answer: naming the limitation yourself proves you saw it coming.

There are two symmetrical ways to ruin it. One is the short, defensive reply, «that is what the field uses», «my supervisor told me to», which hands responsibility to someone else and leaves the panel thinking you do not understand your own analysis. The other is the avalanche: five minutes of technical detail to hide the fact that you have no criterion. Thirty to forty seconds per question is enough; if they want more, they will ask. This piece covers only the statistics. The general ones, design, contribution, generalisability, are in the 12 questions your committee will ask.

«Why this test and not another?» and «how did you justify the sample?»

1. Why this test and not another? The answer is built from four pieces, in this order: what your unit of analysis is, what scale your outcome is on, how your data are structured (independent, paired, nested) and which assumptions you could actually sustain. «I compared two independent groups on a continuous score, with unequal variances by Levene, so I used Welch's t rather than Student's» is a complete answer in fifteen seconds. What does not work is the reverse order: picking the test and shopping for a justification afterwards.

There is almost always a second half, and it is the one that throws people: someone proposes the model they would have used. It is not an ambush, it is what any researcher does when reading somebody else's work. Acknowledge the alternative without fighting it, say what it would have added, and say why yours was reasonable given what you had: «a proportional-odds model would have respected the five-point ordinal response better; I checked that the conclusion did not change when I analyzed it as continuous, which is why I kept the original». Arguing with the premise of the question is the worst road available.

2. How did you justify your sample size? This is the headline question of the block. If you ran the calculation in advance, give all three parameters and the result: «to detect d = 0.50 with .80 power and a two-tailed alpha of .05 I needed 64 per group; I recruited 51 and 47 because the clinic closed referrals in March». The second half of that sentence matters as much as the first, because a panel takes a well-explained fieldwork limit and does not take a number with no origin. The whole procedure is in the guide to calculating sample size for a thesis.

What you must not do, even if the room asks for it, is compute observed power after the fact from your own effect: it is a one-to-one transformation of the p value and adds no new information (Hoenig and Heisey, 2001). The correct alternative is called a sensitivity analysis and it answers the real question: with 51 and 47 participants, the smallest effect detectable at .80 power was d = 0.57. Put that way, you are running the conversation: «my design could see medium effects, small ones would have escaped it, and that is why I claim nothing about them».

«Did you check the assumptions?» and «why not a mixed model?»

3. Did you check the assumptions of your analyses? Almost nobody fails on the yes or no. People fail on the question that lands immediately after: and what did you do when one of them failed? Know what each test required, and watch two places where half of all candidates trip. In a regression the normality that matters is of the residuals, not of the outcome variable; and in large samples Shapiro-Wilk will call any trivial departure non-normal, so the decision is made by looking at the plot rather than at the p value alone.

The answer that works is the one that includes what did not change. «Levene came out at p = .03, so I moved to Welch, which is a sound default even when Levene is non-significant (Delacre, Lakens and Leys, 2017), and I re-ran the comparison with Mann-Whitney U as a check: the conclusion was identical.» A sensitivity analysis run in parallel disarms the question before it is fully formed. The exact wording for the manuscript is in writing results when assumptions are violated.

4. Why did you not use a mixed model? This comes up whenever you have repeated measures and attrition, and there is a very concrete number behind it. Picture 84 participants measured at three time points, 19 of whom missed a session, for 26 lost measurements in all. A repeated-measures ANOVA deletes listwise and analyses 65 people: F(2, 128) = 4.31, p = .015, partial eta squared = .063. A mixed model uses the 226 available observations out of a possible 252, does not discard a whole person over one empty cell, and estimates the change more precisely: -2.8 points, 95% CI [-4.9, -0.7].

If you did not run one, the worst escape is inventing a criterion on the spot. The good answer is two sentences: why the ANOVA was defensible at the time (sphericity met, limited attrition that looked random, a balanced design) and what the mixed model would have added (keeping incomplete cases instead of deleting them, and modeling individual variation in slopes). And if you have run it since submission, say so: «I re-ran it as a mixed model while preparing the defense and the effect holds» closes the topic and earns credit. The full comparison is in mixed models versus repeated-measures ANOVA.

«How do you know this is not an artifact?»

«Artifact» is the word examiners reach for when they suspect your result was produced by the procedure rather than by the phenomenon. Behind that single word sit three quite different suspicions, and it pays to work out which one you are being asked, because the answers have nothing in common: one particular case is driving the result, a bias was introduced by measuring everything the same way, or an effect showed up because you looked many times.

5. How do you know a single case is not producing this? You answer this with an analysis already done, not with an intuition. In a correlation with N = 96 giving r = .34, p < .001, dropping the case with the largest Cook's distance leaves r = .21, p = .041 with N = 95: still significant, but nearly 40% smaller, and you should be the one to say so. Bring the influence diagnostics with you and a backup slide showing the result with and without that case. The line is «the effect does not depend on any individual case, although its magnitude is sensitive to one», and it is worth vastly more than «I checked for outliers».

The second suspicion arrives the moment your whole study is a self-report questionnaire completed in one sitting: part of the correlation between your variables may come from the method rather than the construct (Podsakoff, MacKenzie, Lee and Podsakoff, 2003). If you present Harman's single-factor test, present its limit too, because it is a weak test that almost never detects anything and a clean result from it does not demonstrate the absence of bias. What convinces is design control, temporal separation between measures, a different informant, an unrelated marker; and if you had none of those, say which direction the bias pushes your estimates. It is worked through in common method bias and Harman's test.

6. Did you correct for multiple comparisons? Nine variables in a matrix give 36 correlations, and if every null hypothesis were true you would expect 1.8 significant ones at the .05 level by chance alone. If five came out, with p values of .0007, .009, .021, .038 and .041, the Benjamini and Hochberg (1995) procedure applied across all 36 lets exactly one survive: the first. That does not mean the other four are false, it means they are not a confirmatory finding. If you did not correct, the right answer is not to defend all five, it is to relabel: «this block is exploratory, I present it as hypothesis-generating, and only the first survives control of the false discovery rate». A panel forgives declared exploratory analysis. It does not forgive exploratory analysis dressed as confirmatory.

«Your data are cross-sectional» and «it is significant, but is it meaningful?»

7. With cross-sectional data, why are you talking about effects or mediation? This is the question that sinks the most manuscripts later, in review, so the panel uses it as a warning. Bootstrapping an indirect effect, ab = .14, 95% CI [.05, .26], is an inference technique, not a causal machine: it does not turn a photograph into a film. Maxwell and Cole (2007) showed that mediation estimated cross-sectionally can be badly biased relative to the longitudinal process it claims to represent. And there is a simpler argument still: with three variables measured at once the model is saturated, so the version with X and M swapped fits exactly the same. Fit cannot tell them apart, so fit proves nothing here.

The way out is not to withdraw the analysis, it is to narrow the verb. «This design cannot establish temporal sequence; what I show is a pattern of association consistent with the theoretical model, and also with its reversed version, which fits exactly the same.» If you also arrive having tested the alternative model, and can name the study that would separate the two, that question becomes your best minute of the afternoon.

8. It is significant, but is it meaningful? Repeating the p value is no help here: you have to translate the effect into the units of the instrument. A d = 0.44 on an inventory whose standard deviation is 5.5 points is a 2.4-point difference between groups. And because the interval runs from 0.04 to 0.84, the honest translation is uncomfortable: your data are compatible with an improvement of 0.2 points and with one of 4.6. If the smallest change your clinicians would call useful is 3 points, your study neither confirms it nor rules it out, and saying so yourself beats hearing it.

Two warnings about that answer. Cohen's (1988) cut-offs are conventions he proposed in the absence of field-specific benchmarks, not laws: in a brief, cheap intervention delivered to thousands of people a d = 0.20 can be enormous, and in an expensive drug trial it can be worth nothing. And the sentence that resolves the question is always the same: «the point estimate is this, the interval reaches this far, and in the units your clinician cares about that means this». One translated number is worth three paragraphs of defending.

«What would you do differently?», the question that sets the tone of the ending

9. What would you do differently if you started tomorrow? It arrives near the end and it sounds friendly. It is not entirely: it checks that you hold a view of your own work, and it can be answered badly in two opposite ways. Say «nothing, I would do it the same» and you are saying you learned nothing in four years. Open your chest and confess six errors in a row and you have just written the panel's report for them.

A good answer has a recognisable shape: two concrete changes, of design or of measurement, never of effort. «I would have measured at four time points rather than three, because with three I cannot model a non-linear trajectory» is a design change and it demonstrates command; «I would have worked harder» says nothing. And the criterion for choosing which ones to name is this: something you would change that does not invalidate what you did. If the change that comes to mind invalidates your thesis, that is not the answer to this question, it is a limitation that should already be written into your discussion.

There is a rougher version of the same question, phrased as «do you not think you should have done X?». The way out is identical: concede the point, say what X would have added, and hold the line on why your choice answered your question. And you are allowed not to know something, and to say so. «I do not have command of that procedure; what I know about it is this, and I would study it before applying it to these data» is a perfectly acceptable answer in a viva. Faking command and having the next question expose it is not.

Rehearsing it out loud is half the work

Reading nine questions is not preparing them. The gap between having them clear in writing and answering them on your feet, with three people watching and nothing to consult, is precisely what breaks on the day. What works is one card per decision: the decision on one side (the test, the N, the correction, the model you rejected), and on the other the three sentences, what I did, on what criterion, what it costs, written out in full. If they do not fit in three sentences, you do not yet know why you did it.

Then rehearse them out loud, timed. Thirty to forty seconds each. The ones that take you a minute and a half are exactly the ones the panel will follow up on, because hesitation carries to the front row. And record yourself once: it is uncomfortable, and it is the fastest method I know for discovering that you have spent four years saying «I guess» in front of the word assumption.

If you want the questions for your actual thesis rather than the generic nine above, with your design, your N, your model and the limitation you already suspect, there is a defense simulator that generates them from your abstract and lets you answer them before somebody with a vote asks.

Frequently asked questions

What statistics questions does a thesis committee ask in a defense?

Nine come up again and again: why that test and not another, how you justified your N, whether you checked the assumptions and what you did when they failed, why not a mixed model, whether the result could be an artifact of one case or of the method, whether you corrected for multiple comparisons, why you talk about effects with cross-sectional data, whether the effect is meaningful as well as significant, and what you would do differently. None of them has a single correct answer: they are checking whether the decision was yours and whether you know what it cost.

How do I justify my sample size if I never ran a power analysis?

With a sensitivity analysis, never with post hoc power. Power computed after the fact from your own observed effect is a one-to-one function of the p value and adds nothing (Hoenig and Heisey, 2001). Work out the smallest effect your sample could detect at .80 power (with 51 and 47 participants, d = 0.57) and explain the real reason for the recruitment you achieved. A well-explained fieldwork limit is accepted; a number with no origin is not.

What do I say if the panel asks why I did not use a mixed model?

Two sentences. Why the model you used was defensible with your data (sphericity met, limited attrition, a balanced design) and what a mixed model would have added: keeping incomplete cases instead of deleting them listwise, and modeling individual variation in slopes. If you have run it since submission and the effect holds, say so, because it closes the topic. What does not work is inventing a methodological criterion on the spot.

Can I say I do not know something in my viva?

Yes, and answered well it earns credit. The formula is to concede the limit, contribute what you do know, and say what you would do about it: «I do not have command of that procedure; I know what it is used for, and I would study it before applying it to these data». What sinks a defense is not an «I do not know», it is faking command and having the next question expose it.

What do I do if an examiner proposes a different analysis?

Do not argue with the premise. Acknowledge the alternative, say what it would have added, and say why yours was reasonable for your question and your data. And if you have time before the defense, run the alternative: walking in able to say «I checked, and the conclusions do not change» turns the most uncomfortable criticism of the session into your best answer.

Do people fail a thesis because of the statistics?

Outright failure on the analysis alone is rare, but statistics is a frequent reason for the major corrections that cost months. And what triggers that outcome is almost never a wrong analysis: it is an analysis the candidate cannot justify out loud, or a conclusion that goes further than the data allow.

A defense is decided in two places: in what you did, which is already written and cannot be changed, and in how you justify it, which can be prepared in a week. The nine questions here cover the second. Write out the three sentences (what I did, on what criterion, what it costs) for every statistical decision in your thesis, say them out loud and time them. On the day, that is the whole difference between an answer and a silence.