Choosing a Statistical Test: The Three-Question Rule

You have had the same flowchart open for twenty minutes: thirty boxes, arrows crossing each other, four tests with nearly identical names and a footnote that says it depends. Your actual question is far smaller than that chart. You want to know whether this is a t test, an ANOVA or a chi-square, you want to know today, and you want to be able to redo it from memory in two months without opening anything. That is not a guide, it is a rule. Here it is, with the full tree behind it and a separate version for intervention designs, which are the ones that go wrong most often.

Three questions, and none of them is “are my data normal?”

The rule fits in three lines. One: what kind of variable is your outcome? Two: how many groups or time points are you comparing? Three: are the observations independent or paired? Answer them in that order, before you open SPSS, jamovi or R, and the branch is fixed. Everything after that is choosing a variant inside that branch.

Three questions are enough because all three describe the design, and the family of tests follows from the design rather than from the data. Thirty-box flowcharts become unreadable because they draw two different decisions on the same page: which family applies to you (settled when you planned the study) and which version of that family survives your data (visible afterwards, with the residuals in front of you).

That is why normality does not come first. The sequence almost everyone is taught — run Shapiro-Wilk, let the result choose the test — is back to front: Shapiro-Wilk does not know whether your observations are paired or how many groups you have, and those are precisely what determine the test. No assumption changes the answer to question 1, 2 or 3. Assumptions move you from Student to Welch, or from the parametric version to the rank version or a bootstrap, always inside the same branch.

The rule has a boundary and it is fairer to say so now: the moment you have more than one predictor acting at once, a latent variable, or an outcome that is “time until something happens”, it leaves you at the door, and the last section tells you what lies beyond it. For everything else — the dissertation, the pilot, the two-arm trial, the service evaluation — three questions will do. If you want the long version, with the reasoning behind each test one at a time, that is how to choose a statistical test. This is the pocket card.

Question 1: what kind of variable is your outcome

The outcome is the variable you want to explain, the one measured afterwards, the one that is supposed to move. It is not the variable that defines your groups. One of the most repeated questions on stats forums is “my variable is gender, which test do I use?” Gender does not choose the test; it only contributes a number of levels to question 2. If you are comparing anxiety in men and women, your outcome is anxiety.

Continuous. A scale total (PHQ-9, GAD-7, DASS-21), a reaction time in milliseconds, a biomarker, age. One convention is worth naming here: the total score of several Likert items is analysed as continuous, not because it strictly is, but because summing four or more items produces enough distinct values for mean-based methods to behave well. A single item with five options is a different animal: that is ordinal, and it belongs in the ranks branch.

Categorical. Binary (reliable recovery yes or no, dropped out or completed, met diagnostic criteria or not) or nominal with more than two categories (attachment style, primary diagnosis, clinic). And there is a third case that is neither: counts (number of relapses in a year, number of missed sessions). They are floored at zero, they pile up on the small values and they do not behave like a continuous variable; they belong in Poisson or negative binomial models, not in a t test.

Keep the translation. Continuous outcome → the means family. Categorical outcome → the frequencies family. Ordinal outcome → the ranks family. Three families, and one question has already ruled out two of them. A continuous outcome with a badly skewed distribution and a small sample does not open a fourth family: you stay in the branch you were given and use its rank-based variant, which is a later decision, taken with the residuals already in front of you.

Question 2: how many groups or time points are you comparing

There are four possible answers and only four. None: you are not comparing groups, you have two variables measured on the same people and you want to know whether they move together. One: a single group against a reference value (a published norm, a cut-off). Two. Three or more. Time points count exactly like groups: pre and post is “two”; pre, post and follow-up is “three or more”.

Count levels of the factor, not people. If you are comparing CBT, medication and a waiting-list control, that is three levels and it calls for an ANOVA, not three separate t tests. The three pairwise tests are not miscalculated, they are ungoverned: each one spends its own 5% type I error rate and the set as a whole ends up well above the rate you claim to control. The ANOVA runs one global test and post hoc comparisons then share that error out across the pairs.

This is also the rule's first exit door. Once you have two factors at once — group and time, or group and sex — you are no longer answering “how many groups”, you are in a factorial or mixed ANOVA, and the interesting question stops being each factor separately and becomes the interaction. And if your predictor is continuous, do not split it at the median so you can run a t test: turning a continuous variable into two groups throws away information, costs you power and invents a threshold that does not exist in nature (MacCallum et al., 2002). That case is a regression.

Question 3: independent or paired (the one people get wrong)

The test for deciding is mechanical. If you can match each observation in one column with one and only one observation in the other, for a reason that existed before you collected the data, they are paired. It is the same person measured twice; it is a matched case-control pair; it is the two members of a couple; it is a twin pair; it is the same participant going through both conditions. If the pairing does not exist outside your spreadsheet, they are independent. And equal group sizes pair nothing: that is a coincidence of n, not a link between observations.

Getting this wrong is not a naming issue, it changes the answer. Thirty patients measured on the PHQ-9 before and after eight sessions: mean improvement M = 4.2 points, with a standard deviation of the difference scores of 6.0. Analysed as what it is, paired, you get t(29) = 3.83, p < .001, d_z = 0.70 (the subscript flags that the d is computed on the difference scores). Analysed by mistake as two independent groups of 30 — which is exactly what your software does if you hand it two columns and ask for an independent-samples t test — with standard deviations of about 9.0 points at each time point, you get t(58) = 1.81, p = .076.

Same data, same improvement, opposite conclusions. The paired version uses the correlation between the two measurements (here r = .78: people who score high at baseline tend to score high afterwards) and with it removes almost all the between-person variability. Throwing the pairing away is giving precision back. The opposite mistake — treating unpaired data as paired — is worse still, because then the pairing is invented by the order of your rows.

There is also a third state that question 3 does not settle with a yes or no: nested data. Patients spread across eight therapists, pupils within classrooms, participants inside group sessions. Neither independent nor paired: two patients seen by the same therapist resemble each other more than they resemble patients of the therapist next door, and ignoring that resemblance inflates your significance. The answer there is not a test, it is a multilevel model.

The full tree, fourteen branches

Continuous outcome. No comparison and two variables measured on the same people → Pearson's r. One group against a known value → one-sample t test. Two separate groups of people → independent-samples t test, and by default the Welch version, which does not assume equal variances and loses almost nothing when they are equal. Two time points on the same people → paired t test. Three or more independent groups → one-way ANOVA, with post hoc comparisons only after the global test. Three or more time points on the same people → repeated-measures ANOVA or, if people drop out along the way, a mixed model.

Categorical outcome. Two or more independent groups and an outcome that is a category → chi-square test of independence on the contingency table; if any expected frequency falls below 5, the convention is to switch to Fisher's exact test. The same people with a binary outcome at two time points (met criteria before, meets them now) → McNemar's test, the paired equivalent of chi-square; confusing the two is the classic error in this branch, and it is question 3 all over again. A single variable against an expected distribution (are dropouts spread evenly across the four clinics?) → chi-square goodness-of-fit test.

Ordinal outcome, or continuous with a badly skewed distribution and a small sample. Two independent groups → Mann-Whitney U. Two paired measurements → Wilcoxon signed-rank test. Three or more independent groups → Kruskal-Wallis. Three or more paired measurements → Friedman. Two ordinal variables you want to relate → Spearman's rho. Notice that the structure is identical to the continuous branch: the statistics change, the answers to the three questions do not.

The effect size is attached to the branch. Each branch brings its own and they are not interchangeable: two means → Cohen's d; a one-way F → partial eta²; a 2x2 table → an odds ratio; an association → r itself. Cohen's (1988) benchmarks — 0.20, 0.50 and 0.80 for d — are conventions for orienting yourself in an unfamiliar field, not laws: in a clinical trial a “small” effect on mortality matters more than a “large” one on a laboratory task. The effect size calculator derives it from what is already in your output: means and standard deviations, a t, an F, an r or the contingency table.

The version for health intervention designs

This design — two arms, one measurement before and one after — accounts for a large share of test-choice errors on its own, and not because the rule fails. It fails because the rule gets applied twice instead of once. The routine that goes wrong is this: a paired t test inside the intervention arm, another paired t test inside the control arm, and the conclusion that “the intervention group improved and the controls did not, therefore the intervention works”.

With numbers: thirty per arm, PHQ-9. The intervention arm drops M = 4.2 points, t(29) = 3.83, p < .001. The control arm drops M = 1.8 points, t(29) = 1.64, p = .111. It looks settled. But what your conclusion rests on is the difference between those two improvements, and that difference is 2.4 points with a 95% confidence interval of [−0.70, 5.50]: the group × time interaction gives F(1, 58) = 2.40, p = .127, partial eta² = .040. The comparison that answered your research question was never run. One effect being significant and the other not does not mean the two differ from each other (Gelman and Stern, 2006).

For these designs the rule holds with one correction: question 2 is not “how many time points” or “how many arms”, it is that you have two factors and what you are comparing is a difference of differences. Three correct ways to test it, in order of practical preference: ANCOVA on the post score with baseline as a covariate, usually the most powerful of the three and the one Vickers and Altman (2001) recommend for randomised trials; a 2 × 2 mixed ANOVA read at the interaction, which tells the same story in a more familiar format; and a mixed model if you have three or more time points or people dropping out, because it does not require everyone to have every measurement (compared in detail in mixed models versus repeated-measures ANOVA).

If your outcome is categorical, the tree is the same but in the frequencies branch. Reliable recovery at discharge, 24 of 60 in the intervention arm (40%) against 12 of 60 in control (20%): chi-square(1, N = 120) = 5.71, p = .017, odds ratio = 2.67. That covers the simple comparison; the moment you want to adjust for baseline severity or for site, you are in a logistic regression, not a chi-square. And one last check on question 3 that health research routinely skips: if you randomised clinics, therapy groups or therapists rather than individuals, your observations are not independent however your rows look, and the analysis has to acknowledge that level.

Where the rule runs out

The rule covers one-factor comparisons on one outcome. It runs out as soon as any of five things appears: more than one predictor at a time (multiple regression), a hypothesis about mechanism or condition — whether the effect travels through a third variable or changes with it — (mediation and moderation), latent variables measured by several items (factor analysis, structural equation modelling), time to an event with people who leave before it happens (survival analysis), and several correlated outcomes you want to handle together. None of these is one more branch of the tree: they are models, and in a model you choose a specification, not a test.

Inside the tree, the order is always the same: the design fixes the family, the residuals fix the variant. Pick the branch with the three questions, fit the model, and only then check what the test takes for granted — normality of the residuals rather than of the raw variable, homogeneity of variance, independence. If something does not hold you have replacements inside the same branch (Welch, robust standard errors, a bootstrap, the rank version), and how to check each one is in how to verify statistical assumptions. What you cannot do is let a normality test move you to a different branch.

One use of the rule saves arguments later: write the three answers into your analysis plan before you collect anything. “Outcome: PHQ-9 total, continuous. Comparison: two arms by two time points. Structure: repeated measures within person, people nested within therapist.” Then, the day a reviewer asks why you used that test, your answer is a description of your design rather than a justification assembled after seeing the result.

Frequently asked questions

When do I use chi-square and when do I use a t test?

Question 1 decides it, and question 1 is about the outcome. If what you measure at the end is a score (PHQ-9, reaction time, a biomarker), you are comparing means and you are in the t test and ANOVA family. If what you measure at the end is a category (recovered or not, dropped out or completed), you are comparing proportions and you are in the chi-square family. How many groups you have is irrelevant here: that is question 2, and it only decides whether you need a t test or an ANOVA inside the first family, a 2x2 or an RxC table inside the second.

Can I run a t test on Likert data?

On the total score of a scale, yes, and that is what the literature does: summing four or more items produces enough distinct values for mean-based methods to behave well, even though the measurement level is strictly ordinal. It is an accepted convention, not a theorem. On a single item with five response options, no: the distance between “somewhat agree” and “agree” is not comparable to the distance between “agree” and “strongly agree”, so you go to Mann-Whitney or Wilcoxon depending on question 3.

How do I know whether my data are paired or independent?

Ask whether you can link each observation in one column to one and only one observation in the other for a reason that exists outside your spreadsheet: it is the same person, the same matched case-control pair, the two members of a couple, a twin pair. If yes, paired. If the only link is the order of the rows, independent. And ignore the false clue of equal group sizes — that pairs nothing.

Which test do I use with two groups and two time points?

None of the basic ones on their own, because you have two factors. What tests your hypothesis is the group × time interaction: a 2 × 2 mixed ANOVA, an ANCOVA on the post score with baseline as a covariate (the most powerful option in a randomised trial), or a mixed model if there are dropouts. Running a paired t test inside each arm and comparing their p values does not test the difference between arms: you can have p < .001 in one arm, p = .111 in the other and an interaction at p = .127.

Do I need to run a normality test before choosing my test?

No, and the order matters. Normality cannot choose your test because it does not know how many groups you have or whether they are paired. Pick the branch with the three questions, fit the model, then check the assumptions on the residuals rather than on the raw variable. If they fail you switch variant inside the same branch — Welch, a bootstrap, the rank version — not branch. On top of that, with a large N Shapiro-Wilk flags trivial departures as non-normal, and with a small one it misses serious ones.

What if my outcome is a count, such as number of relapses?

A count is neither continuous nor categorical: it is floored at zero, it clusters on low values and its variance grows with its mean, so a t test or an ANOVA fits it badly even though your software will run them without complaint. It belongs in a Poisson regression or, if there is more spread than Poisson allows (the usual case for relapses or units consumed), a negative binomial one. Questions 2 and 3 still apply: they decide what goes in as a predictor and whether you need a random term per person.

Keep the three questions in this order: what kind of variable your outcome is, how many groups or time points you are comparing, and whether the observations are independent or paired. Answer them from your design rather than from your data and the branch appears on its own in almost any psychology or health sciences study. If your design falls off the tree — two nested factors, an outcome that is time to an event, several correlated endpoints — you are no longer choosing a test, you are specifying a model, and a specification is decided before you collect data and written down in your analysis plan.