Shapiro-Wilk Is Significant but Your Data Look Normal

You open the output, look at the histogram and see a bell. You check the Q-Q plot and the points sit on the line, with the faint curl at the end that every real variable has. Directly underneath, in the normality table, sits p = .003 announcing that your data are not normal. What usually happens next is one of two things: you bin the t test you had planned and switch to Mann-Whitney without being sure what you are now testing, or you log-transform the outcome and hope nobody asks. Both moves cause more damage than the problem they are meant to solve. This is about why the test says that, how you actually decide whether the departure matters, and what sentence to write so that nobody argues with you about it.

Shapiro-Wilk is not answering your question

The null hypothesis of the Shapiro-Wilk test (1965) is that your sample came from an exactly normal population. Not "roughly normal", not "normal enough for a t test": normal, the Gaussian curve with its two parameters and its infinite tails. When the test comes out significant, what you are reading is "with these data I can rule out exact normality", and nothing more. It is a statement about what your sample lets you rule out, not about how far your variable sits from the bell.

The trouble is that this null hypothesis is false before you collect a single case. Micceri (1989) went through 440 real psychometric and achievement datasets and found not one that matched the normal curve; the title of that paper puts the normal distribution in the same category as a unicorn. That is not surprising. Likert scores are discrete and bounded, reaction times cannot be negative, a symptom questionnaire in a community sample has a floor, and a wellbeing measure has a ceiling. No psychological variable is exactly normal, so any normality test will eventually reject if you give it enough sample.

That is the machinery behind the p = .003 on your screen. Like every significance test, Shapiro-Wilk folds two separate things into one number: how far the sample departs from the model, and how many people you measured. A microscopic departure with a large sample and a violent one with a small sample can return the same p. It leaves you with an odd instrument, one that hands you a yes-or-no verdict on a question that is entirely a matter of degree, with the verdict driven largely by a quantity that has nothing to do with the shape of your distribution.

Your question is a different and much more specific one: is my departure from normality large enough to damage the analysis I want to run? That depends on the size of the departure, on the analysis, on N and on the design, and it has no yes-or-no answer. It is precisely the question a p value cannot address, and the reason the methodological literature has spent two decades saying these tests should not be used as decision rules (I take that apart in more detail here).

One distribution, three sample sizes

The fastest way to see the mechanism is to build a population and draw samples of different sizes from it. I simulated a variable with skewness 0.40 and essentially zero excess kurtosis: a bell tilted slightly to the right, the sort a distress scale produces in a community sample, with a mean of 20 and a standard deviation of 6. Nobody eyeballing that histogram would call it anomalous. I drew a sample of 40, one of 200 and one of 1000 from it, and ran Shapiro-Wilk on all three.

At N = 40 the result was W = .969, p = .324, skewness 0.41, kurtosis 0.00. At N = 200, W = .986, p = .043, skewness 0.40, kurtosis −0.03. At N = 1000, W = .992, p < .001, skewness 0.36, kurtosis 0.05. Look at what W did, because W is the quantity that actually measures the misfit: it went UP. W behaves like a correlation between your ordered data and the normal scores you would expect (that correlation squared is literally the statistic of the Shapiro-Francia variant), so the closer to 1, the better the bell fits. It climbed from .969 to .992 while the p value fell through three orders of magnitude. The fit index says the sample matches the bell better each time and the verdict says the opposite.

There is no contradiction: what changes is the bar W is measured against. To reject at .05 you need W below .906 at N = 20, below .975 at N = 100, below .986 at N = 200, below .997 at N = 1000 and below .998 at N = 2000. A W of .99 is a decisive rejection in a sample of a thousand and a comfortable pass in a sample of forty. Same number, same fit, opposite verdict.

This is not one unlucky draw either. Repeating the sampling twenty thousand times from that same skewness-0.40 population, the test rejects 8.7% of the time at N = 20, 15.7% at N = 50, 28.4% at N = 100, 51.8% at N = 200, 90.9% at N = 500 and 99.8% at N = 1000. The population never shifted by a tenth across that whole range. If a reviewer asks whether your data are normal, the honest reply is "exactly as normal as they were at N = 200, where this same test said yes".

With small samples it misses the departures that would matter

The other half of the problem gets discussed far less and is just as damaging. Take reaction times from a lexical decision task, with the right skew they always carry: in my simulation, a lognormal with population skewness 2.26 and excess kurtosis 10.27 (excess kurtosis is what SPSS and R report, with the normal sitting at 0). This is no longer a slightly tilted bell, it is a textbook illustration of non-normality. How often does Shapiro-Wilk catch it? At N = 10, 32% of the time. At N = 15, 50%. At N = 20, 65%. At N = 30, 84%. With twenty participants per condition, one in three replications of the same study sails straight through the check.

Bimodality is worse still, and it is the case that should genuinely worry you. I simulated a 50/50 mixture of two subpopulations whose means sit 2.8 standard deviations apart: responders and non-responders to a treatment, or two sites running slightly different protocols pooled into one file. That distribution has two humps, skewness 0 and excess kurtosis −0.88. Shapiro-Wilk detects it 9.8% of the time at N = 20, 14.8% at N = 30, 27.6% at N = 50 and 59.2% at N = 100. Five out of every six samples of thirty pass the check with two distinct populations sitting inside them.

Put the two halves together and you have an instrument that works backwards relative to what you need. It has no power where non-normality has consequences (small samples, where the central limit theorem has not rescued anything yet) and enormous power where it no longer does (large samples, where it has). And the departure it almost always misses, two subpopulations blended together, is exactly the one that changes your conclusions, because the mean of a bimodal distribution describes nobody: it lands in the valley between the two groups, where no participants live.

There is a third and subtler problem with the very act of chaining one test to another. Rasch, Kubinger and Moder (2011) and Zimmerman (2004) studied this carefully: once you choose which test to run based on what a preliminary test told you about those same data, the error rate of the whole procedure is no longer the one you think you have, because you have conditioned a decision on a random outcome from the same sample. A fixed rule written into the protocol, "Welch's test for all comparisons", has better properties than a conditional one improvised at the analysis stage.

What to look at instead: the plot, the skewness, the kurtosis

The Q-Q plot comes first, and it helps to know what you are looking for. An S-shaped curve is skewness. Ends peeling away from the line at both extremes are heavy tails, more extreme values than a normal would produce. A staircase pattern means you have few distinct values, which is not a normality problem but a measurement one: a five-point Likert item, or a count with a pile of zeros. A kink halfway along, with two straight stretches that do not join up, is the signature of two subpopulations. A histogram, by contrast, changes shape with whatever bin width the software picked and conceals exactly where the problem usually lives, which is the tails.

Then the raw skewness and kurtosis, not converted into anything. The most cited conventions come from the structural equation modeling literature: Curran, West and Finch (1996) found that maximum likelihood estimation holds up reasonably well to absolute skewness of 2 and absolute kurtosis of 7, while Kline (2016) works with 3 and 10; a stricter rule of thumb, repeated in many textbooks, puts both at ±1. These are conventions, not laws. They mark where problems started appearing in those particular simulations, for those particular analyses, and outside that context they get cited with far more faith than they earned.

Here is the warning almost everyone walks past: do not divide skewness by its standard error to get a z and check whether it clears 1.96. That reintroduces the identical disease. The standard error of skewness is roughly SE = √(6/N), so it shrinks with sample size: 0.51 at N = 20, 0.24 at N = 100, 0.17 at N = 200, 0.077 at N = 1000 and 0.055 at N = 2000. Translated: with two thousand participants, a skewness of 0.11 is already "significant", and a skewness of 0.11 harms nothing. Read the raw value and compare it against a stated convention. If you want the shortcut, drop your descriptives into the normality interpreter and it tells you which convention you fall inside and which analysis it touches.

And the most expensive mistake of all: what is assumed normal are the residuals, not the variable. In regression, ANOVA and the t test, the assumption sits on the model errors. A predictor can be as skewed as it likes (age, income, years since onset) and violate nothing, because no distribution is assumed for predictors. Worse: if you have two groups and a large effect, the pooled outcome will look bimodal by construction, because that bimodality IS your effect, and it vanishes the moment you subtract the group means. Running Shapiro-Wilk on the raw outcome without splitting by group means flagging your own finding as an assumption violation.

t tests and ANOVA hold up better than you think, and what actually breaks them

The assumption behind the t test was never that your data are normal, but that the sampling distribution of the mean is, and the central limit theorem takes care of that once the sample grows a little. The numbers are blunt. I simulated the Type I error rate of the independent-samples t test using the reaction-time distribution above, the one with skewness 2.26, drawing both groups from the same population so that every significant result is a false positive by construction. At n = 15 per group it came out at 4.50%. At n = 30, 4.67%. At n = 50, 4.83%. The nominal rate was 5%, and note the direction of the drift: the test stays below it, never above, and edges back toward 5% as the sample grows. The distribution Shapiro-Wilk rejects at p < .001 without blinking does not make the t test lie; at worst it makes it slightly conservative.

For ANOVA the work has been done properly: Blanca and colleagues (2017) worked with non-normal distributions taken from real psychological data, spanning the skewness and kurtosis values that actually turn up in the literature, and showed that F held its Type I error rate close to nominal across the conditions they simulated. Boneau had shown the same for the t test back in 1960. The robustness has genuine limits (extreme skewness with tiny, unequal groups, and heavy tails that cost you POWER even when they do not cost you error rate, which is the point Wilcox has been making for years), but "p = .003 in Shapiro-Wilk at N = 400" is nowhere near those limits.

What does break the t test, and what the normality test cannot see, is unequal variances combined with unequal group sizes. The numbers, this time with perfectly normal data: a group of 60 with a standard deviation of 1 against a group of 20 with a standard deviation of 3 makes Student's t reject a true null hypothesis 20.49% of the time instead of 5%. Swap the group sizes and it drops to 0.35%, so conservative that the test stops finding anything at all. Welch's t stays at 5.07% and 4.95% respectively. Impeccably normal data, a normality test passing with distinction, and the analysis broken underneath it.

The practical conclusion is short: Welch by default, always, including with equal group sizes. In that same simulation, at 40 versus 40 with those same unequal standard deviations, Welch returned 5.02% and Student 5.29%; it costs nothing and it covers you against the one case that genuinely hurts (Delacre, Lakens and Leys, 2017). And one more assumption no normality test will ever flag: independence. Repeated measures treated as independent rows, participants nested within classrooms, therapists or sites, several trials from the same person entered as separate cases. That wrecks your standard errors far more thoroughly than a skewness of 0.40 ever will, and it appears in no normality table (I go into it in writing results when assumptions are violated).

What to write, and what to do if the problem is real

Start with the sentence, because that is what heads off the reviewer's email. Something like: "Normality of residuals was assessed by visual inspection of Q-Q plots and by skewness (0.41) and kurtosis (−0.12), both within the limits recommended by Curran et al. (1996). The Shapiro-Wilk test was not used as a decision criterion, as its power depends on sample size (N = 412) and it detects departures with no consequences for the estimated model. All comparisons used Welch's t test." That paragraph does three jobs at once: it says what you looked at, what you decided, and why, which is the difference between a methodological choice and an omission. The formatting rules for those indices, italics and decimals included, are in how to report normality tests in APA 7.

If the departure genuinely is large (skewness above 2, fat tails, or a small sample with a visible break in the Q-Q plot), then there is work to do, and it pays to do it in order. Welch is already in place. The next step is bootstrapping: resample 5000 times and take the percentile or BCa confidence interval for the effect. It assumes normality of nothing, it reports just as cleanly, and in practice it is the tidiest answer to a reviewer who doubts the distribution; what it does need is a sample that is not tiny, because with fifteen cases you are resampling fifteen cases. Third come robust methods: 20% trimmed means and Yuen's test, which hold the error rate under heavy tails and recover the power you were quietly losing.

Then there is the good option, the one almost nobody takes: model the process that generated the data instead of forcing it to look normal. A count with many zeros (relapses, appointments attended, binge episodes in the past month) is not a normal variable that got bent out of shape; it is a count, and it wants Poisson or negative binomial regression. A five-point Likert item is ordinal and wants an ordinal model. Reaction times are lognormal or ex-Gaussian by construction. Transformations (log, square root) sit at the bottom of the list precisely because they change the hypothesis: the mean of the logs is not the log of the mean, and your conclusion is now about a geometric mean that nobody in your field knows how to interpret.

One final warning, because it is the most common reflex: Mann-Whitney is not the automatic substitute. It does not compare means, and it only compares medians if you add the assumption that both distributions have the same shape. Without that assumption it tests stochastic dominance, the probability that a randomly drawn score from one group exceeds one from the other. That is a perfectly legitimate hypothesis and sometimes the one you actually want, but it is a DIFFERENT one, and it also misbehaves when variances differ. Switching to it "because normality failed" means changing the question halfway through the analysis without telling anyone, starting with yourself.

Frequently asked questions

Can I still run a t test if Shapiro-Wilk is significant?

In the overwhelming majority of cases, yes. The t test assumption concerns the sampling distribution of the mean, not your raw data, and unless the sample is tiny the central limit theorem handles it. In simulation, a distribution with skewness 2.26, far more skewed than anything Shapiro-Wilk is flagging in your N = 300 dataset, gave a Type I error rate of 4.50% at 15 per group and 4.83% at 50, against a nominal 5%: below it, never above. Before switching tests, look at the skewness, the kurtosis and the Q-Q plot, and use Welch's version.

What skewness and kurtosis values count as acceptable?

The most cited conventions are |skewness| < 2 and |kurtosis| < 7 (Curran, West and Finch, 1996), and |skewness| < 3 with |kurtosis| < 10 in Kline (2016); a stricter rule found in many textbooks puts both at ±1. These came out of specific simulations of specific analyses rather than being universal thresholds: they mark where problems began to show up in that context. Use them as a benchmark you state in the method, cite where you took them from, and do not present them as a formal requirement.

Should normality be checked on the variable or on the residuals?

On the residuals. In regression, ANOVA and the t test the assumption applies to the model errors, not to the observed variables. A skewed predictor violates nothing. And there is one case where confusing the two is expensive: with two groups and a large effect, the pooled outcome will look bimodal, because that bimodality is your effect; split by group, or inspect the residuals, and it disappears. Save the residuals from the model you intend to estimate and examine those.

Is Kolmogorov-Smirnov or Anderson-Darling a better choice?

No, because the issue is not which test you pick but using a test for this at all. All three share the same dependence on sample size: with a large N they reject trivial departures, with a small N they detect almost nothing. Kolmogorov-Smirnov is additionally weak when you estimate the mean and standard deviation from the sample itself, which is always what happens, and needs the Lilliefors correction to avoid being outright misleading. Shapiro-Wilk is usually the most powerful of the three across most comparisons, which in this context is more of a drawback than a selling point.

My sample is small and Shapiro-Wilk was not significant, so am I fine?

No, and this is the mirror image of the other error. With small samples the test sees almost nothing: in simulation, a distribution with skewness 2.26 slips past it 35% of the time at N = 20 and 50% at N = 15, and a clear mixture of two subpopulations passes in 85% of samples of 30. A non-significant result in a small sample is not evidence of normality, it is absence of power. Inspect the Q-Q plot anyway, and with a small N also go through the extreme cases one by one.

How do I answer a reviewer who insists on a normality test?

Give them the number and the argument, not a refusal. Something along these lines: "Residuals were examined via Q-Q plots and by skewness (0.41) and kurtosis (−0.12), both within conventional limits (Curran et al., 1996). We chose not to base the decision on the Shapiro-Wilk test because its power depends on sample size and at N = 412 it detects departures that are irrelevant to the model; all comparisons use Welch's t test, which is robust to heteroscedasticity (Delacre et al., 2017)." If they press, report the test in a footnote and keep the analysis as it stands: reporting a test is not the same as deciding with it.

Keep the order in mind: the Q-Q plot first, then the raw skewness and kurtosis against a convention you cite, and the normality test nowhere in the decision chain. Check the residuals rather than the variable, make Welch your default, and save the heavy machinery (bootstrapping, trimmed means, count models) for departures that are genuinely large. If what you are facing is a reviewer who will not settle for that, or a variable you suspect wants an entirely different model, that is what my statistical consulting is for.