What to Do With Outliers and How to Justify It in Your Method

The sentence that sinks a Results section is not “outliers were removed”. It is the version that never says how many, on what basis, or when the decision was made. A reviewer who reads “outlying cases were excluded” and then meets a p of .043 three lines later has two facts and no way of knowing whether the second depends on the first, so they will do the only sensible thing and ask for the analysis with every case retained. This is the full decision tree — identify, decide, act — with the thresholds people actually use, what each route costs you, and wording you can lift straight into your Method. The wider map of preliminary checks is in how to check statistical assumptions.

Identifying, deciding and acting are three different steps

Most of the damage comes from treating “outlier” as a category with an automatic consequence attached. It is not. Detecting that a case sits far from the rest is an arithmetic operation that tells you nothing about what to do with it. Between detection and action sits a step almost nobody writes down, and it is the only one a reviewer can evaluate: why that case is far away. An impossible value, a participant who never engaged with the task, and a person who genuinely scores 41 on the DASS-21 depression subscale are all “far away”, and all three are handled differently.

The distinction worth carrying in your head while you stare at the output is the one Aguinis and colleagues set out in 2013. Error outliers. The value cannot exist, so you correct it or flag it as missing — that is data cleaning, not results management. Interesting outliers. The value is real and rare, it usually stays, and occasionally it is the best thing in the study. Influential outliers. The value moves the model's estimates, which a z score cannot detect at all; you need influence diagnostics, and in any regression this is the category that actually matters.

The second failure is one of timing. A rule chosen after you have seen the test is not a rule, it is a researcher degree of freedom — the sort Simmons, Nelson and Simonsohn documented in 2011, which can push almost any p below .05 without a single number being falsified. Their own fix was one sentence long: report how you determined your sample size, all data exclusions, all manipulations and all measures. APA's reporting standards ask for the same disclosure. Where that paragraph lives is covered in how to write a data analysis plan.

In practice your data-analysis subsection needs four things on paper before any result exists: which index you will use to flag extreme cases, at what threshold, what you will do if any appear, and where the count will be reported. Four lines written in advance are worth more than two pages of retrospective argument, and they spare you the awkward exchange with Reviewer 2.

Identifying: z scores, the IQR and Mahalanobis distance

The standardized score is the most used criterion and the most often misused. The usual threshold in psychology is |z| > 3.29, which corresponds to a two-tailed probability of .001 under a normal curve; the familiar |z| > 3 is the rounded version. The catch is that z has a ceiling set by sample size: the largest value possible in a sample of n cases is (n − 1) divided by the square root of n. At n = 12 that ceiling is 3.18, so no case can reach 3.29 however outrageous it looks. Below n = 13 the criterion detects nothing at all, and that is precisely the range of most pilot studies and nearly every single-case design.

The second flaw is masking: the mean and standard deviation you standardize against are themselves contaminated by the extreme case, which inflates them and so hides itself. The robust alternative is the modified z of Iglewicz and Hoaglin, which swaps the mean for the median and the standard deviation for the median absolute deviation (0.6745 times the difference between the value and the Mdn, divided by the MAD), flagged at 3.5. The other alternative is Tukey's fences, the ones behind every boxplot: 1.5 times the interquartile range beyond the third quartile marks a case as mild, and 3 times marks it as extreme. SPSS distinguishes the two on the plot — circles for mild, asterisks for extreme — and hardly anyone notices.

Criteria disagreeing is the norm, not an oddity. In a sample of N = 243 with M = 11.86 and SD = 8.24, a score of 41 gives z = 3.54 and clears any threshold, while a score of 38 gives z = 3.17 and clears the rounded threshold but not 3.29. With the first quartile at 6 and the third at 16, Tukey's mild fence sits at 31 and flags both, but the extreme fence at 46 flags neither. Pick the criterion after seeing which one leaves you the tidier result and you are doing exactly what the previous section describes. Choose one, write it down first, and apply it identically to every variable.

Once several variables are in play none of that helps, because a case can be unremarkable on each variable separately and impossible as a combination: a 19-year-old with 30 years of work experience stands out on neither age nor experience. That is what Mahalanobis distance is for — it measures how far each case sits from the multivariate centroid while accounting for the correlations. Save it from Analyze > Regression > Linear > Save > Distances > Mahalanobis in SPSS, or use mahalanobis() in R, and test D² against a χ² with as many degrees of freedom as variables entered, at a deliberately strict α of .001. With three predictors the critical value is 16.27; in the running example two cases exceeded it, at D² = 17.84 and 19.02.

One clarification that saves a lot of confusion: this is not the normality check, even though both happen at the same moment and often from the same SPSS table. A distribution can be badly skewed without a single extreme case, and can hold three extreme cases while staying symmetrical. What to do with a Shapiro-Wilk p is a separate question, covered in why the Shapiro-Wilk test gets misread.

An extreme case is not an influential case

This is the section that prevents half of all unnecessary deletions. A case can sit miles away and change nothing. In regression it pays to separate three quantities: leverage, which measures how unusual the case is on the predictors; discrepancy, which measures how badly the model predicts it (the studentised residual); and influence, which is essentially the product of the two and the only one that answers the question you care about — would the coefficient change if this case went away? A high-leverage case with a small residual is far out and holding the line steady, not bending it.

The indices come with their own conventions. Cook's D is usually read against the classic cut-off of 1, which is deliberately permissive; the alternative rule of 4 divided by n is so strict that in a sample of 243 the cut-off lands at 0.016 and flags dozens of participants, which makes it useless except as a queue for inspecting cases one at a time. Leverage is compared against 3 times the number of parameters divided by n — 0.049 here — and DFBETAS, which tell you how far each individual coefficient shifts, against 2 divided by the square root of n, or 0.13. In R they come from cooks.distance(), hatvalues() and dfbetas(); in SPSS, from the same Save tab that produced the Mahalanobis distances.

Follow the example through. The two cases that exceeded the multivariate criterion had a maximum Cook's D of 0.11, well under 1, and no DFBETAS above the 0.13 cut-off. Translated: they are unusual cases that are not moving the model. The full model gave β = .29, p < .001 and the model without them gave β = .26, p < .001. With those two numbers in hand, removing them is not merely unnecessary — it is worse, because it alters the sample you described in the Method in exchange for nothing.

The practical consequence is that the index that should drive your decision is influence, not distance. A case with a Cook's D of 2.8 in a sample of 90 deserves a paragraph even if its z is only 2.4; a case with z = 4.1 and a Cook's D of 0.03 deserves one line saying you checked. If only one of the two makes it into the manuscript, make it the influence.

Deciding: coding error, ineligible case, or genuine extreme

Branch one: the value cannot exist. A 7 on a 1-to-5 Likert item, an age of 199, a DASS-21 subscale total of 78 when the ceiling is 42, a reaction time of 12 ms. There is no methodological debate here: go back to the original file or the paper questionnaire, correct it if the true value can be reconstructed, and if it cannot, set it to missing. That is data cleaning and it is reported as such, no permission required. Watch one detail people forget, though: by flagging that value as missing you have created missing data, and you now owe the reader an account of how you handled it.

Branch two: the number is valid but the case does not belong to the population you defined. Someone who answered 21 items in 14 seconds, a straight-line response pattern running down an entire questionnaire, a participant who reported not understanding the task, a Stroop trial with a latency below 200 ms because the finger went early. These can be excluded, and often should be, on two strict conditions: the rule must be written in advance and it must be independent of the outcome variable. A rule that looks at completion time is legitimate; a rule that looks at the score on your dependent variable is not, however elegantly it is phrased.

Branch three: it is a genuine extreme case. This is the default branch and the one that gets treated worst. In clinical and health psychology the extreme participant is usually the clinically meaningful one: the person with the most severe symptoms, the adolescent with eleven hours of daily screen time, the patient who does not respond to treatment. Removing them is not cleaning your data, it is deleting the phenomenon and artificially narrowing the very variance you set out to study. If your measure has a ceiling and your sample is clinical, those cases are also carrying a large share of your usable range.

The test for telling which branch you are on is uncomfortable and it works every time: would you have removed this case if it had pushed the result the other way? If the honest answer is no, the case stays. And if what worries you is a handful of genuine cases distorting the test, the answer is not the bin, it is the next section.

Acting: retain, winsorise, exclude, or go robust

Retaining and reporting the analysis twice is the default and the option that generates fewest questions. Run the analysis with every case and again without the flagged ones, and report both. If the conclusion does not change, one sentence closes the matter permanently; if it does change, that is the finding and it needs telling, not hiding. It is the same logic as a sensitivity analysis in any meta-analysis, and it takes ten minutes. To see quickly whether the effect really moves, put the means and standard deviations of both versions into the effect size calculator and compare the d values: the shift shows up there long before it shows up in the p.

Winsorising trims rather than deletes: the extreme value is replaced by the most extreme value still inside your limit, typically the 5th and 95th percentiles. It preserves your sample size and is perfectly defensible, but it carries a cost you have to declare, because it changes the standard deviation and with it your standard errors and confidence intervals. If you winsorise, the descriptives you report must be those of the winsorised variables rather than the originals, and you must state what percentage of observations were affected and on which measures. The classic inconsistency is an untouched descriptives table sitting next to tests computed on trimmed data.

Going robust demands the fewest arbitrary decisions, because nobody is removed — the estimators themselves downweight the tails. Twenty per cent trimmed means with Yuen's test, winsorised or percentage-bend correlations, MM-estimation in regression, or simply bootstrap confidence intervals with 5,000 resamples and bias-corrected accelerated adjustment, which assume nothing about shape. In R they live in the WRS2 package (yuen(), pbcor()) and in MASS::rlm(). In the running example, Welch's t test on all cases gave t(238.42) = 2.87, p = .004, d = 0.37, and Yuen's test on 20% trimmed means gave t(138.2) = 3.02, p = .003: the same message, and now nobody can claim the effect rests on four people.

Excluding is the last resort and the only route that obliges you to account for yourself in writing. If you exclude, the manuscript needs the a priori rule, the number of cases, the percentage of the initial sample and a final N that agrees with everything else in the paper. “Outliers were removed” with no figures is the phrasing most likely to trigger a desk return at journals with a methods checklist. One technical warning too: if you delete a single value rather than the whole case, you have solved nothing and created missing data, and that participant now enters some analyses and not others, so your N drifts from table to table with no explanation on offer.

What you never do, on any of the four routes, is transform the variable to make the outliers go away and say nothing about it. A log transformation changes the interpretation of every coefficient and turns a difference of means into a ratio. It can be entirely justified by the shape of the variable, but justifying it by its effect on a few specific cases will not survive review, and the B you report no longer means what your reader thinks it means.

Wording you can lift straight into your Method

These belong in the data-analysis subsection of the Method, in the past tense, written before any result exists. The count of affected cases goes later, in the preliminary-analyses block of the Results or in the participant flow diagram. Change the numbers and the variable names; the structure is what does the work.

If you declare the rule and nothing turns up. “Prior to the main analyses, data were screened for out-of-range values and extreme cases, defined a priori as |z| > 3.29 on individual variables and as a Mahalanobis distance significant at p < .001 across the set of predictors. No impossible values were identified and no case met both criteria, so all analyses were conducted on the full sample (N = 243).”

If you retain genuine extreme cases. “Two cases exceeded the multivariate criterion (maximum D² = 19.02; critical χ² with 3 degrees of freedom = 16.27). Inspection of their records revealed no coding errors and no breach of the eligibility criteria, and influence diagnostics remained within conventional limits (maximum Cook's D = 0.11; no DFBETAS above 0.13). Both cases were retained. Re-running the model without them produced equivalent estimates (β = .26 versus β = .29), and the model based on the full sample is reported.”

If you exclude for error or ineligibility. “Five participants (2.1% of the initial sample) were excluded under criteria set before analysis and unrelated to the outcome variables: three for a total completion time below 120 seconds and two for an invariant response pattern across all three questionnaires. A sixth participant returned an out-of-range score on the anxiety subscale, which was recoded as missing after the original response sheet was checked. The final sample was N = 238.”

If you winsorise. “Values above the 95th and below the 5th percentile of each scale were winsorised to those percentiles, a procedure specified in the analysis plan. Winsorising affected 4.3% of observations. All descriptive statistics and inferential tests reported below were computed on the winsorised variables.”

If you move to robust estimators. “Given the presence of genuine extreme cases that we did not consider it justifiable to exclude, mean comparisons were repeated using Yuen's test on 20% trimmed means, and confidence intervals were estimated by bootstrapping with 5,000 resamples and bias-corrected accelerated adjustment. Both sets of results are reported; the conclusions did not differ.”

One final check that costs a minute and heads off the most tedious review comment there is: make sure the N stated in the Method is the same one heading your descriptives table, the same one implied by the degrees of freedom in your tests, and the same one in your abstract. When a case gets removed late in the process, that figure stops agreeing in two places out of five, and two places out of five is exactly how it gets noticed.

Already drafted it? Paste your Results section into Review one section and you get the first objection a reviewer would raise, quoting the exact sentence from your text. Free, no sign-up.

Frequently asked questions

When is a data point considered an outlier?

It depends on the criterion you declared, and there are three common ones. On a single variable, a standardized score exceeding 3.29 in absolute value (a two-tailed probability of .001) or Tukey's boxplot fences at 1.5 and 3 times the interquartile range. Across several variables, Mahalanobis distance tested against a chi-square with as many degrees of freedom as variables at an alpha of .001. These criteria frequently disagree with each other, so you pick one before looking at the data and apply it identically to every variable.

Can you remove outliers from your data?

You can remove values that are demonstrable errors, and cases that breach an exclusion rule written before analysis and unrelated to the outcome variable. A genuine extreme case is not removed simply for being extreme; in clinical samples it is often the most informative participant you have. If you do exclude, the Method must state the rule, the number of cases, the percentage and the final N, and it is wise to show the analysis with nobody excluded as well.

How do you report outliers in APA 7 style?

In two places. The data-analysis subsection of the Method carries the criterion you intended to use, the threshold, and what you would do if cases appeared. The preliminary-analyses block of the Results, or the participant flow diagram, carries how many cases were detected, what happened to them and the final N. Give the figures with their index: maximum Cook's D = 0.11, maximum Mahalanobis D2 = 19.02 against a critical chi-square of 16.27 with 3 degrees of freedom.

What is Mahalanobis distance used for?

It measures how far each case sits from the center of the multivariate cloud while accounting for the correlations between variables, so it catches impossible combinations that no single variable reveals: a 19-year-old with 30 years of work experience is unremarkable on age and on experience, but not as a pair. Save it from the Save tab of linear regression in SPSS or use mahalanobis() in R, then compare D2 with a chi-square on as many degrees of freedom as variables entered, at an alpha of .001.

What is the difference between an outlier and an influential case?

An outlier sits far from the rest; an influential case changes the model's estimates when you remove it. They are different things measured differently: distance with z scores or Mahalanobis, influence with Cook's D, leverage and DFBETAS. A case can have z = 4.1 and a Cook's D of 0.03, which means it is far out and moving nothing at all. Influence, not distance, is the index that should drive what you actually do.

What does winsorising do and when should I use it?

Winsorising replaces an extreme value with the most extreme value still inside your chosen limit, usually the 5th and 95th percentiles, instead of deleting the case. It suits situations where you want to keep your sample size and have no grounds for excluding anyone. The cost is that it changes the standard deviation and therefore your standard errors and intervals, so the descriptives you report must be those of the winsorised variables and you must state what percentage of observations were affected.

With the rule declared in the Method, the count and percentage in the Results, a clear separation between extreme and influential, and the analysis reported both ways, your outliers stop being an exposed flank and start reading as evidence that you know what you are doing. If the case worrying you is one of the genuinely influential ones, and you are unsure whether it calls for robust estimators, a different model, or simply a line in the limitations, we can work through it together in my statistical consulting.