In 2010 a trader's 'fat finger' briefly wiped almost a trillion dollars off US markets — one extreme data point among billions. Delete it as noise and you erase the most consequential event of the day. Keep it unquestioned in your averages and every summary is distorted. Outliers force the one question statistics can't answer for you: is this a mistake, or is this the story?
What you'll learn
What
Outlier detection is finding values that lie unusually far from the rest: the IQR rule (beyond 1.5×IQR outside the quartiles — what box plots draw), z-scores (beyond ~3 standard deviations), and their treatments — investigate, fix, remove, cap (winsorize), or deliberately keep.
Why
Means, standard deviations, correlations, and regression are all outlier-sensitive: a single wild value can move them arbitrarily far. But outliers are also fraud, failures, and breakthroughs. Detection without judgement deletes discoveries; no detection at all reports distorted numbers.
Where it's used
Fraud detection, sensor-fault screening, quality control, financial risk, and as a standard pre-modelling step — plus every EDA, where the extremes are often the first thing worth explaining.
Where this runs in production
A card that usually spends $60 suddenly charging $4,000 abroad is an outlier against that customer's own history — flagged in milliseconds. Here outliers aren't cleaned away; they're the product.
Battery telemetry showing an impossible temperature spike may be a failing sensor (drop the reading) or a genuine thermal event (never drop it). Engineering pipelines separate physically-impossible from merely-extreme before any filtering.
A hotel accidentally listed at €1 per night would dominate 'best deal' rankings. Listing pipelines cap or quarantine price outliers so one typo doesn't distort an entire market's results.
q1 = df['amount'].quantile(0.25) q3 = df['amount'].quantile(0.75) iqr = q3 - q1 lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr outliers = df[(df['amount'] < lower) | (df['amount'] > upper)] print(len(outliers), lower, upper)
Five lines you'll write hundreds of times. Because quartiles barely move when extremes change, the fences themselves are robust — wild values can't hide the fence that should catch them.
z = (x − mean) / std measures how many standard deviations a value sits from the mean; |z| > 3 is the usual flag. It's principled for roughly normal data (99.7% lies within ±3σ) — but both the mean and std are themselves dragged by outliers, so a huge outlier inflates the std and can mask itself and others. This 'masking' is why z-scores mislead on contaminated or skewed data.
z = (df['amount'] - df['amount'].mean()) / df['amount'].std() flagged = df[z.abs() > 3]
Two lines, but read the caveat above: on skewed data (income, prices, durations) the mean and std don't describe the bulk of the data, so the flags don't either. Default to IQR unless you know the distribution is roughly symmetric.
1) Investigate: is it physically/logically possible? An age of 250 is an error; a $2M order might be real. 2) Fix: a misplaced decimal or unit mix-up (cm vs m) has a recoverable true value. 3) Remove: only for confirmed errors you can't fix — and count what you removed. 4) Cap (winsorize): clip to the fence or a percentile when extremes are real but would dominate — df['x'].clip(lower, upper). 5) Keep: real, important extremes stay — and you switch to robust statistics (median, IQR) or report with-and-without.
Every removed point changes your conclusions, and 'it was far from the mean' is not evidence of error. The discipline: separate impossible (violates known constraints — negative quantities, ages over 130) from implausible (extreme but possible). Impossible values are data errors; implausible ones require investigation. When in doubt, run the analysis both ways and report both — if conclusions flip on a handful of points, that fragility IS the finding.
An outlier detector is a smoke alarm: it tells you where smoke is, not whether it's burnt toast or a house fire. Silencing every alarm (deleting outliers) means never finding fires (fraud, failures, discoveries); treating every alarm as a fire (keeping everything unexamined) distorts daily life (your summary statistics). The alarm's job is to trigger investigation — never to make the decision.
Click each node — detection triggers investigation, and only investigation picks the treatment.
One wild value versus the mean and the median.
Five typical values and one extreme. Watch what each summary does with it.
Your task
Session durations (minutes) include some extremes. Compute the IQR fences, count the outliers, and compare the mean before and after capping at the upper fence.
1 39.3 17.9
Write your solution in the editor on the right, then hit Run.
Under the IQR rule, a value is an outlier when it lies:
The mean of a column is 875 while its median is 52. What does this gap most likely signal?
Why can z-scores fail to flag a huge outlier that the IQR method catches?
What's the right treatment?
Using the IQR rule, print the outlier values in the Series, then print the median of the data with outliers excluded. s = pd.Series([200, 220, 210, 215, 225, 205, 1500, 230])
A row has height 2.10 m and weight 50 kg — each value passes its own column's IQR check. Which statement is correct?
Compare the IQR rule and z-scores for outlier detection. Which do you default to and why?
Both measure distance from the centre, but with different yardsticks. Z-scores use the mean and standard deviation: principled for roughly normal data, where |z|>3 flags the outermost 0.3%. Their weakness is that the yardstick itself is outlier-sensitive — a wild value inflates the std and can shrink every z-score below threshold, masking itself. The IQR rule uses quartiles, which barely move when extremes change, so the fences stay honest on skewed or contaminated data — and business data (revenue, durations, counts) is almost always right-skewed. So I default to IQR: it's the box-plot rule, it needs no distributional assumption, and it resists masking. I reach for z-scores when the data is genuinely near-normal or when the audience expects sigma-based language, and for anything multivariate I move past both to Mahalanobis distance or Isolation Forest, since per-column fences can't see anomalous combinations.
You've flagged outliers — walk me through deciding what to do with them.
The flag starts an investigation; it never decides. First I split impossible from implausible using domain constraints: negative quantities, ages over 130, dates in the future are errors by definition. For those I try to fix before removing — misplaced decimals and unit mix-ups have recoverable true values — and anything I do remove gets counted and documented. For the merely extreme, I ask two questions: is it real, and does it distort what I'm reporting? A genuine $2M corporate order stays; if it would dominate a mean, I either switch to robust statistics like the median, cap at the fence with clip (winsorizing — disclosed, since it biases the data toward the centre), or report the analysis with and without it. That last one matters most: if the conclusion flips on a handful of points, the fragility is itself the finding. What I never do is silently delete whatever fell outside the fence — in fraud, failures, and demand spikes, the outliers are frequently the entire point of the analysis.
How do outliers interact with the rest of the cleaning pipeline, and why treat them last?
Outlier detection assumes the values it measures are real, so every earlier cleaning step protects it from false alarms. Wrong types make it impossible — you can't fence a price column stored as strings. Unfixed structural issues create fake extremes: a unit mix-up where half the heights are in centimetres puts a cluster of 'outliers' at 100× scale; a misparsed CSV can shift columns entirely. Duplicates distort the quartiles that define the fences, and unimputed sentinel values like -999 for 'missing' show up as a spike of phantom outliers that are really nulls in disguise — a classic trap. So the order from the audit lesson holds: structure, types, duplicates, missing values, category consistency, and only then outliers, when every extreme value that remains is at least a genuine measurement. Treating them earlier means investigating artefacts of dirt rather than facts about the world — wasted effort at best, deleted real data at worst.
Common Mistakes to Avoid
1) Deleting whatever the fence flags — detection is an alarm, treatment needs investigation. 2) Using z-scores on skewed data and missing outliers through masking. 3) Forgetting sentinel values: -999 'missing' codes masquerade as outliers when they're really nulls. 4) Detecting outliers before fixing types, units, and duplicates — you'll chase artefacts. 5) Capping or removing without disclosure — winsorizing changes distributions and must be reported. 6) Checking columns only individually — plausible values can be jointly impossible (multivariate outliers). 7) Reporting only the cleaned result when conclusions differ with and without the extremes — that difference is the finding.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'ELI5: the smoke-alarm view of outlier detection.' • 'Show me masking: build a dataset where z-scores miss what IQR catches.' • 'Quiz me on choosing treatments: fix, remove, cap, or keep.' • 'Explain Mahalanobis distance for the height-weight example.' • 'Interview mode: defend keeping an extreme value in a revenue report.'
Glossary
Outlier — a value unusually far from the bulk of the data. IQR — interquartile range, Q3 − Q1, the span of the middle half. IQR rule — flag outside Q1 − 1.5×IQR / Q3 + 1.5×IQR (the box-plot rule). Z-score — (x − mean)/std, distance in standard deviations. Masking — an outlier inflating the std enough to hide itself from z-scores. Winsorizing / capping — clipping extremes to a fence or percentile (.clip). Robust statistic — one that resists outliers (median, IQR). Sentinel value — a code like -999 standing in for missing. Multivariate outlier — a row anomalous in the combination of its values.
Recommended Resources
• Docs: pandas quantile/clip references; scikit-learn's outlier detection guide (IsolationForest, LocalOutlierFactor) for the multivariate tier. • Read: any explainer of the box plot's construction — the 1.5×IQR whisker convention comes from Tukey's exploratory data analysis tradition. • Practice: take a real sales dataset, compare mean vs median per column, fence with IQR, and write one sentence per flagged value: fix, remove, cap, or keep — and why. • Next in DSM: cleaning is complete. The Data Transformation module begins with the most-used tool in analytics: GroupBy & Aggregation.
Recap
✓ IQR rule: fence at Q1 − 1.5×IQR and Q3 + 1.5×IQR — robust, assumption-free, the box-plot standard. ✓ Z-scores suit near-normal data but suffer masking: outliers inflate the very std that judges them. ✓ A mean far from its median is itself an outlier alarm. ✓ Treatments in order: investigate → fix → remove (counted) → cap with clip → keep with robust stats or with-and-without reporting. ✓ Separate impossible (constraint violations — always errors) from implausible (extreme but maybe real — often the story). ✓ Univariate fences miss anomalous combinations — that's the multivariate tier (Mahalanobis, Isolation Forest). Next up: GroupBy & Aggregation — the Data Cleaning module is done, and Data Transformation begins with split-apply-combine, the pattern behind almost every business metric you've ever read.
Run your code to see the output here.