Two datasets can have the same 12% missing values and demand opposite treatments. In one, sensors dropped readings at random — fill with the median and move on. In the other, high earners skipped the income question — fill with the median and you've just erased the most important signal in the data. Handling nulls well starts with asking why they're missing, not how many there are.
What you'll learn
What
This lesson covers the mechanics of finding nulls (isna, notna, per-column and per-row counts), the three missingness mechanisms — MCAR (missing completely at random), MAR (missing at random, explained by other columns), and MNAR (missing not at random, related to the hidden value itself) — and the treatment menu: dropping rows or columns, constant/statistic imputation, group-based imputation, and flagging.
Why
Every treatment embeds an assumption. Dropping rows assumes the remaining rows still represent the population; median-filling assumes the missing values look like the observed ones. If the mechanism contradicts the assumption, the 'cleaned' data is quietly biased — and no downstream model can undo that.
Where it's used
Every pipeline that feeds a model or dashboard has a null-handling step. Feature engineering, survey analysis, sensor processing, and financial reporting all live or die by whether missingness was diagnosed before it was patched.
Where this runs in production
New listings have no review scores — missingness that is structural, not random. Imputing the city average would make untested listings look average, so models carry an explicit 'no reviews yet' flag instead.
Patients who leave a trial early often do so because of side effects — the missing outcome is related to the outcome itself (MNAR). Regulators require analyses that model the dropout mechanism rather than pretend the data is complete.
Most user-movie ratings are missing, and not at random — people rate what they chose to watch. Recommendation models are built around this MNAR structure instead of filling the gaps with averages.
df.isna().sum() # count per column df.isna().mean().round(3) # fraction per column df.isna().sum(axis=1) # count per row df[df.isna().any(axis=1)] # show the incomplete rows
Four views of the same mask. The fraction view is the one to report — '3 nulls' means nothing without knowing if the table has 10 rows or 10 million.
MCAR — missing completely at random: the gap has nothing to do with any data (a sensor glitch). MAR — missing at random: the gap is explained by OTHER observed columns (younger users skip the income field). MNAR — missing not at random: the gap depends on the missing value itself (high earners hide their income). Treatment safety degrades in that order: MCAR is safe to drop or fill, MAR needs group-aware fills, MNAR can't be fixed from the data alone — it must be modelled or flagged.
# Does income-missingness depend on age group?
df['income_missing'] = df['income'].isna()
print(df.groupby('age_band')['income_missing'].mean())
# 18-25 0.42 <- concentrated here: NOT completely at random
# 26-40 0.08
# 41-65 0.05One groupby answers 'is this MCAR?'. A flat profile across groups supports random missingness; a spike in one group refutes it and tells you which grouping a fill should respect.
1) dropna(rows) — safe when missingness is MCAR and rare (<~5%). 2) Drop the column — when it's mostly empty and low-value. 3) fillna with median/mode — numeric skew-safe default for MCAR. 4) Group-based fill — transform('median') within a segment, for MAR. 5) Flag + fill — add a was_missing indicator column so the model keeps the signal. 6) Interpolation/forward-fill — for ordered time series only.
If you compute a fill value (mean, median) from the whole dataset and then split into train and test, the test rows influenced the fill used in training — a data leak. In modelling pipelines, always fit imputation on the training set only and apply it to the test set. For pure reporting this doesn't apply, but the habit matters the moment a model is involved.
A pothole you just fill and drive on. A crime scene you investigate first: who was here, why is the evidence gone, was it removed deliberately? Treating every null like a pothole — pour median in, smooth it over — destroys the evidence exactly when the absence itself was the most informative thing in the room. Ask why it's missing before you decide what to pour in.
Click each node to follow the decision path from detection to treatment.
Report how much of each column is missing, as a fraction.
Column a has 2 of 4 values missing; b is complete.
Your task
A delivery dataset has missing ratings. Quantify the gaps, check whether they concentrate in one city, and fill using each city's own median.
0.5
{'Goa': 0.67, 'Pune': 0.33}
[4.0, 4.5, 5.0, 3.0, 3.0, 3.0]Write your solution in the editor on the right, then hit Run.
df.isna().mean() returns:
Survey respondents with very high incomes tend to leave the income field blank. This missingness is:
Why is the median usually preferred over the mean for filling a skewed numeric column?
What is the main problem with this plan?
Add a 0/1 column 'score_missing' flagging rows where 'score' is null, then fill 'score' with its median. Print the DataFrame's score and score_missing columns as lists. df = pd.DataFrame({'score':[80.0, None, 60.0, None]})
In a modelling pipeline, why must imputation values be computed from the training set only?
Explain MCAR, MAR, and MNAR with an example of each, and how the mechanism changes your treatment.
MCAR means the gap is unrelated to anything — a sensor randomly dropping readings. There, dropping rows or filling with a median is safe because the missing values resemble the observed ones. MAR means the gap is explained by other observed columns — younger users skip the income question. A global fill would bias the young segment, so I fill within the explaining group, for example a per-age-band median via groupby-transform. MNAR means the gap depends on the hidden value itself — high earners hiding income. No fill computed from observed data is honest there, because the missing values are systematically different from what I can see; I keep an explicit missing-flag, consider modelling the mechanism, and disclose the bias. The practical point: the same 10% missing can be harmless or fatal depending on mechanism, so I always probe before I patch.
How would you check, with pandas, whether missingness in one column is random?
I build the missingness indicator — df['col'].isna() — and study it like any other variable. First I group its mean by candidate explanatory columns: df['col'].isna().groupby(df['segment']).mean(). If the missing rate is flat across segments, randomness is plausible; if it spikes in one group, it's at least MAR and that grouping must drive the fill. For numeric relationships I compare the distribution of another column between missing and non-missing rows, for example df.groupby(df['col'].isna())['age'].describe(). What I cannot fully test from the data alone is MNAR, since it depends on values I never observed — that diagnosis comes from domain knowledge about how the data was collected, which is why I always ask how a field gets populated before trusting any statistical check.
When would you drop rows with nulls versus impute, and what risks does each carry?
Dropping is defensible when the affected rows are a small share, missingness looks MCAR, and I can afford the sample loss — it keeps every remaining value real. Its risks are losing statistical power and, if the missingness is not random, silently biasing the sample: dropping all rows missing income removes exactly the segment that skips that question. Imputation preserves rows and works for MAR when done group-aware, but every imputed value is invented data — it shrinks variance, can distort correlations, and a careless global fill bakes in bias. Column-dropping is the third option when a field is mostly empty and low-value. In modelling contexts I add two safeguards regardless of choice: an indicator column so the fact of missingness stays available as signal, and fitting any fill statistic on the training split only so evaluation stays honest.
Common Mistakes to Avoid
1) Reporting null counts without fractions — 300 nulls is noise in 10M rows and a crisis in 1,000. 2) Filling before asking why values are missing — the mechanism decides the method. 3) Using the mean on skewed columns; the median resists outliers. 4) Global fills on MAR data — fill within the explaining group via transform. 5) Flagging missingness AFTER filling — the mask is all False by then; create the indicator first. 6) Computing fill statistics on the full dataset in a modelling pipeline — fit on train only. 7) Forgetting that np.nan != np.nan — always detect with isna(), never with == comparisons.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'ELI5: MCAR vs MAR vs MNAR with a school-attendance example.' • 'Show me how transform("median") fills within groups step by step.' • 'Quiz me: give me missingness scenarios and I'll name the mechanism.' • 'Why does imputing before a train/test split leak information?' • 'Interview mode: challenge my choice to drop rows with nulls.'
Glossary
isna()/notna() — boolean masks for missing/present values. Missingness mechanism — the process that decides which values go missing. MCAR — missing completely at random, unrelated to any data. MAR — missing at random, explained by other observed columns. MNAR — missing not at random, dependent on the hidden value itself. Imputation — replacing missing values with estimates. Group-based fill — imputing with a statistic computed within each row's group (groupby + transform). Missing-flag / indicator — a 0/1 column recording where values were missing. Data leakage — test-set information influencing training, e.g. via a full-dataset fill statistic.
Recommended Resources
• Docs: pandas 'Working with missing data' user guide — the canonical reference for isna, fillna, and interpolate. • Read: the MCAR/MAR/MNAR taxonomy traces to Rubin (1976); any short summary of 'missing data mechanisms' covers it. • Practice: take the Titanic dataset, probe whether Age-missingness depends on Pclass, then compare a global median fill against a per-class fill. • Next in DSM: with gaps handled, the next quality issue is rows that appear twice — Deduplication.
Recap
✓ Measure missingness as fractions per column (isna().mean()) and inspect per-row completeness too. ✓ MCAR is random, MAR is explained by other columns, MNAR depends on the hidden value itself. ✓ Probe the mechanism by grouping the isna() indicator by other columns before choosing a fix. ✓ Median/mode fills suit MCAR; groupby-transform fills respect MAR; MNAR demands flags and honesty, not fills. ✓ Create was_missing indicators BEFORE filling, and fit fill statistics on training data only. ✓ Every treatment embeds an assumption — document what you filled and why. Next up: Deduplication. Nulls are gaps; duplicates are the opposite — the same fact counted twice. You'll learn to find and remove them without deleting real data.
Run your code to see the output here.