Every real dataset has holes. A sensor drops out, a user skips a form field, a join finds no match — and suddenly a column is peppered with NaN. What you do about those holes silently decides your averages, your model, and your conclusions. This lesson is about making that decision on purpose, not by accident.
What you'll learn
What
Missing data is any absent value in a DataFrame, represented in pandas as NaN (Not a Number) for numeric data or NaT for dates. Handling it means detecting where it is, understanding why it's there, and choosing to drop it or fill it.
Why
Missing values break arithmetic, skew statistics, and crash many models. But the wrong fix is worse than the problem: dropping too much throws away signal, filling carelessly invents data. The choice is a judgement call with real consequences.
Where it's used
Cleaning survey responses, reconciling merged tables, preparing model features, and computing trustworthy KPIs all begin with a missing-data decision.
Where this runs in production
Vehicle telemetry occasionally drops a reading. Engineers forward-fill short gaps in time-series signals so downstream models see a continuous stream, but flag long gaps for review rather than inventing data.
Hosts leave fields like 'cleaning_fee' blank. Analysts fill these with 0 where blank genuinely means 'none', but treat a missing 'review_score' very differently — dropping or modelling it rather than assuming zero.
Patient records with missing measurements are audited by missingness pattern before any imputation, because whether data is missing at random changes which statistical methods are valid.
import pandas as pd
import numpy as np
df = pd.DataFrame({'name': ['Amara','Jamal','Priya'],
'age': [28, np.nan, 42]})
print(df.isna()) # boolean DataFrame: True where missing
print(df.isna().sum()) # count of missing per columndf.isna() returns a same-shape DataFrame of True/False. Chaining .sum() counts the True values per column — your standard one-line missingness audit. notna() is the inverse.
A missing value is unknown, not zero. An empty string '' and a 0 are real values, not NaN — isna() returns False for both. Treating a genuine 0 as missing (or a NaN as 0) changes your results. Always confirm what 'empty' means in your source before deciding.
NaN == NaN is False — that's why you use isna(), never == np.nan. And NaN propagates: 5 + NaN is NaN. But most pandas aggregations (mean, sum) skip NaN by default, so df['age'].mean() averages only the present values.
df.dropna() # drop rows with ANY NaN df.dropna(subset=['age']) # drop only where 'age' is missing df.dropna(thresh=2) # keep rows with >= 2 non-null values df.dropna(axis=1) # drop COLUMNS that contain NaN
subset is the most surgical option — it drops rows only when the columns you actually care about are missing, sparing rows that are merely missing something irrelevant.
Think of an emergency room. Dropping a row is discharging a patient — fine if there are plenty of others and this one tells you nothing, costly if patients are scarce. Filling is treatment — a median fill is a safe generic remedy, a forward-fill assumes the condition hasn't changed since the last reading. You choose based on how much data you have and how much you trust the assumption.
Click each node to walk the decision from detecting a gap to choosing a treatment.
You just loaded a customer table and want to know where the gaps are.
Two columns have one missing value each.
Your task
A survey dataset arrives with gaps. Audit it, then clean each column according to what its missingness means.
2 1 [34.0, 31.0, 28.0, 31.0] [0.0, 120.0, 0.0, 45.0]
Write your solution in the editor on the right, then hit Run.
Which expression counts missing values in each column?
Why does `df['x'] == np.nan` fail to find missing values?
What does `df.dropna(subset=['email'])` do?
What's the concern with a median fill here?
Given the ordered daily readings below, forward-fill the missing values and print the mean of the filled column rounded to 1 decimal. df = pd.DataFrame({'reading':[10.0, None, None, 16.0, 20.0]})
You have an int column [1, 2, 3]. You set the middle value to NaN. What is the column's dtype afterwards?
How do you detect missing values in a DataFrame, and why can't you just check for equality with NaN?
I use isna() — or its inverse notna() — usually as df.isna().sum() to get a per-column count of gaps, which is my first read on any new dataset. I can't check df['x'] == np.nan because NaN is defined to be unequal to everything, including itself, so that comparison always returns False. isna() is built to recognise the missing markers — NaN for numbers, NaT for datetimes, None for objects — regardless of that equality quirk. Once I know where and how much is missing, I can decide how to handle it.
How do you decide between dropping and filling missing values?
It comes down to how much data the gap represents and why it's missing. If only a small fraction of rows are affected and the missingness looks random, dropping — ideally with subset so I only remove rows lacking a column I actually need — is simple and low-risk. If dropping would cost too much data, or the column is important, I fill: a constant like 0 when blank has a real meaning, a median or mean for roughly random numeric gaps, or forward/backward fill for ordered time series where a value persists until it changes. The reason matters as much as the amount — for data that's missing not at random, like high earners skipping an income field, both dropping and naive imputation bias the result, so I'd flag the missingness or model it instead. There's rarely a universally correct choice; I pick the option whose assumptions I can defend and I document it.
Walk me through the MCAR, MAR, and MNAR distinction and how it changes your handling strategy.
These describe the mechanism behind missingness. MCAR — missing completely at random — means the gap is unrelated to any variable, observed or not; here dropping is unbiased, just wasteful, and simple imputation is fine. MAR — missing at random — means missingness depends on other observed columns but not the missing value itself; for example a device that fails to log at night, where the gap correlates with time. Because the dependence is on observed data, I can impute conditionally, using models or group-wise fills that borrow strength from the correlated columns. MNAR — missing not at random — means the missingness depends on the unobserved value, like sicker patients dropping out of a trial or high earners hiding income; here every naive method biases results, so I either add an explicit missingness indicator so a model can learn the pattern, use methods that model the missingness jointly, or go back for better data. In practice I can't always prove which regime I'm in, but reasoning about it stops me from applying a clean-looking median fill that quietly corrupts the analysis.
Common Mistakes to Avoid
1) Checking for missing with == np.nan — NaN never equals anything; use isna(). 2) Treating NaN, 0, and '' as the same thing — only NaN is missing; 0 and empty string are real values. 3) Calling a blanket dropna() and silently deleting most of your rows — prefer subset to target the columns that matter. 4) Filling with the mean when the column has outliers — the median is usually the safer numeric default. 5) Forward-filling unordered data — ffill only makes sense when rows are in a meaningful sequence like time. 6) Imputing MNAR gaps with a statistic and biasing exactly the group you care about — flag or model the missingness instead.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'ELI5: what does NaN actually mean in a dataset?' • 'Show me when ffill is right and when it's dangerous.' • 'Quiz me on dropna options: how, thresh, subset.' • 'Explain MCAR vs MAR vs MNAR with a fresh example.' • 'Interview mode: ask me how I'd decide to drop or fill a 30%-missing column.'
Glossary
NaN — Not a Number; pandas' marker for a missing numeric value. NaT — the missing marker for datetimes. isna() / notna() — detect missing / present values. dropna() — remove rows or columns containing NaN; controlled by how, thresh, subset, axis. fillna() — replace NaN with a constant or statistic. ffill / bfill — carry the last / next valid value across a gap. Imputation — filling missing values with estimated ones. MCAR / MAR / MNAR — the three missingness mechanisms that decide which fixes are valid. Missingness indicator — a flag column marking which rows were originally missing.
Recommended Resources
• Docs: 'Working with missing data' in the pandas User Guide — the authoritative reference for isna, dropna, fillna, and interpolation. • Read: a short primer on Rubin's MCAR/MAR/MNAR taxonomy to ground your imputation choices. • Practice: take a dataset, audit it with isna().sum(), then clean three columns three different ways — drop, median-fill, and a missingness flag — and compare how each changes the column mean. • Next in DSM: you can handle gaps within a table — next you'll order and rank data with Sorting & Ranking, surfacing the top and bottom of any column.
Recap
✓ pandas marks missing values as NaN (or NaT for dates); detect them with isna(), never == np.nan. ✓ df.isna().sum() is your one-line missingness audit. ✓ NaN is not zero and not an empty string — confirm what 'empty' means before acting. ✓ dropna() removes gaps; use subset to target only the columns that matter. ✓ fillna() imputes — constant, median/mean, or ffill/bfill for ordered series. ✓ The reason data is missing (MCAR/MAR/MNAR) decides whether dropping or filling is safe. Next up: Sorting & Ranking. Your data is complete and trustworthy — next you'll order it, rank it, and pull the top and bottom performers out of any column.
Run your code to see the output here.