A dataset can be perfectly readable, perfectly structured — and still be wrong. A blank cell, a customer entered twice, a date written five different ways, a price of 9,999,999: none of these break the table, but every one of them quietly poisons an average, a count, or a chart. Professionals never trust data on sight. Before any analysis, they run a quality audit — a quick, systematic hunt for the four flaws that fool everyone else. This lesson teaches you that audit.
What you'll learn
What
Data quality is how much you can trust a dataset. A quality audit checks four things: missing values (blank cells), duplicates (the same thing recorded twice), inconsistencies (the same fact written different ways), and outliers (values far outside the normal range).
Why
Analysis inherits the flaws of its data. 'Garbage in, garbage out' is the oldest rule in the field: a flawless model trained on dirty data produces confident, wrong answers. Most of a data scientist's time goes into finding and fixing these flaws — because it's the difference between an insight and a mistake.
Where it's used
The first thing anyone does with a new dataset, in every job. Loading a CSV, cleaning a survey, preparing data for a dashboard or a model — all begin with a quality check.
Where this runs in production
If the same apartment is uploaded twice, average-price stats for that area count it twice. Airbnb's data teams deduplicate listings before publishing any market figures.
Survey questions people skip create missing values. Whether those blanks are ignored or estimated can shift funding decisions worth millions, so statisticians handle them explicitly.
A $2,000,000 coffee purchase is an outlier. Fraud systems must decide, in milliseconds, whether it's a real anomaly worth blocking or a data-entry error worth ignoring.
Missing values — a cell has no data (blank, NaN, 'N/A', or a fake stand-in like 0 or -1). Duplicates — the same real-world thing recorded in more than one record. Inconsistencies — the same fact written different ways ('NY' vs 'New York', kg vs lb, '5' vs 'five'). Outliers — values far outside the normal range, which may be errors or genuine rare cases.
Imagine a stack of paper forms where some people left 'age' blank. You have three honest choices: leave it blank and count around it, guess a sensible value (say the average age), or throw that form out. What you must never do is treat a blank as if it were a zero — that would tell you people are newborns. The same three choices, and the same trap, apply to missing cells in a dataset.
Two rows sharing a city or a price are fine — different customers can live in the same city. A duplicate is when the whole record (or the key) repeats, meaning the same real-world thing is counted twice. Always check against the key and granularity from the last lesson: two rows with the same key are a genuine duplicate; two rows that merely share some feature values are not.
country
USA
United States
usa
U.S.
# To a computer these are FOUR different values.
# A groupby('country') would report four countries, not one.Every row here means the United States, but because the text differs, software treats them as four distinct categories. Standardising the values (all to one spelling) before counting is the fix — and a classic cleaning step.
Click each check to see what to look for and how to fix it. Run all four on any new dataset before you trust a single number.
A small table of survey responses has an empty age cell.
Luc's age cell is blank — a missing value. It's not zero and not any number; it's the absence of data, and it must be handled deliberately rather than ignored.
One missing value (Luc's age). Decide: count around it, estimate it, or drop the row.
Your task
You're given a list of ages where missing values were stored as None. Complete two checks: count the missing values, and compute the average using only the real ages (ignoring the missing ones). Run it to see a small, honest audit in action.
Missing: 2 Average of real ages: 35.5
Write your solution in the editor on the right, then hit Run.
Which of these is a 'disguised' missing value?
Two rows in a table share the same city but have different customer_ids (the key). Is this a duplicate?
A column of countries contains 'Spain', 'spain', and 'ESP'. What kind of data-quality problem is this?
What's the best response?
Standardise inconsistent category text, then count distinct values. Given a list of country strings in mixed case, lowercase them all and print how many distinct countries remain. Hint: use .lower() and set().
Why is this risky?
How do you handle missing values in a dataset?
First I find them all — not just blanks and NaN, but disguised ones like a 0 or -999 standing in for 'unknown', by checking whether each column's values are actually plausible. Then I ask why they're missing, because that guides the fix. There are three honest options: exclude the incomplete records from that particular calculation, estimate a sensible fill (for example the column's median, or a value predicted from other features), or, if a column is mostly empty, drop the column. The one thing I never do is fill with 0 when 0 is a real possible value, because that invents false data and drags averages. Whatever I choose, I document it, since how missing data is handled can change the result.
What's the difference between a duplicate and a legitimate repeated value, and how do you tell them apart?
A duplicate is the same real-world thing recorded more than once — it inflates counts and skews averages. A legitimate repeat is two different things that happen to share a feature value, like two different customers in the same city. I tell them apart using the key and granularity: if the key repeats, the same entity was recorded twice and it's a true duplicate to remove; if only some non-key features match but the keys differ, they're distinct records and I leave them alone. In practice I also watch for near-duplicates — the same entity entered slightly differently, like a name with and without a middle initial — which need fuzzy matching rather than an exact-key check.
How do you decide whether to remove an outlier?
I never remove outliers automatically, because an outlier is just a value far from the rest — it can be an error or it can be the most important thing in the data. So I investigate each one and ask whether it's plausible. If it's impossible or clearly a data-entry mistake — a person weighing 5000 kg, an extra zero on a price — I treat it as an error and fix or exclude it, documenting what I did. If it's genuinely real, like a record-breaking sales day or a legitimately huge transaction, I keep it, because deleting it would erase a real signal. And when outliers are present, I lean on robust summaries like the median instead of the mean, since one extreme value can drag the mean a long way but barely moves the median. The judgment call is always driven by domain context, not by a blanket rule.
Common Mistakes to Avoid
1) Filling missing values with 0 when 0 is a real possible value — this invents false data. 2) Missing the disguised blanks (0, -1, -999, 'unknown') because they look like real numbers. 3) Deduplicating on a shared feature instead of the key, which deletes genuinely different records. 4) Counting categories without standardising first, so 'NY' and 'New York' become two places. 5) Deleting every outlier on sight — you may be erasing your most important finding.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'Give me a messy 6-row table and ask me to find all four quality flaws.' • 'ELI5: why does one outlier wreck the mean but not the median?' • 'Show me a column with disguised missing values and see if I spot them.' • 'Quiz me: duplicate or legitimate repeat?' • 'Explain three ways to handle missing data with a fresh example.'
Glossary
Data quality — how much a dataset can be trusted. Missing value — an absent entry (blank, NaN, 'N/A', or a disguised stand-in). NaN — 'not a number', the standard marker for missing numeric data. Duplicate — the same real-world thing recorded in more than one record. Inconsistency — the same fact written in different forms. Outlier — a value far outside the normal range. Mean — the average (sensitive to outliers). Median — the middle value when sorted (robust to outliers). Data cleaning — the process of fixing these flaws.
Recommended Resources
• Concept: search 'garbage in, garbage out' to see why data quality decides everything downstream. • Reference: skim the pandas guide 'Working with missing data' to preview how isna() and dropna() automate today's checks. • Practice: open any spreadsheet you own and run the four-point audit by eye. • Reading: the article 'Tidy Data' by Hadley Wickham (introduced last module) also motivates clean, consistent columns. • Next in DSM: you can now find flaws inside a dataset — next you'll learn about a subtler flaw that lives in how the data was collected: bias.
Recap
✓ Data quality is how much you can trust a dataset — analysis inherits its flaws (garbage in, garbage out). ✓ Missing values include blanks, NaN, and disguised stand-ins like 0 or -999; handle them deliberately, never with a blind 0. ✓ Duplicates are the same real thing recorded twice — detect them with the key, not with shared feature values. ✓ Inconsistencies are one fact written many ways ('NY' vs 'New York'); standardise before counting. ✓ Outliers may be errors or real; investigate before removing, and prefer the median when they're present. ✓ Run the four-point audit — missing, duplicates, inconsistencies, outliers — on every new dataset. Next up: Bias in Data. Even a perfectly clean dataset can lie if it was collected from the wrong people. Next you'll learn to spot selection, survivorship, and sampling bias — flaws no quality audit will ever reveal.
Run your code to see the output here.