The most expensive bugs in data science aren't in the model — they're in the data that fed it. A duplicated order, a price stored as text, a category spelled three ways: none of these throw an error, they just quietly poison your results. This lesson gives you a checklist so the mess never makes it downstream.
What you'll learn
What
Data quality issues are the ways real-world data deviates from what your analysis assumes: missing values, duplicates, wrong types, inconsistent categories, invalid values, structural problems, and outliers. An audit is the systematic first pass that finds them.
Why
Every conclusion rests on the data being what you think it is. Garbage in, garbage out isn't a cliché — it's the single biggest source of wrong analyses. Catching issues early is cheaper than debugging a broken dashboard three weeks later.
Where it's used
The first thing any analyst does with a new dataset is audit it. It's the gate between 'received data' and 'trusted data' in every pipeline, report, and model.
Where this runs in production
Before any pricing analysis, engineers audit trip logs for impossible values — negative fares, zero-distance paid trips, timestamps out of order — because a single class of bad rows can skew a city's entire average.
Cuisine labels arrive as 'N. Indian', 'North Indian', and 'north indian'; an audit with value_counts surfaces the variants so they can be standardised before menus are grouped.
Credit bureaus audit incoming records for duplicate identities and conflicting entries, since a duplicated account can double-count debt and misprice risk.
1) Missing values (NaN, blanks). 2) Duplicates (whole rows or key collisions). 3) Wrong types (numbers stored as text, dates as strings). 4) Inconsistent categories (same thing spelled many ways). 5) Invalid values (negative ages, future birthdates, out-of-range codes). 6) Structural issues (wrong headers, merged columns, mixed units).
import pandas as pd
df = pd.read_csv('orders.csv')
print(df.shape) # scale: rows, cols
print(df.info()) # dtypes + non-null counts (types + missing)
print(df.describe()) # numeric ranges (invalid values, outliers)
print(df.isna().sum()) # missing per column
print(df.duplicated().sum())# duplicate row countFive commands cover four of the six categories in seconds. info() catches wrong types and missingness; describe() exposes impossible ranges; duplicated() counts repeats. This is the standard opening ritual.
print(df['city'].nunique()) # how many distinct values? print(df['city'].value_counts()) # each value and its frequency # 'Mumbai' 400, 'mumbai' 12, 'Bombay' 3 -> same city, three labels
A column that should have, say, 30 cities but shows nunique() of 47 is a red flag: the extra 17 are almost always spelling variants. value_counts() lists them so you can see the long tail of typos.
Pilots don't eyeball the plane and take off — they run the same written checklist every flight, because the cost of a missed item is catastrophic and memory is unreliable. A data audit is the same discipline: the same checks, every dataset, every time, so a problem is never missed just because you were confident the data looked fine.
Order matters. Fix structural problems first (correct headers, split merged fields), then types (so comparisons and ranges work), then duplicates, then missing values, then inconsistent categories, and treat outliers last once everything else is trustworthy. Imputing before fixing types, or deduplicating before fixing the key column, just creates new messes.
Click each stage to see the checks it runs and the issues it surfaces.
You just loaded a CSV and want a first read on its structure.
Three rows. Note price is quoted — it loaded as text, not numbers.
Your task
A product dataset just landed. Run a quick audit that quantifies each quality issue, then read off the summary.
Duplicates: 1 Category variants: 3 Invalid prices: 1
Write your solution in the editor on the right, then hit Run.
Which single command best reveals both column dtypes and missing-value counts at once?
A city column shows nunique() of 47 but you expected about 30 cities. What does this most likely indicate?
Why should you fix data types before deduplicating or imputing?
What should you do before reporting the average?
Given the DataFrame below, print the number of rows that have at least one problem, where a problem is: a missing value in any column, OR a negative amount. df = pd.DataFrame({'id':[1,2,3,4], 'amount':[10.0, -5.0, None, 20.0], 'name':['A', None, 'C', 'D']})
Which data quality dimension can an audit of the data alone usually NOT verify?
Walk me through how you audit a new dataset for quality issues.
I run the same checklist every time so nothing gets missed. First shape, to confirm the scale and catch gross parse errors. Then info(), which shows dtypes and non-null counts together, flagging wrong types and missingness at once. Then describe() for the numeric columns to expose impossible ranges and hints of outliers. For text columns I use value_counts() and nunique() to surface inconsistent spellings and stray categories. I count duplicated() rows and separately check duplicate keys. The output isn't a cleaned dataset — it's a prioritised quality report listing every issue by category and impact, which then drives the cleaning plan. Auditing before cleaning keeps me from fixing symptoms in the wrong order.
What are the main categories of data quality issues, and why does the order you fix them in matter?
I group them into missing values, duplicates, wrong types, inconsistent categories, invalid or out-of-range values, and structural problems like bad headers or merged columns. Order matters because the fixes depend on each other. I fix structural issues first so the columns even mean what I think they do. Then types, because duplicate detection, range checks, and comparisons only behave correctly once a number is a number and a date is a date. Then duplicates, so I'm not imputing or aggregating over repeated rows. Then missing values, then category standardisation, and I leave outliers for last because I want everything else trustworthy before I decide whether an extreme value is an error or a real signal. Cleaning out of order — say imputing before fixing types, or deduplicating on a still-messy key — tends to create fresh problems.
How do you distinguish between validity, accuracy, and consistency, and what are the limits of an automated audit?
Validity is whether a value obeys its own rules — an age in 0 to 120, a status from an allowed set. Accuracy is whether the value is actually correct — the age really is that person's age. Consistency is whether related values agree — an order date before its ship date, or one customer having a single spelling of their name. From the data alone I can check validity against rules, consistency against internal relationships, and completeness against required-field lists, and I can automate all of that with assertions or a validation library. What I generally can't verify without an external source of truth is accuracy: a perfectly valid, internally consistent age of 37 can still be wrong. That's the fundamental limit of an audit — it catches data that violates rules or contradicts itself, but a plausible, self-consistent error can pass straight through, which is why critical pipelines add reconciliation against a trusted system on top of the audit.
Common Mistakes to Avoid
1) Skipping the audit because the data 'looks fine' in head() — the first five rows hide most problems. 2) Trusting describe() to catch category issues — it ignores text columns; use value_counts/nunique for those. 3) Cleaning in the wrong order — fix structure and types before duplicates and imputation. 4) Checking only whole-row duplicates and missing duplicate keys, where the key repeats but other columns differ. 5) Treating an implausible value as automatically wrong — investigate before deleting; some 'outliers' are real. 6) Assuming a passing audit means accurate data — internal checks can't verify real-world accuracy.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'ELI5: why is a data audit like a pre-flight checklist?' • 'Show me an audit that finds all six issue categories in one dataset.' • 'Quiz me on which command catches which issue.' • 'Explain validity vs accuracy vs consistency with fresh examples.' • 'Interview mode: ask me to justify the order I clean issues in.'
Glossary
Data audit — the systematic first-pass inspection of a dataset for quality issues. info() — shows dtypes and non-null counts per column. describe() — summary statistics for numeric columns. value_counts() — frequency of each distinct value in a column. nunique() — number of distinct values. duplicated() — flags repeated rows. Inconsistent categories — the same real value stored under multiple labels. Invalid value — a value that violates a rule (negative age). Structural issue — a problem with the table's shape (wrong headers, merged fields). Validity / Accuracy / Consistency / Completeness — the dimensions of data quality.
Recommended Resources
• Docs: pandas 'Essential basic functionality' for info/describe, and the 'Working with text data' guide for category checks. • Read: a short overview of data quality dimensions (validity, accuracy, completeness, consistency) to frame your audits. • Practice: take any public CSV, run the five-command audit ritual, and write a one-paragraph quality report naming each issue category you find. • Next in DSM: you can spot missingness at a glance — next, in Detecting & Handling Nulls, you'll go deeper into missingness patterns and the imputation choices that follow from them.
Recap
✓ Data quality issues fall into six categories: missing values, duplicates, wrong types, inconsistent categories, invalid values, and structural problems. ✓ Audit every new dataset with a fixed ritual: shape, info, describe, isna, duplicated, and value_counts/nunique for text. ✓ describe() finds invalid numeric ranges; value_counts/nunique find inconsistent categories. ✓ Clean in order: structure → types → duplicates → missing → categories → outliers. ✓ An audit produces a prioritised quality report, not a cleaned dataset. ✓ Internal checks verify validity, consistency, and completeness — but not real-world accuracy. Next up: Detecting & Handling Nulls. You know missingness is one of the six issues — next you'll study its patterns in depth and match each to the right imputation strategy.
Run your code to see the output here.