In 1943, engineers examined bombers returning from missions and mapped where they were riddled with bullet holes — the wings, the tail. The obvious move: add armor where the holes are. A statistician named Abraham Wald stopped them. The holes, he said, show where a plane can be hit and still come home. Armor the places with no holes — the engines — because planes hit there never returned to be studied. The data was clean. The data was complete. And it pointed exactly the wrong way, because of who wasn't in it. That's bias, and no quality audit will ever catch it.
What you'll learn
What
Bias is a systematic distortion in how data was collected, so the data doesn't represent the reality you think it does. This lesson covers three of the most common and dangerous forms: selection bias, survivorship bias, and sampling bias.
Why
Last lesson's flaws live inside the data — you can find a blank cell or a duplicate by looking. Bias lives in what's absent: the people who didn't respond, the companies that failed, the planes that didn't return. It can't be cleaned away, only reasoned about — and a biased dataset produces confident, precise, completely wrong conclusions.
Where it's used
Every survey, every A/B test, every model trained on historical data. Product analytics, medical trials, hiring algorithms, and market research all live or die by whether their data represents the population they claim to describe.
Where this runs in production
An experimental résumé-screening model was trained on a decade of past hires — mostly men. It learned that pattern and penalised résumés mentioning 'women's', a textbook case of bias inherited from non-representative historical data.
Polls have called elections wrongly when their samples over-represented people who were easy to reach — landline owners, or those willing to answer. The math was fine; the sample didn't match the electorate.
People who leave reviews are disproportionately delighted or furious. The quiet, satisfied majority rarely rates, so an app's average score reflects who chose to speak, not the typical user.
Selection bias — the group in your data was chosen in a way that makes it unrepresentative (e.g. studying only your current customers to learn about all shoppers). Survivorship bias — only the 'survivors' reach your data; the failures dropped out and are invisible (e.g. analysing only funded startups). Sampling bias — your sample doesn't match the population's makeup (e.g. an online poll about a product used mostly offline).
Picture studying 'habits of successful founders' by interviewing billionaire CEOs. You'll find they dropped out of college, took huge risks, worked 100-hour weeks — and conclude that's the recipe. But thousands of people did exactly the same things and failed; they're just not on your interview list. You only see the survivors, so any trait shared by survivors looks like a cause of success when it might be a cause of failure too. The failures are the missing armor on Wald's bombers.
It's tempting to think a huge dataset must be trustworthy. But if the collection method is biased, size only makes the wrong answer more precise. A poll of 2,000,000 landline owners predicts landline owners' opinions extremely accurately — and the electorate's poorly. Always question how the data was collected before you trust its size.
# Survey: 'How many hours a week do you exercise?' # Distributed only inside a running-club app. avg = 7.4 # hours/week # Looks authoritative. But the sample is runners. # The national average is a fraction of this.
Every response here can be honest and every calculation correct, yet the headline number is meaningless as a national figure. The distortion is in who was asked — selection and sampling bias — not in any individual value.
Click each node. Bias is the gap between the population you care about and the sample you actually captured — and who fell through it.
Select a type to see its full definition, operations, and data science usage.
A gym surveys people inside the gym to ask 'How often do you work out?' and reports the national average.
By asking only people at the gym, the survey selected the most active group. Everyone who never exercises had no chance to be included.
Selection bias — the group studied is not representative of the whole population.
Your task
A survey about weekly reading habits was distributed only through a book-club newsletter. Below are the responses. Compute the average, then set `is_representative` to False and write one line naming who is missing. Run it to see how a correct calculation can still mislead.
Average hours: 9.3 Representative of all adults: False Missing from the data: people who don't read / aren't in book clubs
Write your solution in the editor on the right, then hit Run.
What is bias in data?
Studying only successful startups to learn 'what makes companies succeed' is an example of which bias?
A poll's sample is 90% urban, but the country is 50% rural. Collecting 10× more responses the same way will:
What's the strongest objection?
Compare a sample's makeup to the population to flag sampling bias. Given the sample's urban fraction and the population's urban fraction, print 'biased' if they differ by more than 0.15, else 'ok'. Use abs() for the difference.
Why might this model be biased, and how would you frame the fix?
What is survivorship bias? Give an example.
Survivorship bias happens when only the 'survivors' of some process make it into your data, while the failures drop out and become invisible — so any pattern you see among survivors can be badly misleading. The classic example is Abraham Wald and the WWII bombers: engineers wanted to armor the planes where returning bombers had the most bullet holes, but Wald realised those were the survivable hits. The planes shot in the engines never came back to be counted, so the armor belonged exactly where there were no holes. A modern version is studying only successful companies to find the 'secret to success' — thousands of failed companies did the same things and aren't in the dataset, so the shared traits might explain nothing. The tell is always asking who or what didn't survive to be measured.
What's the difference between selection bias and sampling bias?
They're closely related and often overlap, but I distinguish them by where the distortion enters. Selection bias is about how units get into the study at all — if the mechanism for entering the dataset is tied to the thing you're measuring, the group is skewed, like surveying only current customers to learn about all shoppers. Sampling bias is specifically about the sample's composition not matching the population's — for instance a sample that's 90% urban when the population is half rural. In practice a biased selection method usually produces a non-representative sample, so they travel together, and the underlying question for both is the same: does the group I actually measured represent the population I want to describe? The fix for both is a representative sampling design, and crucially not just collecting more data.
How would you check a new dataset for bias before trusting conclusions from it?
Since bias is about collection, not the values themselves, I start by asking how the data was generated: who or what could enter the dataset, and by what mechanism. Then I ask the core question — who or what is missing, and would including them change the answer — which surfaces survivorship problems like absent failures and non-response problems like the silent majority in reviews. Next I compare the sample's composition to what I know about the target population on key attributes like age, geography, or device, to catch sampling bias. I also look at whether entry into the data correlates with the outcome I'm studying, which flags selection bias. Importantly, I treat a large dataset as no reassurance — size makes a biased method more precise, not more correct. If I find bias I can't collect around, I at least document the limitation clearly so conclusions are scoped to the group the data actually represents.
Common Mistakes to Avoid
1) Trusting a dataset because it's large — size makes a biased answer more precise, not more correct. 2) Studying only survivors (successful companies, returning planes) and ignoring the invisible failures. 3) Confusing a clean dataset with an unbiased one — bias passes every quality check. 4) Generalising a self-selected group (reviewers, volunteers) to everyone. 5) Trying to 'clean' bias out of the values instead of fixing how the data is collected.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'Give me five short scenarios and quiz me on which bias each one is.' • 'ELI5 the WWII bomber survivorship-bias story.' • 'Show me a survey design and ask who's missing from it.' • 'Explain why more data doesn't fix sampling bias, with a fresh example.' • 'Interview mode: ask me to distinguish selection from sampling bias.'
Glossary
Bias — a systematic distortion in how data was collected, so it misrepresents reality. Population — the full group a conclusion is meant to describe. Sample — the subset actually measured. Selection bias — units enter the data in a way that skews the group. Survivorship bias — only survivors reach the data; failures are invisible. Sampling bias — the sample's composition doesn't match the population's. Non-response bias — those who don't respond differ systematically from those who do. Representative sample — one whose makeup mirrors the population.
Recommended Resources
• Story: read about Abraham Wald and the WWII bomber 'missing bullet holes' — the definitive survivorship-bias example. • Concept: search 'why polls get elections wrong' for real sampling-bias case studies. • Reading: the idea of a 'representative sample' in any intro-statistics resource. • Reflection: pick one dataset you've seen this week and ask 'who is missing from this?' • Next in DSM: you've finished Understanding Data — you can now read, describe, quality-check, and interrogate a dataset for bias. Next you move from judging data to working with it directly in Python and pandas.
Recap
✓ Bias is a systematic distortion in how data was collected — it misrepresents reality and survives every quality check. ✓ Selection bias: the group studied enters the data in a way that makes it unrepresentative. ✓ Survivorship bias: only survivors reach the data; the failures are invisible (Wald's bombers, successful-company studies). ✓ Sampling bias: the sample's composition doesn't match the population's. ✓ More data never fixes bias — it only makes a wrong answer more precise; representativeness does. ✓ The one habit that catches all three: ask 'who or what is missing, and would including them change the answer?' Module complete: Understanding Data. You can now read a table, describe its anatomy, audit its quality, and interrogate it for bias — everything needed to judge whether data is worth trusting. Next you'll stop judging data from the outside and start working with it directly in Python and pandas.
Run your code to see the output here.