Two analysts get the same dataset and the same afternoon. One produces forty charts and no conclusions. The other produces five findings, each with a number, a caveat, and a recommendation — and becomes the person leadership calls. The difference isn't talent or tooling: it's that the second one worked through a structured loop of questions, while the first just... looked around.
What you'll learn
What
Exploratory Data Analysis is the disciplined first investigation of a dataset: framing questions before touching data, profiling structure and quality, exploring from single columns outward to relationships, and communicating findings with their limitations. This lesson is the map; the next three lessons are the territory.
Why
EDA is where analyses succeed or fail. Skip it and you model artifacts, report distorted averages, and miss the segment that explains everything. Do it aimlessly and you drown in charts without answers. The workflow — not any individual technique — is what makes exploration produce decisions.
Where it's used
The first days of every data science project, due diligence on any new data source, pre-modelling feature understanding, incident investigations ('why did the metric drop?'), and the analysis sections of every serious report.
Where this runs in production
Pricing-model work starts with structured EDA on listing data — distribution checks, segment splits, outlier review — because a model trained on unexplored data ships whatever pathologies the data carried.
Data journalists run question-driven EDA on public datasets — framing hypotheses, profiling quality, testing relationships — and the published story is the communicate phase, caveats included.
Consulting teams profile a client's data in days: quality audit first, then targeted questions tied to the engagement's hypotheses — a workflow, because there's no time for wandering.
Write down 3–7 questions the analysis should answer, plus your current guesses (hypotheses). 'What drives churn?' becomes 'Do short-tenure customers churn more? Does plan type matter? Is there a usage cliff before cancellation?'. Questions decide which columns matter, what 'done' looks like, and protect you from the infinite chart-space. They can evolve — but wandering starts the moment none are written.
Before any question: what IS this data? Structure — shape, dtypes, one row means what, keys and grain. Quality — the full cleaning-module audit: nulls, duplicates, bad types, inconsistent categories, outliers. Summaries — describe() for numerics, value_counts for categoricals, date ranges for time. The profile phase decides whether answers can be trusted at all, and its output is the quality caveats your final report must carry.
# The profile pass, in one screen df.shape; df.dtypes # structure df.isna().mean().sort_values() # quality: missingness df.duplicated().sum() # quality: repeats df.describe() # numeric summaries df['category'].value_counts(normalize=True) # categorical balance df['date'].agg(['min', 'max']) # time coverage
Every command here is from earlier modules — EDA's profile phase is the cleaning audit wearing its analysis hat. The point isn't running them; it's writing down what they say before moving on.
Univariate first: understand each important column's distribution alone — centre, spread, shape, outliers. Bivariate second: relationships between pairs — the correlations, group differences, and cross-tabs where most findings live. Multivariate last: interactions and segments — where a third variable changes a pairwise story. The order matters because each level's surprises are only interpretable if you understood the level below.
The deliverable is not the notebook. It's a short structured summary: the 3–7 original questions with their answers, each finding stated with its number and its caveat ('churn concentrates in month 1 — but month-1 data predates the pricing change'), plus the new questions exploration raised. An EDA that ends with sharper questions than it started with has succeeded.
Aimless tourism: charts without questions, hours in, nothing to report — prevented by Phase 1. Premature conclusions: shipping the first interesting pattern without checking whether it's an artifact (a default value, a merge fan-out, a segment mix change) — prevented by the insight log's finding-vs-artifact discipline and by asking 'what else would explain this?' before believing anything.
A detective arrives at a scene with questions — who benefits? when did it happen? — and lets evidence revise them. A tourist arrives with a camera and takes pictures of whatever looks striking. Both walk the same rooms; only one closes cases. The dataset is the scene, charts are the photographs, and the difference between analysis and decoration is whether a question preceded each shot.
Click each phase — the arrows loop because answers breed better questions.
A churn dataset lands. Write the questions first.
Three concrete questions, each answerable with a number. 'Understand churn' is a wish; these are tasks.
Your task
Run a mini profile-then-explore pass on a subscriptions table: capture the profile dict, answer one framed question with a groupby, and flag a suspicious pattern for the artifact check.
{'rows': 6, 'churn_rate': 0.6666666666666666, 'min_months': 0}
{'basic': 1.0, 'pro': 0.0}
1Write your solution in the editor on the right, then hit Run.
What belongs FIRST in an EDA, before any pandas is run?
Why explore univariate distributions before bivariate relationships?
In the insight log, what distinguishes a 'finding' from an 'artifact'?
What does the workflow demand before that slide ships?
Produce a mini profile report: print a dict with n_rows, n_missing_total (count of NaN cells in the whole frame), and top_city (the most frequent city). df = pd.DataFrame({'city':['Pune','Delhi','Pune', None,'Pune'], 'amount':[100, None, 250, 300, 90]})
Why should patterns discovered during open-ended exploration be confirmed on new data or with formal tests before becoming claims?
Walk me through how you approach a dataset you've never seen.
Four phases. First I frame: before opening the file I write the questions the analysis exists to answer and my prior guesses — for a churn dataset, maybe 'does churn concentrate early?' and 'which plans leak?'. Second I profile: structure (shape, dtypes, what one row means, the key and grain), quality (the standard audit — missingness, duplicates, type problems, category inconsistencies, outliers), and baselines (describe, value_counts, date coverage). The profile sets the trust level and produces the caveats my findings will carry. Third I explore in strict order: univariate to understand each key column's distribution, bivariate for the relationships where findings usually live, multivariate to check whether stories survive segmentation. Throughout I keep an insight log — one sentence, one number, one tag: finding, artifact, or question. Fourth I communicate: the original questions with answers and caveats, artifacts disclosed, and the sharper questions exploration raised. The phases loop — good answers breed better questions — but skipping a phase is how analyses go wrong quietly.
How do you avoid fooling yourself during EDA?
Three disciplines. First, hypotheses written before looking: when I predict 'monthly plans churn more' and the data agrees, that's modest evidence; when a pattern I never predicted appears, I treat it with extra suspicion, because surprise is where both discoveries and artifacts live. Second, the artifact interrogation: before believing any striking pattern I ask what data mechanics could produce it — default values (a spike at exactly 0 or -999), tracking-era gaps (segments that predate instrumentation), merge fan-out inflating one group, or mix shifts masquerading as trend changes. The exactly-zero LTV segment that turns out to predate LTV tracking is the canonical case. Third, respecting the exploration/confirmation boundary: cutting data enough ways always yields something striking by chance, so exploratory findings are labelled hypotheses and confirmed on new data or with proper tests before becoming claims. And mechanically: the insight log with finding/artifact/question tags — memory is an unreliable narrator of an afternoon's charts, and the log is what keeps the final report honest.
What makes an EDA deliverable good, and what are the signs of a bad one?
A good deliverable answers the framed questions with numbers and carries its uncertainty visibly: each finding is one sentence with a magnitude ('churned customers' median tenure is 2 months vs 19'), a caveat where one exists ('plan analysis excludes the 8% with missing plan'), and artifacts are disclosed rather than silently dropped. It ends with the new questions raised — an EDA that sharpens the question set has done its job even when answers are 'the data can't tell us'. It's short: the notebook may hold forty charts, the summary holds five findings, because selection is the analyst's value-add. Bad deliverables have signatures: chart dumps with no prose (the reader is left to do the analysis); conclusions without denominators or caveats; means reported on distributions the analyst never looked at; causal language on observational cuts; and no mention of data quality at all — which tells me the profile phase never happened, and none of the numbers can be trusted to the digit they're quoted to.
Common Mistakes to Avoid
1) Opening the data before writing any questions — tourism mode engaged. 2) Skipping the profile phase and discovering the duplicate rows AFTER presenting the revenue numbers. 3) Jumping straight to correlations without univariate context — skew and outliers change which statistics are honest. 4) Believing the first striking pattern without the artifact interrogation. 5) Keeping insights in your head instead of a tagged log. 6) Delivering the notebook as the report — selection is the job. 7) Causal claims from observational cuts ('support contact halves churn'). 8) Quality caveats discovered but not attached to the findings they limit.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'ELI5: detective vs tourist EDA.' • 'Help me frame questions for a food-delivery orders dataset.' • 'Quiz me: finding, artifact, or needs-more-info for scenarios you invent.' • 'Show me three ways an artifact can masquerade as an insight.' • 'Interview mode: I present an EDA summary, you challenge its caveats.'
Glossary
EDA — exploratory data analysis: the disciplined first investigation of a dataset. Framing — writing questions and hypotheses before touching data. Profiling — structure + quality + summary pass; produces the trust level and caveats. Grain — what one row represents. Univariate/bivariate/multivariate — one column alone / pairs / three-plus interactions. Insight log — running list of observations, each with a number and a tag. Artifact — a pattern produced by data mechanics rather than real behaviour. Hypothesis candidate — an exploratory pattern awaiting confirmation. Confirmation — testing on new data or with formal statistics before claiming.
Recommended Resources
• Read: the opening chapters of Tukey's 'Exploratory Data Analysis' for the philosophy, and any modern write-up of 'question-driven EDA' for the practice. • Tools: ydata-profiling (automated profile reports) — use it to accelerate Phase 2, never to replace Phases 1 and 4. • Practice: pick any Kaggle dataset, write 5 questions BEFORE downloading, and hold yourself to the four-phase loop with a tagged insight log. • Next in DSM: the workflow's explore phase begins in earnest — Univariate Analysis, one column at a time.
Recap
✓ EDA is a loop: Frame (questions + hypotheses) → Profile (structure, quality, summaries) → Explore (uni → bi → multivariate) → Communicate (answers + caveats + next questions). ✓ Questions before data: they select columns, define done, and prevent chart tourism. ✓ The profile phase is the cleaning audit powering trust — its caveats attach to every finding. ✓ Keep an insight log: one sentence, one number, one tag — finding, artifact, or question. ✓ Interrogate striking patterns for artifacts (defaults, tracking gaps, fan-out) before believing them. ✓ Exploration generates hypotheses; confirmation on new data or formal tests earns claims. Next up: Univariate Analysis. The workflow says start with single columns — next you'll master distributions: centre, spread, shape, and what value_counts and describe are really telling you.
Run your code to see the output here.