In 1973, UC Berkeley was sued for gender bias: 44% of male applicants were admitted versus 35% of female. Then someone split the data by department — and within most departments, women were admitted at HIGHER rates. Women had applied to the most competitive departments. The bias verdict flipped with one groupby. That's multivariate analysis: the level where two-variable 'truths' go to be tested.
What you'll learn
What
Multivariate analysis examines three or more variables together: segmenting a bivariate relationship by a third variable (does the pattern hold within each group?), detecting interaction effects (does the EFFECT of X on Y depend on Z?), and reading multi-key pivot tables. The star exhibit is Simpson's paradox — an aggregate trend that reverses inside every subgroup.
Why
Bivariate findings drive decisions, and third variables are how those findings lie: mix effects masquerade as trends, confounders masquerade as causes, and a treatment that helps everyone can look harmful in aggregate. One segmentation check is often the difference between insight and lawsuit-grade error.
Where it's used
Any A/B readout sliced by segment, fairness audits (the Berkeley case), marketing mix analysis, cohort retention views, pre-modelling interaction hunting, and every 'why did the metric move?' investigation where composition shifted.
Where this runs in production
The 1973 admissions data is the textbook Simpson's paradox: aggregate rates suggested bias against women; department-level rates showed the opposite — applicant mix, not admissions behaviour, produced the aggregate gap.
A/B platforms report lift by segment because average effects hide heterogeneity: a checkout change can help mobile users and hurt desktop users, netting to 'no effect' overall.
Clinical results are reported within age strata precisely because aggregate rates across differently-sized, differently-risked groups produce Simpson-style distortions.
# The segmentation check, two ways
df.groupby(['source', 'device'])['converted'].mean().unstack()
df.pivot_table(index='source', columns='device',
values='converted', aggfunc='mean')
# device desktop mobile
# source
# ads 0.10 0.08 <- pattern within ads
# organic 0.22 0.19 <- pattern within organicOne grid answers three questions at once: the X–Y pattern within each Z (read across rows), the Z–Y pattern within each X (read down columns), and whether any cell breaks the story. Carry cell COUNTS in a twin table — a rate over 12 rows is a rumour.
A trend can hold in aggregate yet reverse within every subgroup (or vice versa). Mechanism: the groups differ in SIZE and BASELINE, and the aggregate mixes them by weight. Berkeley: women applied more to departments with low admission rates for everyone — aggregate admission gap, department-level near-parity or advantage. The check is mechanical: compute the comparison within each level of plausible third variables; if the sign flips, the aggregate was a composition story, not a behaviour story.
# Aggregate says treatment B wins…
df.groupby('treatment')['success'].mean()
# …but within each severity level, A wins
df.groupby(['severity', 'treatment'])['success'].mean().unstack()
# Why: B was given mostly easy cases. Mix, not merit.The kidney-treatment version of the paradox: B looks better overall because it was assigned the easy cases. The within-severity view is the honest one HERE — because severity influenced treatment assignment. Which view is 'true' depends on the causal structure, which is why the paradox can't be resolved by arithmetic alone.
Confounding: Z drives both X and Y, creating a spurious X–Y link (or masking a real one); conditioning on Z removes the distortion — the Berkeley case. Interaction: X genuinely affects Y, but the SIZE (or sign) of the effect depends on Z — a discount lifts conversion 2% on desktop and 11% on mobile. Confounders are noise to remove; interactions are findings to report. Both are discovered by the same segmentation grid, distinguished by the question: did the effect DISAPPEAR (confounding) or VARY (interaction)?
Every additional split divides your data: 1,000 rows → 2 sources × 3 devices × 4 regions = 24 cells averaging 42 rows, some near-empty. Rates from tiny cells swing wildly and 'insights' bloom by chance (multiple comparisons again). Rules: always display cell counts, set a floor (no rate quoted below ~30–50 rows without a caveat), split by variables you have REASONS to suspect, and accept that beyond 2–3 conditioning variables the honest tool is a model (regression — the ML domain), not a deeper pivot.
One room at 10°C with 9 people, another at 30°C with 1 person: 'the average person experiences 12°C'. True by arithmetic, experienced by nobody. Now move people between rooms and the 'average temperature' changes with nobody touching a thermostat — that's a mix effect: an aggregate moving purely from composition. Segment-level views are thermometers per room; multivariate analysis is refusing to discuss the building's 'temperature' before checking who's standing where.
Click each stage — from bivariate claim to multivariate verdict.
Conversion by device — now split by traffic source.
Eight sessions across source × device. The bivariate view (by device alone) is about to gain a dimension.
Your task
A support dataset claims 'chat resolves tickets better than email'. Run the aggregate, then condition on ticket difficulty and see whether the claim survives.
{'chat': 0.75, 'email': 0.62}
channel chat email
difficulty
easy 0.83 1.0
hard 0.50 0.5
channel chat email
difficulty
easy 6 2
hard 2 6Write your solution in the editor on the right, then hit Run.
What is Simpson's paradox?
A discount lifts conversion +50 points for small baskets and 0 points for large ones. This is best described as:
Why must a segmentation grid always ship with a twin table of cell counts?
What explains the 'improvement'?
The aggregate says plan 'pro' retains better. Condition on company size: print the aggregate retention per plan, then the (size, plan) retention grid, and see the flip. df = pd.DataFrame({'plan':['pro']*6+['basic']*6, 'size':['big']*4+['small']*2+['big']*1+['small']*5, 'retained':[1,1,1,0,1,0, 1, 1,1,0,1,0]})
When a comparison flips under conditioning, which view — aggregate or within-group — is the 'true' one?
Explain Simpson's paradox with a concrete example and how you check for it.
Simpson's paradox is an aggregate trend that reverses within every subgroup, produced by composition rather than behaviour. The Berkeley admissions case is the canonical example: 44% of men admitted versus 35% of women in aggregate, yet within most departments women were admitted at equal or higher rates — women had disproportionately applied to the most selective departments, so the aggregate gap measured application mix, not admission bias. The mechanism is always the same: subgroups with different sizes and baselines, mixed by weight. My check is mechanical: for any decision-bearing comparison, I list third variables that plausibly differ between the groups AND relate to the outcome — case severity, department, device, tenure — and recompute the comparison within each level via a two-key pivot with a twin counts table. Same sign everywhere: the finding strengthens. Sign flips: I report the within-group view with the mix explanation, because the aggregate is now known to be a composition artifact. What arithmetic can't decide is which view is causally honest — that requires knowing whether the third variable influenced group membership, which is domain reasoning, not pandas.
How do you tell confounding apart from an interaction effect, and why does the difference matter?
Both are discovered by the same segmentation grid, but they're different phenomena with opposite fates in the report. Confounding: the third variable drives both X and Y, manufacturing (or masking) an X–Y association — condition on it and the apparent effect shrinks or vanishes. Berkeley's department variable is a confounder of the gender-admission link. Confounders are distortions: the analysis is wrong until they're handled. Interaction: X genuinely affects Y, but the effect's size or direction depends on Z — free shipping lifting small-basket conversion by 50 points and large-basket by zero. Conditioning doesn't make it vanish; it makes it VARY. Interactions are findings, often the most actionable kind, because they say where to target. Diagnostically: after conditioning, ask 'did the effect disappear or diversify?' — disappearance points to confounding, heterogeneity to interaction. The stakes: treating a confounder as a finding ships a false insight; averaging over an interaction ships a true-but-useless one ('the feature adds 2% on average' when it adds 11% on mobile and nothing elsewhere).
A metric moved and leadership wants to know why. Describe your investigation.
First I decompose the move into rate versus mix, because most metric fire-drills die there. I compute the metric within each major segment for both periods, alongside each segment's weight in both periods. Three outcomes: within-segment values moved (a behaviour change — investigate what changed in those segments); weights moved while within-segment values stayed flat (pure composition — the 'decline' might be a growing low-baseline segment, which can be good news); or both. Second, I check denominators and definitions: did the population entering the metric change — a tracking fix, a bot filter, a new market — because 'the metric' often measures a different set of things than last quarter. Third, timing: did the move coincide with a launch, a pricing change, a data pipeline deploy? Sharp discontinuities usually mean instrumentation, gradual drifts mean behaviour or mix. Throughout, cell counts guard against over-slicing, and the deliverable follows the EDA-workflow shape: the decomposition table, the finding in one sentence with numbers ('AOV fell because app share rose from 30% to 45%; within-platform AOV was flat'), the caveats, and the follow-up question — which is often about why the mix shifted, a healthier question than the one we started with.
Common Mistakes to Avoid
1) Shipping an aggregate comparison without one within-group check — Simpson's paradox is a groupby away. 2) Reading a mix effect as a behaviour change ('AOV fell!' when the segment weights moved). 3) Deleting an interaction by averaging over it — 'the average effect is 2%' hides the segment where it's 11%. 4) Quoting rates from cells of 5 rows — always ship the counts twin table. 5) Splitting by every available column and 'discovering' patterns in the noise — condition on suspects you have reasons for. 6) Conditioning on variables CAUSED by the exposure (mediators/colliders) and creating new distortions. 7) Believing arithmetic settles which level is 'true' — the causal structure decides, and that's domain knowledge.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'ELI5: Simpson's paradox with the two-rooms temperature story.' • 'Build me a tiny dataset where the aggregate and subgroup trends disagree, and walk me through why.' • 'Quiz me: confounder, interaction, or mix effect for scenarios you invent.' • 'Decompose a metric move into rate vs mix on example data.' • 'Interview mode: I present an aggregate finding, you demand the conditioning checks.'
Glossary
Multivariate — three or more variables analysed jointly. Conditioning / segmenting — recomputing a relationship within levels of a third variable. Simpson's paradox — aggregate trend reversing within every subgroup via composition. Mix effect — an aggregate moving because segment WEIGHTS shifted, not segment values. Confounder — Z driving both X and Y, distorting their association. Interaction — X's effect on Y changing size/sign across Z levels. Stratum/strata — the subgroup levels you condition on. Cell — one combination in a multi-key grid; small cells breed noise. Mediator — a variable on the causal path from X to Y (don't condition on it casually).
Recommended Resources
• Look up: the Bickel et al. (1975) Berkeley admissions paper and the kidney-stone treatment study — the two canonical Simpson cases. • Read: a gentle introduction to confounding vs mediation (any epidemiology or causal-inference primer's first chapter). • Practice: take last month's most confident bivariate finding at work (or from a Kaggle notebook) and subject it to three conditioning checks with counts tables — write down whether it held, flipped, or varied. • Next in DSM: everything converges — the capstone Project: EDA on a Real Dataset runs the full workflow end to end.
Recap
✓ The core move is conditioning: recompute any bivariate finding within levels of plausible third variables (two-key pivot + counts twin). ✓ Simpson's paradox: aggregates can reverse inside every subgroup when sizes and baselines differ — composition masquerading as behaviour. ✓ Confounding makes effects vanish under conditioning (a distortion); interaction makes them vary (a finding — often the actionable one). ✓ Metric moves decompose into rate changes vs mix changes — check within-segment values against segment weights before declaring behaviour change. ✓ Cells shrink fast: show counts, floor your rates, condition with reasons, and graduate to models beyond 2–3 variables. ✓ Which level is 'true' is a causal question — arithmetic finds the flip; domain knowledge adjudicates it. Next up: 🏗 Project — EDA on a Real Dataset. The full workflow, one real dataset, one deliverable report: frame, profile, explore across all three levels, and communicate findings that survive interrogation.
Run your code to see the output here.