'Revenue by region.' 'Average order value per customer segment.' 'Signups per week.' Every dashboard you've ever seen is the same operation wearing different clothes: split the rows into groups, compute something per group, glue the answers together. Learn groupby once and you can build all of them.
What you'll learn
What
groupby implements split-apply-combine: df.groupby(key) splits rows into groups by key value, an aggregation function (sum, mean, count, or your own) is applied to each group, and pandas combines the per-group results into a new indexed structure. The agg method — especially named aggregation — computes several statistics at once.
Why
Raw rows answer no business questions; per-group summaries answer almost all of them. groupby is the single most-used pandas operation in professional analytics — it IS the 'GROUP BY' of SQL, the pivot table of Excel, and the backbone of feature engineering for models.
Where it's used
Every KPI report (metrics per region/product/cohort), A/B test readouts (metrics per variant), feature engineering (per-customer aggregates), and data validation (counts per source, per day).
Where this runs in production
Every store's dashboard — revenue by day, orders by product, sales by channel — is a groupby over the orders table, computed for over a million merchants.
Average prep time, order counts, and ratings are grouped per restaurant per hour to drive courier dispatch and restaurant ranking decisions.
Wrapped is a giant groupby('user'): total minutes, top artist, top genre — per-listener aggregates computed across hundreds of billions of stream events.
df.groupby('region')['revenue'].sum()
# region <- group keys become the index
# East 42000
# North 31500
# South 27800
df.groupby('region')['revenue'].mean()
df.groupby('region').size() # rows per group (counts NaN too)
df.groupby('region')['deal'].count() # non-null values in 'deal' per groupThe pattern reads like a sentence: group by region, take the revenue column, sum it. Group keys become the result's index. size() counts rows; count() counts non-null values of a specific column — on clean data they agree, on real data they don't.
Split: rows are partitioned by key value (one group per distinct key). Apply: the aggregation runs once per group, seeing only that group's rows. Combine: the per-group answers are assembled into a Series or DataFrame indexed by the keys. Every groupby, however complex, is these three steps.
# Multiple keys -> MultiIndex, one row per key combination
df.groupby(['region', 'product'])['revenue'].sum()
# Multiple statistics at once
df.groupby('region')['revenue'].agg(['sum', 'mean', 'count'])
# Named aggregation: report-ready column names, mixed sources
df.groupby('region').agg(
total_revenue=('revenue', 'sum'),
avg_deal=('revenue', 'mean'),
n_orders=('order_id', 'count'),
)agg is the multiplexer. The list form gives one column per function; named aggregation — new_col=(source_col, func) — is the professional idiom because the output columns are named exactly what the report needs.
groupby splits on exact values, so ' Mumbai' and 'mumbai' become separate groups — every string-cleaning lesson lands here. Also, by default groupby SORTS groups by key and DROPS rows whose key is NaN: dropna=False keeps a NaN group if missing keys matter, and observed= matters for categoricals. A groupby silently shrinking your row total is usually NaN keys vanishing.
A mail room sorts the day's letters into one pigeonhole per department (split), a clerk counts each pigeonhole's stack (apply), and the totals go on one summary sheet in department order (combine). Nobody counts the unsorted pile — and a letter with no department written on it never reaches any pigeonhole, which is exactly what happens to NaN keys.
Click each stage to follow rows from raw table to per-group summary.
The canonical groupby: total a numeric column per category.
Five orders across two regions — the question is 'how much per region?'.
Your task
A food-delivery orders table needs a per-city report: total order value, average order value, and number of orders — then identify the top city by total.
total avg orders city Delhi 750 375.0 2 Goa 500 500.0 1 Pune 750 250.0 3 Delhi
Write your solution in the editor on the right, then hit Run.
What does df.groupby('region') return, before any aggregation is applied?
In split-apply-combine, what does the 'apply' step operate on?
For a group containing rows whose 'score' column has some NaN values, which is TRUE?
What's the most likely explanation?
Group by category, and for each category print a report with columns n (row count) and avg_price (mean price, rounded to 1 decimal), sorted by avg_price descending. df = pd.DataFrame({'category':['toys','books','toys','books','games'], 'price':[20.0, 10.0, 30.0, 14.0, 50.0]})
Why is agg(total=('rev','sum')) generally preferred over a lambda like agg(total=('rev', lambda s: s.sum())) at scale?
Explain split-apply-combine to someone who only knows Excel.
It's a pivot table, formalised. Split: pandas sorts every row into buckets by the values of the key columns — like dragging 'Region' into the Rows area. Apply: for each bucket, a summary function runs on just that bucket's rows — like choosing Sum of Revenue in the Values area. Combine: the per-bucket answers become one small table indexed by the keys. The code reads like the sentence you'd say aloud: df.groupby('region')['revenue'].sum() — group by region, take revenue, sum it. Where it outgrows Excel is composability: group by several keys at once, compute many named statistics in one pass with agg, feed the result straight into a merge or a chart, and rerun it identically tomorrow on new data — the pivot table becomes a reproducible pipeline step rather than a manual artifact.
Your grouped totals don't match the raw column total. What are the usual suspects?
First suspect: NaN group keys. groupby drops rows whose key is missing, so those rows vanish from every group — dropna=False restores them as an explicit NaN group and the totals reconcile. Second: dirty keys splitting groups — ' Mumbai' versus 'mumbai' doesn't change the grand total but silently splits one group's number across two lines, so a specific group looks too small. Third: confusing count() with size() when someone downstream summed a 'count' column — count() skips nulls, size() doesn't. Fourth, for categorical dtype keys, the observed parameter can include empty categories as zero rows or exclude present combinations depending on the setting. My debugging order is: compare df[key].isna().sum() against the gap, then value_counts on the key looking for near-duplicate labels, then check which count semantics the metric actually intended.
How does groupby feed feature engineering for machine learning?
Most predictive signal about an entity lives in its history, and history arrives as many rows per entity — transactions per customer, sessions per user. Models want one row per entity, so groupby is the bridge: group by the entity key and aggregate behaviour into features — count of orders, mean and max basket value, nunique products, days since first purchase via min of the date. Named aggregation builds the whole feature block in one pass with clean column names, and the result merges back onto the entity table by key. Two production cautions: first, time-awareness — features for predicting churn as of March must aggregate only data before March, so you filter by date before grouping or you leak the future into training. Second, entities with no history disappear from a plain groupby, so the merge back must be a left join from the entity table with explicit fill values for the newcomers — otherwise your model never learns what 'brand new customer' looks like.
Common Mistakes to Avoid
1) Forgetting that NaN keys are dropped by default — grouped totals stop matching the table total (dropna=False). 2) Grouping on uncleaned strings — casing/whitespace variants become separate groups. 3) Using count() when you mean size() on data with nulls. 4) Selecting the column AFTER aggregating instead of before — aggregate only what you need: groupby(key)['col'].sum(). 5) Fighting the MultiIndex instead of using reset_index() or as_index=False when a flat table is wanted. 6) Writing lambdas for aggregations that already exist as fast built-ins ('nunique', 'first', 'std'). 7) Reporting a mean per group without the group's size next to it — a 5.0 average over 2 rows isn't a 5.0 average over 20,000.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'ELI5: split-apply-combine with the mail-room story.' • 'Show me named aggregation building a 5-column report in one pass.' • 'Quiz me: size vs count vs nunique on data with nulls.' • 'Why did my grouped totals lose 400k — walk me through the NaN-key trap.' • 'Interview mode: have me design per-customer features for a churn model with groupby.'
Glossary
groupby — partition rows by key value(s); returns a lazy GroupBy object. Split-apply-combine — partition, per-group compute, reassemble. Aggregation — a function collapsing many rows to one value (sum, mean, count…). agg — apply one or many aggregations; named aggregation is new_col=(source, func). size() — rows per group, nulls included. count() — non-null values per column per group. MultiIndex — hierarchical index from grouping by multiple keys. reset_index() — move index level(s) back to ordinary columns. as_index=False — keep group keys as columns in the result. dropna=False — keep rows with NaN group keys as their own group.
Recommended Resources
• Docs: pandas user guide 'Group by: split-apply-combine' — the canonical treatment, including named aggregation. • Read: Hadley Wickham's paper 'The Split-Apply-Combine Strategy for Data Analysis' — the idea's origin, language-independent. • Practice: take any transactions CSV and rebuild a dashboard's numbers: totals by category, by month, by category-and-month, each with named aggregation. • Next in DSM: grouped results are long and skinny — Reshaping (pivot, melt) turns them into the wide tables humans read.
Recap
✓ groupby is lazy: the split is a recipe, and aggregation triggers the compute. ✓ Split-apply-combine: partition by key, collapse each group, reassemble indexed by keys. ✓ agg computes many statistics in one pass; named aggregation names output columns for the report. ✓ size() counts rows, count() counts non-nulls — different answers on real data. ✓ NaN keys silently drop out (dropna=False keeps them); dirty string keys split groups. ✓ reset_index()/as_index=False flatten grouped output for plotting, exporting, and joining. Next up: Reshaping (pivot, melt, stack, unstack). Your groupby output has one row per (region, product) pair — next you'll pivot it into the region-by-product grid a human wants to read.
Run your code to see the output here.