Sooner or later you hit a rule no built-in expresses: 'shipping is 40 if express and weight over 5kg, else 25 if express, else 10'. apply lets you run ANY Python function over rows or groups — total freedom. It's also 10–100× slower than vectorized code, and half the apply calls in real codebases didn't need to exist. This lesson teaches both the tool and the judgement.
What you'll learn
What
apply runs a function over a Series' values, a DataFrame's rows, or groupby groups. transform is its shape-preserving sibling for groups: it computes per group but returns a value for EVERY ROW, aligned to the original index — the tool behind group-relative features like 'this order versus the customer's average'.
Why
Custom business logic is inevitable; knowing apply means never being stuck. But pandas' speed comes from vectorized C loops, and apply falls back to Python-per-row. The professional skill is double: write the custom function when needed, and recognise the vectorized form that makes it unnecessary.
Where it's used
Encoding business rules (tiered pricing, eligibility flags), group-relative feature engineering (deviation from group mean, share of group total), text munging beyond .str, and any one-off logic during exploration.
Where this runs in production
Payment fees vary by method, country, and volume tiers — rule tables applied over millions of transactions, where the vectorized-vs-apply choice is the difference between seconds and hours.
A home's list price means little alone; models use price relative to the zip-code median — a groupby-transform feature computed for every listing in the country.
Multi-condition eligibility rules (income band × history × basket size) are encoded as vectorized condition tables rather than per-row Python — auditability and speed from the same refactor.
# Series.apply: function per VALUE
df['grade'] = df['score'].apply(lambda x: 'pass' if x >= 40 else 'fail')
# DataFrame.apply(axis=1): function per ROW (a Series of that row's values)
def shipping(row):
if row['express'] and row['weight'] > 5: return 40
if row['express']: return 25
return 10
df['ship'] = df.apply(shipping, axis=1)The two apply shapes: per-value on a Series, per-row on a DataFrame (axis=1 hands your function each row). Inside the function you write plain Python — any logic at all. That freedom is the point, and the cost.
Vectorized operations (df['a'] * df['b'], comparisons, .str methods) run as compiled loops over whole arrays. apply calls YOUR Python function once per element or row — interpreter overhead, boxing each value, building a Series per row for axis=1. On a million rows that's a million function calls: routinely 10–100× slower. Fine for 10,000 rows in exploration; a problem in pipelines.
# The vectorized ladder — try these BEFORE apply:
df['grade'] = np.where(df['score'] >= 40, 'pass', 'fail') # 2 outcomes
conditions = [df['express'] & (df['weight'] > 5), df['express']]
df['ship'] = np.select(conditions, [40, 25], default=10) # n outcomes
df['tier'] = df['plan'].map({'free': 0, 'pro': 1}) # dict lookup
df['band'] = pd.cut(df['age'], bins=[0,18,65,120],
labels=['minor','adult','senior']) # binningMost apply calls are one of four patterns in disguise: two-way branch → np.where; multi-way rules → np.select (conditions checked in order, first match wins); value lookup → map; range bucketing → pd.cut. Same logic, array speed, and arguably clearer — the rule table is visible at a glance.
# agg: one row per group # transform: one value per ORIGINAL row
df.groupby('store')['sales'].mean() df.groupby('store')['sales'].transform('mean')
# store A 120 # 0 120 <- row 0 is store A
# store B 80 # 1 80 <- row 1 is store B
# 2 120 <- row 2 is store A ...
df['vs_store_avg'] = df['sales'] - df.groupby('store')['sales'].transform('mean')
df['share_of_store'] = df['sales'] / df.groupby('store')['sales'].transform('sum')transform is the broadcast: every row learns its own group's mean/sum/max, enabling group-relative columns in one line. You met it in the cleaning module — fillna with per-group medians was exactly this pattern.
Want a SUMMARY TABLE (one row per group) → agg. Want a NEW COLUMN on the original rows (each row compared to or filled from its group) → transform. Same split, same groups, different shape contract. Filtering rows by a group property — 'keep customers with 3+ orders' — is transform too: df[df.groupby('cust')['id'].transform('size') >= 3].
groupby().apply(func) hands your function each group as a whole DataFrame — maximal power (return anything: scalar, row, frame) and minimal speed, plus output shapes that surprise. Reach for it only after agg, transform, and filter can't express the logic. And avoid df.apply(axis=1) for arithmetic that's just column math — df['a']/df['b'] beats a row lambda every time.
Vectorized operations are a stamping press: one setup, then thousands of identical parts per minute. apply is a skilled machinist with a hand file: any shape you can describe, one part at a time. A good shop uses the press for everything standard and the file for the genuinely custom piece — and a good analyst reads 'if/else on two columns' and hears 'that's a press job (np.select)', reserving the file for logic that truly needs Python.
Click through the decision path — apply is the last resort, not the first.
Grade scores pass/fail both ways and see they agree.
Four scores; the rule is score ≥ 40 passes.
Your task
An orders table needs three engineered columns: a vectorized size label, each order's share of its customer's total, and a flag for customers with 3+ orders.
['small', 'big', 'small', 'big', 'small'] [0.2, 0.6, 0.2, 0.8, 0.2] [True, True, True, False, False]
Write your solution in the editor on the right, then hit Run.
What's the key output-shape difference between groupby agg and groupby transform?
Why is df.apply(func, axis=1) typically much slower than the equivalent vectorized expression?
A rule has four ordered outcomes based on combinations of two columns. The best vectorized tool?
Which one-liner is right?
Add a column 'dev' holding each employee's salary minus their department's mean salary, then print df['dev'] as a list. df = pd.DataFrame({'dept':['eng','eng','sales','sales'], 'salary':[100, 120, 80, 60]})
When is keeping an apply call the RIGHT engineering decision?
Explain the difference between agg, transform, and apply on a groupby.
All three run per group; they differ in what the function receives and what shape comes back. agg receives a group's column and must return ONE value — output is one row per group, the summary-table shape. transform also computes per group but the result is broadcast back to every original row, index-aligned — output has the input's length, the new-column shape; that's what powers group-relative features like deviation from the group mean or per-group median fills. apply receives each group as a whole DataFrame and may return anything — scalar, Series, or frame — making it the most flexible and the slowest, with output shapes that depend on what you return. My selection rule is by output shape first: summary → agg, aligned column or row-filter mask → transform, and apply only when the logic needs to see multiple columns of the group at once and neither simpler tool expresses it.
You've inherited a slow pipeline full of df.apply(axis=1). Walk me through the refactor.
First measure — profile which applies actually dominate runtime, because refactoring a millisecond apply is vanity. For each hot one, I classify its body against the vectorized ladder: pure column arithmetic becomes direct expressions; if/else with two outcomes becomes np.where; ordered multi-condition rules become an np.select table, which also reads like the business spec; dict-style lookups become map; numeric range bucketing becomes pd.cut; and anything comparing rows to their group becomes groupby-transform. Genuinely irreducible logic — say, calling an external parser per row — stays as apply, documented as such. The critical discipline is equivalence testing: run old and new on the same data and assert the outputs match exactly before deleting the old path, because refactors that quietly change edge-case behaviour (NaN handling is the classic) are worse than slow code. Typical outcome: the two or three hot applies become rule tables, runtime drops one to two orders of magnitude, and the rules become reviewable by non-engineers.
Give an example of a group-relative feature and why transform is the natural tool for it.
Take fraud-ish anomaly flags: an order of ₹50,000 is unremarkable for a wholesale buyer and alarming for a customer whose median order is ₹800. The feature that captures this is amount relative to the customer's own history — amount divided by the customer's median, or amount minus the customer's mean. Computing it needs each ROW to see a statistic of its GROUP: exactly transform's contract. df['ratio'] = df['amount'] / df.groupby('cust')['amount'].transform('median') — the group median is computed once per customer and broadcast to their rows, index-aligned, in one line. The agg-then-merge alternative (aggregate to a customer table, merge back, divide) produces the same numbers in three steps with a join to audit — transform IS that pattern fused, minus the merge risks. The same shape covers share-of-group (divide by transform('sum')), within-group ranking context, and cohort-relative metrics — most 'personalised' features in industry models are transforms in this sense.
Common Mistakes to Avoid
1) Reaching for apply before walking the ladder: arithmetic → where/select → map/cut → transform. 2) df.apply(axis=1) for column math that's just df['a'] / df['b']. 3) Using agg when you needed a row-aligned result (then merging it back by hand) — that's transform's job. 4) Nesting np.where three deep instead of one readable np.select table. 5) Getting np.select condition ORDER wrong — first match wins, so specific rules go before general ones. 6) Refactoring an apply without an equivalence test and silently changing NaN behaviour. 7) Cargo-cult 'never apply' — for small one-off jobs the clearest code wins.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'ELI5: the stamping press vs the hand file.' • 'Take this if/else rule and show me its np.select table.' • 'Quiz me: agg, transform, or apply for each task you invent?' • 'Show group-relative anomaly flags with transform, step by step.' • 'Interview mode: hand me a slow apply-heavy pipeline and review my refactor plan.'
Glossary
apply — run an arbitrary function per value (Series), per row/column (DataFrame, axis=), or per group (groupby). Vectorized operation — whole-array computation in compiled code; pandas' fast path. np.where — two-outcome conditional array. np.select — ordered multi-condition rule table with default. map — element-wise dict/function lookup on a Series. pd.cut — bin numeric values into labelled ranges. transform (groupby) — per-group computation broadcast back to original rows, index-aligned. agg — per-group computation collapsed to one row per group. Equivalence test — asserting old and new implementations agree before switching.
Recommended Resources
• Docs: pandas user guide 'GroupBy' (the transform and filter sections) and NumPy's np.select/np.where references. • Read: any 'pandas anti-patterns' article on apply overuse — recognising the ladder patterns in others' code trains the reflex. • Practice: find (or write) a row-wise apply implementing 3+ business rules; refactor to np.select, equivalence-test it, and time both at 10k and 1M rows. • Next in DSM: the Data Transformation module is complete — the EDA module begins with The EDA Workflow, where cleaning and transformation start answering real questions.
Recap
✓ apply runs any Python per value, row, or group — total flexibility at 10–100× vectorized cost. ✓ The ladder before apply: column arithmetic → np.where (2 outcomes) → np.select (rule tables) → map/pd.cut (lookups/bins) → groupby-transform (group-relative). ✓ agg collapses to one row per group; transform broadcasts group stats back to every row, index-aligned. ✓ transform powers group-relative features (deviation, share-of-group) and row filters by group properties. ✓ Keep apply when logic is truly irreducible and the scale tolerates it — but refactors demand equivalence tests. Next up: The EDA Workflow. Transformation tools complete — now the Exploratory Data Analysis module turns them toward their purpose: asking a dataset structured questions and reading its answers.
Run your code to see the output here.