Raw data rarely holds the number you actually want. Revenue isn't in the table — but price and quantity are. Profit margin isn't there — but cost and revenue are. Building the columns you need from the ones you have is feature engineering, and it's where analysis turns into insight. Let's build.
What you'll learn
What
Adding a column means assigning a new named field to a DataFrame; modifying means overwriting an existing one. Both usually come from vectorized operations on existing columns — arithmetic, conditional logic, or transformations applied to a whole column at once.
Why
Models and reports are only as good as their columns. A well-chosen derived column — margin, days-since-signup, price-per-unit — can carry more signal than a dozen raw fields. Doing it vectorized keeps it fast on millions of rows.
Where it's used
Every feature-engineering step in an ML pipeline, every KPI in a dashboard, every ratio in a financial report starts as a new DataFrame column.
Where this runs in production
An analytics team adds a margin column as (price − cost) / price across the entire orders table in one vectorized line, then ranks categories by median margin to guide pricing.
User rows get a derived 'engagement_tier' column via pd.cut on daily active minutes, splitting learners into casual, regular, and power segments for targeted notifications.
A boolean 'free_shipping' column is created with np.where(order_total >= 500, True, False), driving checkout logic and A/B tests on the threshold.
import pandas as pd
df = pd.DataFrame({'price': [10.0, 20.0, 30.0], 'qty': [2, 1, 5]})
df['revenue'] = df['price'] * df['qty'] # vectorized, row by row
df['currency'] = 'USD' # scalar broadcast to all rows
print(df)df['price'] * df['qty'] multiplies the two columns element-wise in compiled code — no loop. Assigning the string 'USD' fills every row with the same value. Both create new columns in place.
Any column you can express as arithmetic or a built-in operation on whole columns should be. df['a'] * df['b'] runs hundreds of times faster than looping rows with iterrows(). Reach for a Python loop only when nothing vectorized exists — and that's rare.
import numpy as np
df['tier'] = np.where(df['revenue'] >= 50, 'high', 'low')
df['band'] = pd.cut(df['revenue'],
bins=[0, 30, 60, 200],
labels=['S', 'M', 'L'])np.where picks 'high' where revenue ≥ 50, else 'low', for every row at once. pd.cut slices revenue into three bands by the bin edges and labels each row S, M, or L. Both are vectorized.
df.assign(margin=...) is like writing a new step on a recipe card and handing back a fresh card, rather than scribbling on the original. It returns a new DataFrame with the column added, leaving the input untouched — ideal for method chains where you don't want to mutate as you go.
result = (df
.assign(revenue=df['price'] * df['qty'])
.assign(margin=lambda d: d['revenue'] * 0.2)
)assign() returns a new DataFrame, so you can chain steps. Using a lambda (lambda d: ...) lets a later assign reference a column an earlier assign just created — d is the DataFrame as it exists at that point in the chain.
df['x'] = ... changes df immediately and permanently. df.assign(x=...) returns a new DataFrame and leaves df unchanged — if you forget to capture the result, your column vanishes. Pick deliberately: in-place bracket assignment for quick mutation, assign() for immutable chains.
Click each method to see what it does and when to reach for it.
You're tagging every row of an export with its source system.
A one-column DataFrame with three rows.
Your task
An e-commerce team hands you an orders DataFrame. Add the derived columns their dashboard needs, using vectorized operations.
[45.0, 40.0, 60.0, 54.0] ['medium', 'medium', 'large', 'medium']
Write your solution in the editor on the right, then hit Run.
What does `df['tax'] = 0.2` do?
Which builds a vectorized conditional column?
How does df.assign(y=...) differ from df['y'] = ...?
What's the idiomatic speed-up?
Given the DataFrame below, add a 'profit' column (revenue − cost) and print the name of the product with the highest profit. df = pd.DataFrame({'product':['A','B','C'], 'revenue':[100,250,180], 'cost':[60,200,90]})
In `df.assign(a=lambda d: d['x']*2, b=lambda d: d['a']+1)`, why can b reference a?
How do you add a new column to a DataFrame, and what determines the values each row gets?
The direct way is bracket assignment: df['new'] = <right-hand side>. If the right side is a scalar, pandas broadcasts it to every row. If it's a Series or a column expression like df['a'] * df['b'], pandas aligns it by index and fills each row with the matching value. That index alignment is important — if I assign a Series with a different index, rows that don't line up become NaN. For a value that depends on a condition I use np.where, and for chaining without mutating I use assign, which returns a new frame.
When would you use df.assign() instead of direct bracket assignment?
I reach for assign when I want to build columns as part of a method chain without mutating the input. Because assign returns a new DataFrame, I can write a clean pipeline — filter, then assign a couple of derived columns, then group — all as one expression, which reads top to bottom and doesn't leave a half-modified frame lying around. It also lets a later column reference an earlier one in the same call via a lambda, since the keywords evaluate in order. Bracket assignment is fine and more concise for quick, one-off mutations, but for reproducible pipelines and functional-style code I prefer assign because it avoids side effects and makes the data flow explicit.
Why is vectorized column creation preferred over row-wise apply or loops, and when is apply still justified?
Vectorized operations push the computation into NumPy's compiled C loops over contiguous typed arrays, so they run one to two orders of magnitude faster than iterating rows in Python and they read more clearly as intent. A loop or apply(axis=1) pays Python interpreter overhead per row and often boxes each row into a Series, which is expensive at scale. So my hierarchy is: plain column arithmetic first, then vectorized helpers like np.where, np.select, and the .str/.dt accessors, then apply only when the logic genuinely can't be expressed vectorized — for instance calling an external API per row, complex parsing that has no vectorized equivalent, or a row-dependent algorithm. Even then I treat apply as a signal to double-check there isn't a vectorized path, and on truly large data I'd consider numba or a groupby-based reformulation before accepting per-row Python.
Common Mistakes to Avoid
1) Looping with iterrows() to build a column that could be vectorized — it's dramatically slower and less readable. 2) Forgetting that df.assign() returns a new DataFrame — if you don't capture the result, the column is lost. 3) Assigning a Series with a mismatched index and getting surprise NaNs — the right-hand side aligns by index, not position. 4) Using pd.cut bins that don't cover all values — anything outside the outer edges becomes NaN. 5) Overwriting a column you still need — df['x'] = ... is destructive; keep the original if a later step depends on it.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'ELI5: what does vectorized mean when I create a column?' • 'Show me np.where versus pd.cut with the same data.' • 'Quiz me on assign() versus bracket assignment.' • 'Explain why a Series with the wrong index gives NaN when I assign it.' • 'Interview mode: ask me when apply is justified over vectorized ops.'
Glossary
Derived column — a new column computed from existing ones. Vectorized operation — an operation applied to a whole column at once in compiled code. Broadcast — filling every row with a single scalar value. np.where(cond, a, b) — element-wise if/else producing a column. pd.cut — binning a numeric column into labelled ranges. assign() — returns a new DataFrame with added columns; immutable. Index alignment — matching a right-hand-side Series to rows by label. rename / drop — column-hygiene methods that return a new DataFrame by default. Feature engineering — creating informative columns for analysis or modelling.
Recommended Resources
• Docs: the pandas 'Essential basic functionality' guide, especially the sections on assign and on vectorized operations. • Read: the NumPy np.where reference to see how the condition/true/false arrays broadcast together. • Practice: take a raw orders table and engineer three columns — a ratio, a conditional flag, and a binned band — using arithmetic, np.where, and pd.cut respectively. • Next in DSM: your columns are only trustworthy if the data behind them is complete — next, in Handling Missing Data, you'll detect NaNs and decide between dropping and filling them.
Recap
✓ Add or overwrite a column by assigning to df['name']; a scalar broadcasts to every row. ✓ Derive numeric columns with vectorized arithmetic across existing columns. ✓ np.where builds conditional columns; pd.cut bins a numeric column into labelled ranges. ✓ df.assign() returns a new DataFrame — ideal for immutable chains, and a lambda can reference columns added earlier in the same call. ✓ Vectorize instead of looping rows — it's far faster and clearer. ✓ rename, drop, and reordering keep the column schema clean. Next up: Handling Missing Data. You can build any column you need — but real columns arrive with gaps. Next you'll detect missing values and choose deliberately between dropping and filling them.
Run your code to see the output here.