If NumPy arrays are the engine, DataFrames are the car. Every data scientist lives in DataFrames. This is the single most important data structure you'll learn — let's make it completely intuitive.
What you'll learn
What
A pandas DataFrame is a 2-dimensional, labeled data structure — like a spreadsheet or SQL table, but programmable. It has rows (observations) and columns (features), and every column can have its own data type.
Why
Raw data arrives as CSVs, databases, or APIs. DataFrames give you a unified interface to load, inspect, clean, filter, aggregate, and analyze it — all in Python.
Where it's used
DataFrames are used at every step of a data science project: loading raw data, EDA, feature engineering, model input preparation, and result presentation.
Where this runs in production
Data scientists load millions of listing rows into DataFrames, filter by city and property type, and compute median prices per neighbourhood to train their dynamic pricing model.
Transaction DataFrames are filtered for anomalies (unusual amount, new merchant, foreign IP) using boolean masks — all in pandas before the ML model sees the data.
COVID-19 country-level data was cleaned and aggregated using pandas DataFrames by researchers worldwide to produce the charts you saw in the news.
Excel is powerful for interactive analysis but breaks for large datasets (row limits, no reproducibility, no version control). A DataFrame is the same idea — rows and columns of data — but it lives in code, handles millions of rows instantly, and every transformation you do is a reusable, auditable script.
import pandas as pd
# Create from a dictionary — keys become column names
data = {
'name': ['Alice', 'Bob', 'Carol', 'Dave'],
'age': [28, 34, 29, 42],
'salary': [72000, 88000, 65000, 105000],
'remote': [True, False, True, True]
}
df = pd.DataFrame(data)
print(df)The dictionary keys become column headers. The lists become the column values. pandas automatically creates a default integer index (0, 1, 2, 3) as the row labels.
df.head(n) — show first n rows (default 5). df.info() — column names, non-null counts, dtypes, memory usage. df.describe() — count, mean, std, min, percentiles, max for every numeric column.
# Column selection ages = df['age'] # Returns a Series subset = df[['name', 'salary']] # Returns a DataFrame (list of cols) # Row selection by label row0 = df.loc[0] # Row with index label 0 alice = df.loc[0, 'name'] # Single cell: row 0, col 'name' # Row selection by position first_row = df.iloc[0] # First row (position 0) cell = df.iloc[0, 2] # Row 0, column 2 (salary)
Single brackets + one string → Series. Double brackets + list → DataFrame. This distinction trips up beginners — remember: [[ ]] keeps the column structure intact.
If you do df2 = df[df['age'] > 30] and then df2['salary'] = 99, you'll get a warning. pandas can't tell if df2 is a copy or a view. Always use .copy() when you want an independent DataFrame: df2 = df[df['age'] > 30].copy()
Click any component to understand its role in the DataFrame structure.
You've just received a sales dataset. Before doing anything, inspect it.
Import pandas with its universal alias 'pd'. This is the first line of virtually every data script.
Your task
You've been given a DataFrame of student exam results. Complete the four analysis tasks below.
High achievers: ['Alice', 'Carol', 'Eve'] Best subject: math
Write your solution in the editor on the right, then hit Run.
What does `df[['name']]` (double brackets) return compared to `df['name']` (single bracket)?
Which method gives you the count of missing values per column?
What is the difference between `.loc[]` and `.iloc[]`?
What is the pandas-idiomatic fix?
Given the DataFrame below, find the department with the highest total bonus payout. Print just the department name. df = pd.DataFrame({'dept':['Eng','HR','Eng','Sales','HR','Sales','Eng'], 'salary':[90,60,95,70,65,80,85], 'bonus_pct':[0.10,0.05,0.12,0.08,0.05,0.09,0.11]}) Hint: Calculate bonus = salary * bonus_pct, then groupby dept and sum.
What is a pandas DataFrame, and how does it relate to a Series?
A DataFrame is a 2-dimensional labeled data structure — rows and columns, like a programmable spreadsheet or SQL table. Each column is a pandas Series: a 1D labeled array backed by a NumPy array. All the columns share a single row Index, so you can think of a DataFrame as a dictionary of Series with a common index. When I select one column with df['col'] I get a Series back; selecting a list of columns with df[['a', 'b']] returns a DataFrame.
When would you use .loc[] versus .iloc[], and what happens when the index is not the default 0, 1, 2… range?
I use .loc[] for label-based selection and .iloc[] for integer position-based selection. With the default RangeIndex the two look interchangeable, which hides the difference — but once the index is dates, IDs, or a shuffled subset after filtering, they diverge. df.loc[5] finds the row whose index label is 5, wherever it sits; df.iloc[5] returns the sixth row regardless of its label. A classic bug is filtering a DataFrame and then calling .loc[0] expecting the first row — if row 0 was filtered out, that raises a KeyError. When I want 'the first row of whatever this is', I reach for .iloc[0]; when I mean 'the row for this specific key', I use .loc[].
A colleague loops over DataFrame rows with iterrows() to compute a derived column and it's slow. Why, and what would you do instead?
iterrows() runs a Python-level loop and materialises each row as a Series, so every iteration pays interpreter and object-creation overhead — on a million rows that's a million round trips out of the fast C layer. Vectorized column arithmetic like df['price'] * (1 - df['discount']) performs the whole computation inside NumPy's compiled loops, typically hundreds of times faster. My hierarchy is: vectorized column operations first, then NumPy functions like np.where or np.select for conditional logic, and .apply() only when the logic genuinely can't be vectorized — and even then I treat it as a flag to reconsider the approach. Row loops in pandas are almost always a design smell, not a performance tuning problem.
Common Mistakes to Avoid
1) Confusing df['col'] with df[['col']] — the first returns a Series, the second a one-column DataFrame; passing the wrong one to a function expecting the other causes shape errors. 2) Combining boolean conditions with Python's `and`/`or` instead of `&`/`|` — pandas raises 'The truth value of a Series is ambiguous'. Use & and | with each condition wrapped in parentheses. 3) Modifying a filtered slice without .copy() — df2 = df[mask] then df2['x'] = 1 triggers SettingWithCopyWarning and may silently not write where you think. 4) Calling .loc[0] after filtering and expecting the first row — the label 0 may be gone; use .iloc[0] for positional access. 5) Trusting read_csv dtypes blindly — a numeric column with one stray text value loads as object (strings), and every arithmetic operation on it breaks downstream.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'ELI5: what is the difference between a DataFrame and a Series?' • 'Quiz me on .loc vs .iloc with tricky index examples.' • 'Show me a real bug caused by forgetting .copy() on a filtered DataFrame.' • 'Explain boolean masking with a fresh analogy.' • 'Interview mode: ask me to describe my first steps after loading an unfamiliar CSV.'
Glossary
DataFrame — a 2D labeled table of rows and columns; the core pandas structure. Series — a 1D labeled array; every DataFrame column is one. Index — the row labels shared by all columns of a DataFrame. dtype — the data type of a column (int64, float64, object, bool, datetime64, category). Boolean mask — a Series of True/False values used to filter rows. .loc[] — label-based row/column selection. .iloc[] — integer position-based selection. Vectorized operation — an operation applied to a whole column at once inside NumPy's compiled code, instead of a Python loop. pd.read_csv() — the function that loads a CSV file into a DataFrame. SettingWithCopyWarning — pandas' warning that you may be writing to a temporary copy instead of the original DataFrame.
Recommended Resources
• Docs: the pandas User Guide chapter '10 minutes to pandas' — the official quick tour of DataFrames from the source. • Read: 'Indexing and selecting data' in the pandas docs for the full .loc/.iloc story, including the chained-indexing trap. • Practice: load any CSV you can find (bank statement export, Kaggle dataset) and run .shape, .head(), .info(), .describe() until the inspection ritual is automatic. • Next in DSM: you've met the DataFrame — next you'll zoom into its building block in Series & Index, where index alignment explains half of pandas' surprising behaviours.
Recap
✓ A DataFrame is a 2D labeled table; every column is a Series sharing one row Index. ✓ Inspect every new dataset with .shape, .head(), .info(), and .describe() before touching it. ✓ df['col'] returns a Series; df[['col']] (a list of names) returns a DataFrame. ✓ .loc[] selects by label, .iloc[] by integer position — they differ whenever the index isn't 0, 1, 2…. ✓ Filter rows with boolean masks, combining conditions with & and | (each in parentheses). ✓ Prefer vectorized column operations over row loops — and use .copy() when you need an independent slice. Next up: Series & Index. Every DataFrame column you touched today is a Series — next you'll work with them directly and see how index alignment decides what happens when two Series meet in one expression.
Run your code to see the output here.