Your groupby says the company operates in 47 cities. It operates in 30. The other 17 are ' Mumbai', 'mumbai', 'MUMBAI', 'Bengaluru '… — invisible whitespace and casing splitting real categories into phantom ones. Every count, join, and chart built on that column is wrong, and no amount of statistics downstream can fix what a .str.strip() would have.
What you'll learn
What
String cleaning is normalising text columns with pandas' .str accessor — vectorised versions of Python's string methods (strip, lower, title, replace) plus pattern tools (contains, extract, split) and value mapping. It turns free-form text into consistent, groupable, joinable categories.
Why
Text is where inconsistency hides. Numbers are either equal or not; strings have casing, whitespace, abbreviations, and typos that make equal things unequal. Since groupby, merge, and value_counts compare strings exactly, unnormalised text corrupts every one of them silently.
Where it's used
Standardising categories before any groupby or join, cleaning survey free-text, parsing product codes and addresses, preparing text features for models, and matching entities across systems that spell them differently.
Where this runs in production
'123 Main St.', '123 main street', and '123 MAIN ST' must resolve to one place. Geocoding pipelines lowercase, strip, and expand abbreviations before matching addresses to locations.
Marketplace sellers type product data by hand; catalogue pipelines normalise casing, strip promotional noise, and extract attributes like size and colour from titles so search and dedup work.
'Sr. SWE', 'Senior Software Engineer', and 'senior software eng' are one role. Mapping raw titles onto a canonical taxonomy powers salary insights and recruiter search.
s = pd.Series([' Mumbai', 'mumbai ', 'MUMBAI', None]) print(s.str.strip().str.lower().tolist()) # ['mumbai', 'mumbai', 'mumbai', None]
Chaining .str methods is the idiom: strip whitespace, then normalise case. Three phantom cities collapse into one, and the None survives as None — .str methods are null-safe.
Nearly every text column benefits from .str.strip() (remove leading/trailing whitespace) followed by a casing rule — .str.lower() for matching keys, .str.title() for display labels. Do this BEFORE deduplication, groupbys, or joins: it's the single highest-value line of cleaning in most pipelines.
# Fix known variants after normalising
df['city'] = df['city'].str.strip().str.lower()
df['city'] = df['city'].replace({'bombay': 'mumbai', 'bengaluru': 'bangalore'})
# Whole-value mapping with a dict (unmapped -> NaN, so it doubles as validation)
df['tier'] = df['plan'].map({'free': 0, 'pro': 1, 'enterprise': 2})Two different tools: Series.replace substitutes listed values and leaves everything else alone — right for fixing known aliases. Series.map sends every value through the dict and turns unmapped ones into NaN — right for controlled vocabularies, where a new NaN means an unexpected label snuck in.
mask = df['comment'].str.contains('refund', case=False, na=False)
refund_rows = df[mask]
# Splitting structured text into columns
parts = df['sku'].str.split('-', expand=True) # 'TV-55-BLK' -> 3 columns
# Regex extraction with named groups
size = df['product'].str.extract(r'(?P<size>\d+)\s?(inch|in)')['size']contains filters, split(expand=True) explodes delimited text into a DataFrame, and extract pulls regex groups into columns. Together they recover the structure hiding inside 'one big text field' data.
A warehouse where every worker handwrites labels gets 'Blu Shirt L', 'blue shirt (large)', and 'SHIRT-BLUE-L' for the same product — and the inventory count is fiction. String cleaning is pasting a printed barcode over each handwritten label: same information, one canonical form, and suddenly scanners (groupby, merge) agree with reality.
Lowercasing collapses 'US' (country) into 'us' (pronoun); stripping punctuation merges 'C++' into 'C'. Aggressive normalisation is lossy — apply it to a copy or a matching key column, keep the original for display, and always compare nunique() before and after to see exactly how many categories each step merged. If a step merges more than you can explain, it merged too much.
Click each stage — order matters, and each step is measurable with nunique().
One city, four spellings — make them one string.
Four values pandas sees as distinct: hidden spaces plus three casings.
Your task
A product feed has messy category labels. Normalise them, fix a known alias, then count how many products mention 'wireless' in their name (any casing, nulls present).
1 ['electronics'] 2
Write your solution in the editor on the right, then hit Run.
What does df['city'].str.lower() do with NaN values in the column?
A category column should contain exactly {'basic','pro'}, and you want any other value exposed as NaN for review. Which tool does this directly?
df[df['note'].str.contains('urgent')] raises 'cannot mask with non-boolean array'. The likely cause?
What's the right fix?
Extract the numeric size from each product name into an integer column and print its values as a list. df = pd.DataFrame({'product': ['Monitor 27 inch', 'TV 55 inch', 'Monitor 24 inch']})
Why is it good practice to keep the original text column and put normalised values in a new column (or a copy)?
A categorical column has far more distinct values than it should. Walk me through cleaning it.
I start by measuring: nunique() against the expected count, and value_counts() to see the long tail — that tail is where the variants live. Then the standard pipeline, verifying nunique after each step. First .str.strip() for whitespace, including internal runs via a regex replace if needed. Then one casing rule, usually .str.lower(). Those two steps typically collapse most phantom categories. Next, known aliases and abbreviations with replace({'bombay':'mumbai'}) — domain knowledge encoded as a dict. If the column has a closed vocabulary, I finish with map over the full vocabulary so anything unexpected becomes NaN and surfaces for review rather than hiding. Throughout, I clean into a matching column and keep the raw one, because normalisation is lossy. What survives after all that is genuine typos, which are a fuzzy-matching problem — a deliberate escalation, not the default.
What are the sharp edges of str.contains that bite people in production?
Three big ones. First, missing values: contains returns NaN for them, so the 'boolean' mask isn't boolean and df[mask] raises — na=False (or na=True, chosen deliberately) is effectively mandatory on real data. Second, the pattern is regex by default: searching for 'C++' or a phone number with '+' explodes or matches wrongly unless you pass regex=False or escape the pattern — a literal search should say regex=False explicitly. Third, casing: contains is case-sensitive by default, and real text isn't consistent, so case=False is usually what the question actually means. There's also a subtler design point — contains does substring matching, so 'art' matches 'smart'; when I need whole-word or full-value matching I use fullmatch or word-boundary regex instead. My habit is to write contains(pattern, case=False, na=False, regex=...) with every argument explicit.
You need to merge two customer tables from different systems on names, and exact matching loses a third of the rows. What's your strategy?
First I exhaust cheap, exact normalisation on BOTH keys, because most 'fuzzy' problems are actually systematic: strip whitespace, lowercase, collapse internal spaces, remove punctuation, expand known abbreviations, and standardise ordering if names appear as 'Last, First'. I re-measure the match rate after each step — typically that recovers most of the gap. For what remains, I escalate to fuzzy matching with edit-distance or token-based similarity, but never on its own: 'Iran'/'Iraq' are one edit apart, so I pair the similarity score with confirming evidence like a shared postcode, phone, or birth date, and route mid-confidence pairs to human review rather than auto-merging. And I treat this as record linkage with an audit trail — every merged pair is logged with its evidence — because a wrong merge on customer data (billing the wrong person) is worse than a missed match.
Common Mistakes to Avoid
1) Forgetting the .str prefix — df['col'].lower() raises; the accessor is what vectorises. 2) Skipping strip() and hunting exotic bugs when the culprit is a trailing space. 3) str.contains without na=False on columns with nulls — the mask stops being boolean. 4) Forgetting contains treats patterns as regex — '+' and '(' need regex=False or escaping. 5) Normalising only ONE side of a merge key. 6) Overwriting the raw column before verifying what each step merged — keep an original for audit. 7) Jumping to fuzzy matching before exhausting exact normalisation, or trusting fuzzy scores without confirming evidence.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'ELI5: why do " Mumbai" and "mumbai" break my groupby?' • 'Show me replace vs map on the same messy column and when each wins.' • 'Quiz me on str.contains gotchas: nulls, regex, casing.' • 'Walk me through extracting sizes from product names with a regex, group by group.' • 'Interview mode: challenge my plan for matching customer names across two systems.'
Glossary
.str accessor — gateway to vectorised, null-safe string methods on a Series. strip()/lower()/title() — whitespace and casing normalisers. replace (Series) — substitute listed values, leave the rest. map — send every value through a dict/function; unmapped become NaN. str.contains — boolean mask for substring/regex match (na=, case=, regex= flags). str.split(expand=True) — split delimited text into columns. str.extract — pull regex capture groups into columns. Normalisation — reducing variants to one canonical form. Fuzzy matching — matching by similarity (edit distance) instead of equality.
Recommended Resources
• Docs: pandas user guide 'Working with text data' — the full .str method catalogue. • Read: a regex refresher (regex101.com lets you test patterns interactively against sample values). • Practice: grab any survey export with a free-text column, run the strip→lower→replace pipeline, and chart nunique() after each step. • Next in DSM: text is consistent, types are right, rows are unique — the last cleaning question is what to do with extreme VALUES: Outlier Detection.
Recap
✓ The .str accessor vectorises string methods across a Series and passes NaN through safely. ✓ Default first pass on any text column: .str.strip() then a single casing rule — measure with nunique() before and after. ✓ replace fixes known aliases and leaves the rest; map enforces a closed vocabulary, exposing surprises as NaN. ✓ str.contains needs deliberate na=, case=, and regex= arguments on real-world data. ✓ split(expand=True) and extract recover structured columns from composite text. ✓ Normalisation is lossy — keep the raw column, clean into a key column, and escalate to fuzzy matching only with confirming evidence. Next up: Outlier Detection. Your strings are canonical and your types are true — the final cleaning question is which extreme values are errors, and which are the most important data you have.
Run your code to see the output here.