A column of product codes: 'SKU-4821 (warehouse B)', 'sku 302 - clearance', 'Item# 77'. Your .split() and .replace() skills — mighty as they are — hit their ceiling here: the structure varies. Regular expressions describe the PATTERN ('some digits after some letters') instead of exact characters, and suddenly all three rows yield their numbers to one line of code.
What you'll learn
What
Regex: a mini-language for describing text patterns. Python's re module applies them — re.search finds a match, re.findall extracts every match, re.sub rewrites them. Character classes (\d digit, \w word char, \s space), quantifiers (+ one-or-more, * zero-or-more, ? optional), and capture groups (parentheses) that pull out exactly the pieces you want.
Why
Real text data is semi-structured: log lines, product descriptions, addresses, survey answers, scraped HTML. Extracting the phone numbers, prices, codes, and dates buried inside is a daily data task — and it's regex or a hundred lines of brittle string surgery. pandas exposes the same engine as .str.extract/.str.contains, so this skill transfers directly to DataFrames.
Where it's used
Log parsing, data cleaning (normalizing phone numbers, stripping units), feature extraction from text, web scraping, validation (is this an email?), and the pandas .str accessor family.
Where this runs in production
Google built the RE2 regex engine to safely run user-supplied patterns over planet-scale logs — pattern extraction is so central to observability that the engine itself became infrastructure.
Logstash's grok — the standard way logs become structured events — is a library of named regular expressions; every %{IPV4:client} you see expands to a regex capture group.
The data-cleaning tool beloved by journalists and archivists leans on regex transforms to normalize the chaos of hand-entered data — dates, names, addresses — exactly this lesson's use case.
import re text = 'Order 4821 shipped to Berlin, order 302 pending' print(re.findall(r'\d+', text)) # ['4821', '302'] print(re.search(r'\d+', text).group()) # '4821' (first match only) print(re.sub(r'\d+', '#', text)) # digits replaced
The three verbs: findall EXTRACTS every match into a list (the data-work workhorse), search finds the FIRST match (None if absent — check before .group()!), sub REWRITES matches. Note findall returns strings — int() them yourself, a type-conversion lesson callback.
+ one or more, * zero or more, ? zero or one (optional), {3} exactly three, {2,4} two to four. They attach to the item just before them: \d+ is 'digits, at least one'; colou?r matches color AND colour; \d{4} is exactly four digits (years!). Default matching is GREEDY — .* grabs the longest possible stretch; adding ? (.*?) makes it lazy, taking the shortest. The greedy-vs-lazy distinction matters the moment two delimiters appear on one line.
import re
log = '2026-07-16 ERROR db timeout; 2026-07-15 INFO ok'
dates = re.findall(r'\d{4}-\d{2}-\d{2}', log)
print(dates) # ['2026-07-16', '2026-07-15']
levels = re.findall(r'[A-Z]{4,5}', log)
print(levels) # ['ERRO', 'INFO'] -- oops!
print(re.findall(r'\b(?:ERROR|INFO|WARN)\b', log)) # ['ERROR', 'INFO']\d{4}-\d{2}-\d{2} reads as the ISO date shape it matches. The second attempt shows a classic miss: {4,5} grabbed 4 letters of ERROR. Fixes: alternation (A|B|C) lists literal options; \b is a word BOUNDARY pin; (?:...) groups without capturing. Regexes fail like this — quietly, plausibly — so always test on sample data and check the output.
[abc] matches ONE character from the set; [a-z0-9] uses ranges; [^0-9] negates (any char EXCEPT digits). Anchors pin position: ^ start of string, $ end. r'^\d{3}$' matches strings that are ENTIRELY three digits — the validation pattern. Without anchors a pattern matches anywhere inside: r'\d{3}' happily finds '123' inside 'x12345'. Validation = anchored (re.fullmatch is the explicit tool); extraction = unanchored.
Parentheses capture pieces: r'(\w+)@(\w+\.\w+)' against 'ada@example.com' captures 'ada' and 'example.com'. With ONE group, findall returns a list of that group's values; with SEVERAL, a list of TUPLES — rows, ready to become dicts or a DataFrame. m.group(1), m.group(2) read them from a search match (group(0) or group() is the whole match). Named groups (?P<user>\w+) document intent, readable back as m.group('user').
String methods cut text at exact marks — split at this comma, replace this word — like scissors following a drawn line. A regex is a STENCIL: it describes a shape ('four digits, dash, two digits'), and wherever you press it against the text, matching shapes show through. Capture groups are windows cut INTO the stencil: press it once and the interesting pieces (the area code, the domain, the price) appear in their own labeled openings. You stop caring what surrounds the shape — the stencil finds it in any mess.
1) Fixed, simple structure → string methods win: 'a,b,c'.split(',') beats a pattern for clarity (the csv module beats both for real CSV with quoted fields). 2) Nested structure (HTML, JSON, code) → use a real parser; regex fundamentally cannot balance nested tags, and trying is a rite of passage everyone should fail exactly once. 3) A regex that works on your three test lines is not done — messy data holds edge cases (empty fields, unicode, doubled delimiters); test on real samples and keep the skip-and-count habit. 4) Readability collapses fast: a 200-character pattern is write-only code — break it up, comment it, or use several simpler passes.
r'(\w+)-(\d{4})' pressed against 'SKU-4821' — each piece has a job. Click through.
Pull every number out of a sentence of mixed text.
\d+ = runs of digits. findall returns every match as STRINGS — the int() conversion is explicit, and sum() closes the loop. Two lines from prose to arithmetic.
['3', '45', '2', '30'] 80
Your task
Inventory lines look like 'WIDGET-042: 15 units @ €3.50' with inconsistent casing and spacing. Extract the product code (letters, dash, digits), the unit count, and the price into tuples using one pattern with three capture groups (case-insensitive for the code letters). Skip lines that don't match, counting them. Print each (CODE, units, price) with the code uppercased, units as int, price as float, then the skip count and total inventory value (units × price summed, 2 decimals).
('WIDGET-042', 15, 3.5)
('GADGET-007', 3, 12.0)
('SPROCKET-113', 40, 0.75)
skipped: 1
total value: 118.50Write your solution in the editor on the right, then hit Run.
What does re.findall(r'\d+', 'a12b345c') return?
Why write patterns as raw strings — r'\d+' not '\d+'?
re.search(r'(\w+)@(\w+)', 'contact: ada@example') — what is .group(1) vs .group(0)?
What's wrong?
From text = 'IDs: A-101, B-22, C-3034 (legacy: 999)', extract every letter-dash-digits code with a pattern capturing letter and number separately, printing '<letter>: <number>' per match. Expected: A: 101 B: 22 C: 3034
You need to pull the src values out of a page of nested HTML with mixed quoting and attributes in any order, robustly. Best tool?
Normalize messy dates to ISO: dates = ['16/07/2026', '3/9/2026', '2026-01-05 already iso']. For DD/MM/YYYY or D/M/YYYY shapes, use re.sub with a pattern capturing day, month, year and a replacement reordering them zero-padded via a function (lambda m: f'{m.group(3)}-{int(m.group(2)):02d}-{int(m.group(1)):02d}'). Leave non-matching text unchanged. Expected: 2026-07-16 2026-09-03 2026-01-05 already iso
When do you reach for regex versus string methods versus a parser? Give the decision rule and examples.
Three tiers by how much STRUCTURE varies. String methods when the structure is fixed and simple: splitting 'a,b,c' on commas, stripping whitespace, checking a literal prefix — split/strip/replace/startswith are clearer, faster, and can't misfire on metacharacters (though real CSV with quoted fields graduates to the csv module). Regex when the structure is a PATTERN with variation: 'some digits after SKU-', dates in two formats, prices with optional decimals, log lines with consistent shape but variable content — this is regex's home turf, extraction and cleaning of semi-structured text. A real parser when the structure NESTS or is a standard format: HTML/XML (BeautifulSoup — regex formally cannot balance nested tags), JSON (json module), CSV with quoting/escaping (csv module), code (ast). The rule of thumb I apply: start with string methods; graduate to regex when I catch myself chaining three splits and an if to express 'a pattern'; graduate to a parser when I catch my regex trying to understand nesting or re-implement a spec. And within regex use, keep patterns modest — two readable passes beat one 200-character write-only pattern, and every pattern gets tested against real messy samples, not just the happy path.
Explain capture groups and how findall's return type changes with them — with the data-work implications.
Parentheses in a pattern do two jobs: they group (so a quantifier or alternation applies to several tokens) and they CAPTURE (the engine records what that region matched). Numbered by opening parenthesis left to right; m.group(1) reads the first from a match object, m.group(0) is the whole match; named versions (?P<price>\d+) read as m.group('price') and document intent. findall's return type follows the group count — no groups: list of whole-match strings; exactly one group: list of THAT GROUP's strings (the whole match is discarded — surprising the first time, exactly what extraction wants after that); two or more: list of TUPLES, one element per group. The data-work implication: findall with several groups converts semi-structured text straight into rows — [('2026-07-16', 'ERROR'), ...] is ready to iterate with tuple unpacking, load into dicts, or feed pd.DataFrame. Two practical notes: when you need grouping WITHOUT capturing (alternation like (?:ERROR|WARN)), the ?: keeps the output tuples clean; and everything captured is a STRING — int()/float() conversion is a deliberate second step, per the type-conversion discipline.
A teammate's regex works on their test cases but corrupts production data — matches that grab too much text. What's the likely cause and your review checklist?
The likely cause is GREEDY quantifiers meeting repeated delimiters: .* and .+ match the LONGEST possible stretch, so r'<(.*)>' against 'a <b> c <d>' captures 'b> c <d' — one giant match spanning from the first < to the LAST >, not two small ones. Test data with one delimiter pair per line hides this; production lines with two pairs expose it. Fixes in preference order: make the quantifier lazy (.*? takes the shortest match), or better, replace dot-star with a NEGATED CLASS stating what the content can't contain — r'<([^>]*)>' ('anything but >') is faster, clearer, and immune to the problem. My review checklist for any pattern heading to production: 1) every .* or .+ justified or replaced with a negated class; 2) literal specials escaped (\$ \. \( — an unescaped dot 'works' until '3x14' matches the pi pattern); 3) anchored where validation is intended (fullmatch), unanchored where extraction is; 4) word boundaries (\b) where substrings could false-positive ('cat' in 'concatenate'); 5) tested against adversarial real samples — empty fields, doubled delimiters, unicode; 6) matches that fail are counted and reported, not silently dropped. Greedy-match bugs are the regex equivalent of the axis bug in NumPy: no error, plausible output, wrong data.
Common Mistakes to Avoid
1) Forgetting the r prefix — some patterns break mysteriously without it; use it always. 2) Unescaped specials: $ . ( ) + ? match nothing or the wrong thing silently — escape literals. 3) .group() on a None search result — guard with if m first. 4) Greedy .* spanning multiple delimiters — use .*? or a negated class [^>]*. 5) Validation without anchors — 'abc123xyz' passes r'\d+'; fullmatch or ^...$ for whole-string checks. 6) Expecting findall to return numbers — everything is strings until you convert. 7) Parsing nested HTML/JSON with regex — parsers exist; use them. 8) Shipping a pattern tested only on clean samples — messy data is the job.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'Show me text and ask me to write the extraction pattern — escalate difficulty as I go.' • 'Explain greedy vs lazy with <tag> examples until I can predict both.' • 'Take my pattern and try to break it with adversarial inputs.' • 'When would .str.extract in pandas replace this loop? Show the translation.' • 'Interview mode: ask me the regex-vs-parser decision question and grade my answer.'
Glossary
regex — a pattern language for text shapes. r'...' — raw string; backslashes pass through to the engine. \d \w \s — digit / word char / whitespace classes (capitals negate). [abc] [a-z] [^x] — custom sets, ranges, negation. + * ? {n} {m,n} — quantifiers (one+, zero+, optional, counts). greedy/lazy — longest vs shortest match (? suffix = lazy). ^ $ \b — start, end, word-boundary anchors. (…) — capture group; (?:…) non-capturing; (?P<name>…) named. re.search — first match or None. re.findall — all matches (tuples when 2+ groups). re.sub — replace matches (accepts a function). re.fullmatch — anchored whole-string validation. re.compile — pre-parse a reused pattern.
Recommended Resources
• Docs: the re module's HOWTO ('Regular Expression HOWTO') — the gentlest official on-ramp. • Tool: regex101.com — paste pattern + sample text, watch matches and groups live with explanations; the fastest way to debug any pattern. • Practice: extract every number, date, and capitalized name from a news article's text — three patterns, one afternoon. • Next in DSM: every tool is on the belt — the module capstone, Project: Build a Data Pipeline, chains files, parsing, error handling, dates, and regex into one end-to-end program.
Recap
✓ Patterns describe SHAPE: \d \w \s classes, [sets], + * ? {n} quantifiers — always in r'raw strings'. ✓ Three verbs: search (first match or None — guard it), findall (extract all), sub (rewrite; takes functions). ✓ Capture groups turn matches into fields; 2+ groups make findall return row-ready tuples — all strings until you convert. ✓ Escape literal specials (\$ \.) — unescaped ones fail silently; prefer negated classes over greedy .*. ✓ Validation anchors (fullmatch, ^ $); extraction doesn't. ✓ String methods for fixed structure, regex for patterns, real parsers for nesting — and test on messy samples. Next up: Project — Build a Data Pipeline in Python. The capstone: read raw files, clean with regex and conversions, handle errors with skip-and-count, date-stamp outputs with pathlib — every module, one program.
Run your code to see the output here.