Every dataset you'll ever analyze starts life in a file: a CSV export, a JSON log, a text dump from some legacy system. pd.read_csv() will eventually do the heavy lifting — but the day the file is malformed, half-encoded, or 40GB, the engineers who understand what's underneath are the ones who fix it. Today you touch the actual bytes.
What you'll learn
What
open() connects your program to a file; mode strings ('r', 'w', 'a') pick read/write/append; the with statement guarantees closing; files iterate line by line; and encoding='utf-8' names how bytes become text. Plus: parsing a CSV by hand, so the format loses its mystery.
Why
File I/O is where programs meet reality — and where resource leaks, silent truncation ('w' on the wrong path!), and encoding mojibake live. The with-statement discipline and encoding awareness you build here transfer directly to databases, sockets, and every pandas read_* call's parameters.
Where it's used
Reading raw data files, writing reports and cleaned outputs, appending to logs, processing files too big for memory, and debugging every 'UnicodeDecodeError' a CSV ever throws at you.
Where this runs in production
Petabytes of playback logs are, at bottom, text lines appended to files — parsed by streaming readers that never load a whole file, exactly the iteration pattern you'll write today.
Public data portals publish tens of thousands of CSVs. Encoding declarations and header rows — the details in this lesson — are the difference between usable open data and mojibake.
encoding=, sep=, header= — every pd.read_csv parameter you'll ever pass corresponds to a by-hand decision you make in this lesson. Knowing the manual version makes the tool debuggable.
with open('report.txt', 'w', encoding='utf-8') as f:
f.write('revenue: $1,234\n')
f.write('orders: 87\n')
with open('report.txt', 'r', encoding='utf-8') as f:
content = f.read()
print(content)The with statement is non-negotiable professional practice: the file CLOSES automatically when the block exits — normally, via return, or via exception (it's the try/finally from two lessons ago, packaged). Unclosed files mean leaked handles and, for writes, buffered data that never hits disk. Note write() does NOT add newlines — you supply \n.
open() is checking a book out of the library: you get exclusive-ish access and the library records the loan. close() is returning it. Forget to return books and the library eventually refuses you new loans (OS file-handle limits are real). The with statement is a librarian who takes the book back the MOMENT you leave the building — even if you leave through the fire exit (an exception). And mode 'w' is asking for the book plus a shredder: if a book by that name exists, it's destroyed the instant your loan starts — before you've written a word.
A file object is iterable, yielding one line per iteration: for line in f: — reading lazily, holding ONE line in memory at a time. That's how you process a 40GB log on an 8GB laptop. f.read() (whole file as one string) and f.readlines() (list of lines) are fine for small files, but the streaming loop is the default posture for data work. Each yielded line KEEPS its trailing \n — line.strip() (or .rstrip('\n')) is the reflexive first touch.
# sales.csv: city,amount\noslo,1250.5\nlima,890.0\ncairo,2100.75
with open('sales.csv', 'w', encoding='utf-8') as f:
f.write('city,amount\noslo,1250.5\nlima,890.0\ncairo,2100.75\n')
total = 0.0
with open('sales.csv', 'r', encoding='utf-8') as f:
header = next(f) # consume the header line
for line in f:
city, amount = line.strip().split(',')
total += float(amount)
print(f'total: {total}')CSV by hand: next(f) advances past the header; each data line strips its newline and splits on the comma; the fields arrive as STRINGS (the Type Conversion lesson's eternal truth) needing float(). This 6-line loop is what read_csv industrializes — with quoting, types, and encodings handled for you.
Files store BYTES; text is an interpretation. encoding='utf-8' names the interpretation — always pass it explicitly, because the default varies by operating system (Windows often cp1252!), and a script that works on your Mac then garbles 'São Paulo' on the Windows server is a rite of passage you can skip. UnicodeDecodeError on read means the file isn't the encoding you claimed — common culprits: Excel exports (cp1252/latin-1) and old systems. Fix by naming the true encoding, not by ignoring errors.
1) Never 'w' a path you didn't verify — one wrong variable and the source data is a zero-byte file; write outputs to a DIFFERENT name than inputs. 2) Appending logs uses 'a' — 'w' would erase history on every run. 3) Don't build gigantic strings then write once; write incrementally. 4) CSV-by-hand splits break on quoted commas ('"Portland, OR"') — the csv module (or pandas) handles quoting; hand-splitting is for learning and truly-simple files. 5) Paths differ across OSes — next lesson (pathlib) fixes the '\\' vs '/' mess properly.
From open() to guaranteed close — on both the happy and the exploding path. Click each node.
The full round trip in eight lines.
'w' creates note.txt (or wipes an existing one). Two writes, each supplying its own \n. Closing (automatic, at dedent) flushes them to disk.
Your task
Write inventory.csv with the given content ('item,qty' header plus 4 rows, one corrupt). Then read it back: skip the header, parse rows with a per-line try/except (count skips), and write low_stock.txt listing every item with qty < 10, one per line as '<item>: <qty>'. Finally print low_stock.txt's content and the skip count.
nut: 4 screw: 7 skipped: 1
Write your solution in the editor on the right, then hit Run.
What does opening an EXISTING file with mode 'w' do?
Why use `with open(...) as f:` instead of f = open(...)?
You must count ERROR lines in a 25GB log on a laptop with 8GB RAM. Which approach works?
What's the likely cause and fix?
Write 'alpha\nbeta\ngamma\n' to words.txt, then read it line by line and print each word uppercased with its length, as '<WORD> (<len>)'. Expected: ALPHA (5) BETA (4) GAMMA (5)
def get_file(): with open('data.txt') as f: return f — a caller then does file.read(). What happens?
grades.csv contains 'name,score\nada,95\nkai,not-found\nmia,88\n' (write it first). Read it, skipping the header; parse scores with try/except (collect failures by name); append — mode 'a' — a line 'checked <n> rows' to the SAME file. Print the parsed pairs, the failures list, then re-read and print the file's last line. Expected: [('ada', 95), ('mia', 88)] ['kai'] checked 3 rows
Why is the with statement the standard way to work with files? What exactly does it guarantee — and not?
with open(...) as f: enters a context manager: __enter__ yields the handle, and __exit__ — guaranteed to run on EVERY exit path (normal completion, return/break, or an exception unwinding through) — closes the file, flushing buffered writes to disk and releasing the OS handle. It's the try/finally cleanup pattern packaged into syntax, which matters because the manual version fails in exactly the case that matters: an exception between open() and close() leaks the handle and can lose buffered data. What it does NOT do: it doesn't suppress exceptions (they propagate after the close — you still need try/except for FileNotFoundError on optional files), and it doesn't keep the handle alive beyond the block — returning f from inside a with hands the caller a closed file (ValueError: I/O operation on closed file); you return the DATA instead. The dunder connection is worth naming in interviews: with works on anything implementing __enter__/__exit__ — files, locks, database transactions — the same protocol thinking as __len__ and __getitem__.
How do you process a file far larger than memory, and which habits follow from that constraint?
Stream it: for line in f: iterates lazily, reading one line at a time — memory use is bounded by the longest line, not the file, so a 40GB log processes on a laptop. The habits that follow: aggregate incrementally (running totals, counting dicts) instead of collecting all rows and post-processing; write output incrementally too, rather than building a giant string; and design per-line error handling (try inside the loop, skip-and-count) since a single corrupt line mustn't kill hour six of a pass. The functions to treat with suspicion are read() and readlines() — both materialize the entire file, fine for configs and small data, fatal at scale; sum-of-generator patterns (sum(float(l) for l in f)) keep even the intermediate list from existing. This streaming posture transfers directly upward: pandas' chunksize parameter, database cursors, and generator pipelines are the same idea with different spelling — and 'how would you count errors in a 25GB log on an 8GB machine?' is a stock screening question aimed precisely at whether you reach for read().
A CSV read is producing garbled characters (mojibake). Walk me through your diagnosis.
Mojibake means the bytes were decoded with the wrong encoding — the file stores one interpretation, the reader assumed another. First, look at the damage pattern: 'é' where 'é' belongs is the signature of UTF-8 bytes decoded as cp1252/latin-1 (each multi-byte UTF-8 character splits into two wrong-but-printable characters); a UnicodeDecodeError with 'invalid continuation byte' is the reverse — cp1252 content read as UTF-8. Second, identify the true source encoding: Excel exports and legacy Windows systems typically produce cp1252; most modern systems produce UTF-8; a BOM (bytes EF BB BF) suggests utf-8-sig. Tools like chardet, or reading the first KB in binary and inspecting, settle it. Third, fix by declaring the truth — open(path, encoding='cp1252') or the equivalent pd.read_csv(encoding=...) — and normalize to UTF-8 on write so the problem dies at your boundary. The habit that prevents the whole class: always pass encoding explicitly, because the default follows the local platform and creates works-on-my-machine bugs. errors='replace' exists for salvage jobs but is data loss by policy — name it as a last resort, never a fix.
Common Mistakes to Avoid
1) Mode 'w' on a file you meant to append to (or worse, on your input path) — instant truncation. 2) Skipping with — leaked handles, unflushed writes. 3) f.read() on huge files — stream with for line in f. 4) Forgetting lines keep their \n — strip before comparing or measuring. 5) Omitting encoding= — platform-dependent mojibake. 6) Returning the file handle from inside a with — it arrives closed; return the data. 7) write() without \n — everything lands on one line. 8) Hand-splitting CSVs with quoted commas — that's the csv module's job.
Ask the AI Tutor
Try these prompts in the AI Tutor panel: • 'Quiz me: which mode for each of these six file tasks?' • 'Walk me through what with guarantees when an exception fires mid-read.' • 'Show mojibake: the same bytes under utf-8 vs cp1252.' • 'Help me refactor this read()-based script to stream.' • 'Interview mode: ask me the 25GB-log question and grade my answer.'
Glossary
open(path, mode, encoding) — connect to a file. Mode 'r'/'w'/'a' — read / write-truncate / append ('b' suffix = binary). with — context manager; guarantees close on every exit. File handle — the OS connection object; a finite resource. Streaming — for line in f: one line in memory at a time. Truncate — 'w' zeroing an existing file at open. Flush — buffered writes reaching disk (close does it). Encoding — the bytes↔text interpretation; always name utf-8 explicitly. Mojibake — text decoded with the wrong encoding ('é'). UnicodeDecodeError — bytes don't fit the claimed encoding. next(f) — consume one line (headers). CSV — comma-separated text; hand-split only when unquoted.
Recommended Resources
• Docs: 'Reading and Writing Files' in the Python tutorial (7.2), and the csv module intro for when hand-splitting stops sufficing. • Read: the Unicode HOWTO's first half — the bytes-vs-text model that ends encoding confusion permanently. • Practice: export any spreadsheet as CSV and write a streaming reader that aggregates one column, with skip-and-count on the rows that surprise you (some will). • Next in DSM: your file code still says 'C:\\data' or '/home/data' by hand — Working with Paths brings pathlib: portable, composable, glob-able paths, and the module that ends string-surgery on filenames.
Recap
✓ open(path, mode, encoding='utf-8') — 'r' reads, 'w' TRUNCATES, 'a' appends. ✓ with guarantees closing on every exit path — non-negotiable; return data, not handles. ✓ Files iterate lazily: for line in f handles any size; strip the \n first. ✓ CSV by hand: next(f) past the header, split(','), convert types, per-line try. ✓ Always name the encoding — defaults vary by OS; mojibake means the claim was wrong. ✓ Outputs get their own paths; logs use 'a'; report skips, never swallow them. Next up: Working with Paths. Hard-coded path strings break across machines and OSes — pathlib gives you portable path objects, existence checks, directory listing, and glob patterns: the last piece of file fluency.
Run your code to see the output here.