Generators & lazy evaluation

Replace 'return' with 'yield' in a function and you get a generator. That swap unlocks one of Python's most powerful patterns — lazy, paused, memory-bounded sequences.

A regular function runs to completion and returns a single value. A generator function, when called, doesn't actually run the body. It returns a generator object. The body runs piece by piece, pausing at each yield, resuming when next() is called on the generator.

The practical effect — you can produce a sequence of values one at a time, holding only the current value in memory, and the consumer can stop anywhere without ever materialising the rest.

Three consequences this enables.

Stream a 100GB file line by line. The 'for line in f' pattern is a generator under the hood — Python's file objects implement __iter__ as a generator. Memory stays low; processing happens chunk by chunk.

Iterate infinite sequences. A generator can yield forever. itertools.count() yields 0, 1, 2, 3, … with no upper bound. Take what you need; stop when you want.

Build composable pipelines. Generators chain. read → parse → filter → transform — each stage a generator, each stage processing one item at a time, total memory bounded by the largest intermediate item.

Most Python libraries you respect hide a generator inside. requests' iter_content() streams response bodies. PyTorch's DataLoader is generator-shaped. TensorFlow's tf.data is generator-shaped. SQLAlchemy's query iteration is generator-shaped.

The trick to thinking in generators is to ask: 'do I need all the values at once, or just one at a time?' If 'one at a time', use a generator. If 'all at once for some reason' — start with a generator anyway and convert to a list at the end if needed. Lazy first, eager last.

Yesterday we covered list comprehensions. Today's twist: replace the square brackets with parentheses and you get a generator expression. Same syntax, different semantics, dramatically different memory.

List comp: [x*x for x in big_iter] — builds the entire list in memory. If big_iter has 10 million items, the list has 10 million items.

Generator expression: (x*x for x in big_iter) — produces values lazily on demand. Memory cost: O(1). Time cost on creation: also O(1). The work happens as you iterate.

The swap from [] to () is one character. The implications are enormous.

When does it matter?

Logging pipelines processing GB-scale logs. List comp would blow your RAM. Generator expression streams.

Reading large datasets where you only need the values that pass a filter. Generator expression filters lazily — never materialises the rejected items.

Anything piped into a single-pass aggregator like sum(), max(), min(), any(), all(). sum(x*x for x in big_iter) doesn't need to materialise a list — it consumes squares one at a time and accumulates the sum. Constant memory. Fast.

The pattern: when the next step is 'consume one item, do something, produce result', use a generator expression. When the next step is 'reuse the collection multiple times', use a list comp.

Gotcha: generators are single-use. Iterate once, they're exhausted. If you need to iterate twice, store as a list or rebuild the generator.

Gotcha 2: generators don't support indexing or len(). They don't know in advance how many items they'll produce.

Gotcha 3: error handling. An exception in the body of a generator expression surfaces only when you reach the failing item. Easier to debug if you put complex logic in a separate generator function with proper error handling.

Imagine you have a 50GB log file from production. You need to find all lines starting with 'ERROR' and print them. You can't load the file into memory.

Generators handle this elegantly. Look at the snippet — three small generator functions chained together.

lines() opens the file and yields one line at a time. The 'for line in f' is itself a generator (Python file iteration), so lines() is just forwarding. Memory cost: one line at a time.

tokens() takes the line generator and splits each line into lowercase tokens, yielding each token. Memory cost: one line plus its tokens.

The outer for-loop consumes tokens one at a time. Filters by 'startswith('error')'. Prints the matches.

Total memory used to process a 50GB file: roughly the size of one line. Could be a few KB. The OS handles file buffering; Python handles the streaming.

This pattern composes. Add another stage. parse_event(token_iter) → filter_severity(event_iter) → bucket_by_hour(filtered_iter). Each stage processes one item at a time, yields one item at a time. The whole pipeline's memory is bounded by the largest single item, not the whole dataset.

It's not just about huge files. The same pattern applies to:

Streaming network responses (requests.iter_content).
Reading from a database cursor (most ORMs).
Processing telemetry events one at a time.
Iterating tokens through an LLM tokeniser.

When people say 'Python is slow', they often mean 'I wrote eager code that materialised everything at every step'. Generators flip this. Lazy by default. Materialise only at the boundary.

Once you start writing generators, you'll quickly hit a pattern — a generator that delegates part of its work to another generator. The naive way:

def outer():
    for x in inner():
        yield x

This works, but Python has a better way:

def outer():
    yield from inner()

'yield from' delegates to another iterable. It yields every value the inner generator yields. Visually cleaner. Functionally identical for the basic case. But it does more — it correctly forwards .send(), .throw(), and .close() calls to the inner generator. And when the inner generator returns a value (yes, generators can return), 'yield from' captures it.

This matters more than it seems. If you're writing a coroutine-shaped generator that another piece of code drives via .send(), the manual for-yield loop breaks the protocol. 'yield from' preserves it.

Where this comes up most often:

Tree traversals. Walking a tree where children are subtrees: yield from traverse(child) cleanly forwards everything from the recursive call.

Flattening nested iterables. yield from chain1, then yield from chain2, then yield from chain3.

Generator pipelines that route through helper generators based on conditions.

Delegating to itertools functions that already return iterables. yield from itertools.chain(a, b).

A tiny but real perf benefit too — 'yield from' is implemented in C in CPython, where the manual loop is interpreted bytecode. The difference is small for short iterables, noticeable for long ones, especially in deep recursion.

Use 'yield from' anywhere you'd otherwise write 'for x in inner: yield x'. It's clearer to read, faster to run, and correctly forwards the generator protocol. Free upgrade.

End of Day 11. Generators are one of those Python features that, once internalised, change how you write almost every data-processing piece of code.

What we covered.

Morning, the conceptual unlock — yield turns a function into a generator that pauses and resumes. Memory-bounded streams, infinite sequences, composable pipelines — all flow from this single mechanism.

Midday, generator expressions. The lazy cousin of list comprehensions. Swap [] for () and you go from O(n) memory to O(1). The right default for sum/max/any pipelines and for any case where you'll iterate exactly once.

Afternoon, a 6-line streaming pipeline that processes huge files line by line in constant memory. Three small generators chained together — read, tokenise, consume. The pattern composes to arbitrary stages. This is how grown-up Python handles big data without reaching for Spark.

Evening, 'yield from' as the clean way to delegate to another generator. Cleaner code, correctly forwards the generator protocol, and slightly faster. Use it anywhere you'd otherwise write a for-yield loop.

A running theme across week 2 — Python rewards explicit, composable, narrow-purpose constructs. Decorators wrap one concern. Dunder methods participate in one protocol. Generators yield one value at a time. The lego pieces are small; the assemblies are big.

Tomorrow, Day 12, context managers. The 'with' statement. Why your file handles never leak in Python and how to write your own context managers for any setup-and-teardown pattern. Plus the rare-but-clean ExitStack for composing several context managers at runtime.

See you in the morning.