Broadcasting — the deep-learning mental model

Day 30. Broadcasting is one of those topics where ten online tutorials each explain it differently. Most of them overcomplicate it. The actual rule is small.

When you do an operation between two arrays of different shapes, NumPy aligns them by:

One — pad the shorter shape with leading 1s until both have the same number of dimensions.

Two — for each dimension, the sizes must either match, or one of them must be 1. Where one is 1, NumPy 'stretches' it to match the other.

If no consistent alignment exists, NumPy raises a ValueError.

That's it. Two rules. Pad and stretch.

Examples:

(3,) + (3,) → (3,). Same shape; element-wise.

(3, 4) + (4,). Pad the (4,) to (1, 4). Stretch the leading 1 to 3. Both are now (3, 4). Element-wise.

(3, 1) + (1, 4). No padding needed. Stretch the trailing 1 in the first to 4. Stretch the leading 1 in the second to 3. Both become (3, 4). Outer-product-style result.

(2, 3, 4) + (4,). Pad (4,) to (1, 1, 4). Stretch to (2, 3, 4). Element-wise.

(2, 3, 4) + (3, 1). Pad (3, 1) to (1, 3, 1). Stretch leading 1 to 2; trailing 1 to 4. Result (2, 3, 4).

A case that fails — (3,) + (4,). Padding gives (3,) and (4,) — same dim count. Sizes 3 and 4 don't match and neither is 1. ValueError.

This rule explains every PyTorch shape error you'll encounter. The framework error message tells you the shapes; you walk through the rule mentally; you find which dim broke.

Memorise the rule. It comes up in every NN forward pass.

Broadcasting isn't just syntactic sugar. The implementation is genuinely efficient — it doesn't materialise the larger array.

When you do (3,) + (3, 4), naive thinking says NumPy first creates a (3, 4) copy of the (3,), then does element-wise addition. That would double the memory.

What NumPy actually does — it iterates with 'virtual strides'. The (3,) array's stride for the first dim is 0, meaning the iterator returns the same row for each i. The data is read from the small array, repeatedly, without copying. Memory cost — bounded by the larger array, not by the broadcast.

This matters enormously in deep learning. Every forward pass through a neural network broadcasts:

Bias addition. Weight matrix output is (batch, features). Bias is (features,). The bias broadcasts across the batch dimension. No (batch, features) copy of the bias is created.

Attention masks. Mask is (seq_len, seq_len). Attention scores are (batch, heads, seq_len, seq_len). The mask broadcasts across batch and heads. One small mask serves billions of attention scores.

Layer normalisation. The scale and shift parameters are (features,). They broadcast across batch and sequence dimensions in transformers.

Without broadcasting's memory efficiency, modern deep learning wouldn't fit on GPUs. Each operation would require allocating a giant fully-expanded version of every small array. Memory would be the bottleneck.

Reading neural-network code with broadcasting awareness — you can spot what's happening from the shape comments. # x: (B, T, D), # bias: (D,) — bias broadcasts across B and T. The architecture is comprehensible from the shapes alone.

Learn to read shapes; you read modern ML.

Cosine similarity between two batches of vectors is the foundation of vector search, RAG retrieval, recommendation systems, and a hundred other ML applications. Without broadcasting, it's a triple loop. With broadcasting and matrix multiplication, it's two lines.

The math — cosine_sim(a, b) = (a · b) / (||a|| * ||b||). The dot product divided by the product of magnitudes.

For batched cosine similarity — given A of shape (n, d) and B of shape (m, d), we want an (n, m) matrix where output[i][j] = cosine_sim(A[i], B[j]).

The trick — normalise each row of A and B to unit length first. After normalisation, cosine similarity equals dot product. The matrix of all pairwise dot products is exactly A @ B.T.

Look at the snippet.

Line 1 — normalise A. np.linalg.norm with axis=1 and keepdims=True gives a (n, 1) array of row norms. Dividing A by this broadcasts (each row scaled by its norm).

Line 2 — same for B.

Line 3 — A @ B.T. Matrix multiplication. Shape (n, d) times shape (d, m) gives (n, m). Each entry is the dot product of one row of A and one row of B. Because both are normalised, that dot product IS the cosine similarity.

O(n*m*d) operations, all running in BLAS-optimised C code. On modern hardware with vectorisation, this is hundreds of times faster than the equivalent loop in Python.

This exact pattern is what runs inside every vector database when you query. Faiss, Qdrant, pgvector — all do batched matmul of normalised embeddings. Now you've seen it in 3 lines. Tomorrow we move to pandas.

Shape mismatches account for the majority of NumPy and PyTorch debugging time. The cure is a habit — comment shapes everywhere, print shapes liberally during bugs.

My rule, applied religiously in any non-trivial array code:

# x: (B, T, D)
x = self.norm(x)
# logits: (B, T, V)
logits = self.head(x)

The comment annotates the expected shape after each operation. B is batch size; T is sequence length; D is hidden dim; V is vocab size. The single-letter convention is dense and readable once you know it.

When something breaks, walk down the function and add print(name, shape) at every step until you find where reality diverges from the comment.

for x in batch:
    print('input', x.shape)   # expected (B, T)
    e = self.emb(x)
    print('emb', e.shape)     # expected (B, T, D)
    h = self.attn(e)
    print('attn', h.shape)    # expected (B, T, D)
    ...

The diagnostic is mechanical. Wherever the actual shape disagrees with the expected shape, that's the bug.

Better than print — assert. assert x.shape == (B, T, D), f'unexpected {x.shape}'. Now the failure happens at the source, not three layers later.

For production code, einops is your friend. einops.rearrange and einops.einsum let you express shape transformations declaratively. einops.rearrange(x, 'b t d -> b (t d)') flattens the last two dims. The named axes turn shape comments into checkable code.

Shape bugs are mostly NOT logic bugs. They're transcription errors between the math you intended and the code you wrote. Liberal printing closes the gap fast.

End of Day 30. Two days into the data stack. Both heavy on principles, light on framework-specific minutiae.

What we covered.

Morning, the broadcasting rule. Two operations — pad the shorter shape with leading 1s, stretch any 1-dim along the matching axis. If no consistent alignment exists, ValueError. This single rule explains every shape error in NumPy and PyTorch.

Midday, broadcasting as a memory win. Virtual strides instead of materialising copies. Foundation of how neural networks fit on GPUs at all. Bias addition, attention masks, layer norm — all rely on broadcasting.

Afternoon, cosine similarity as a 3-line broadcast + matmul. The pattern that runs inside every vector database when you query. Now you've seen the full implementation in plain NumPy.

Evening, the shape-bug debugging discipline. Comment shapes inline. Print .shape liberally during bugs. Use assert for production. einops for declarative reshaping. Shape bugs are usually transcription errors, not logic errors.

A broader theme. NumPy's design choices — vectorisation, broadcasting, shape-aware operations — are the architectural choices that turned Python into the dominant ML language. Without them, models would be too slow and too memory-hungry to be practical.

Tomorrow, Day 31, pandas. The DataFrame as 'a dict of NumPy arrays plus an index'. The five methods that cover 80% of real data work. Plus the most common pandas perf bug — using iterrows when you should be vectorising.

See you in the morning.