NumPy — vectorise everything

Day 29. Week five begins, and we leave pure DSA for the data stack — the libraries that turn Python from 'a great teaching language' into the dominant language of machine learning.

Let's start with NumPy.

Loops in pure Python are slow. Even a simple sum over a million numbers takes hundreds of milliseconds. The same operation in C takes microseconds. The gap is roughly 100x.

NumPy closes the gap. Under the hood, an ndarray is a contiguous block of typed memory (32-bit floats, 64-bit ints, etc.) plus a set of C kernels that operate on that memory. When you write 'a + b' on two NumPy arrays, NumPy doesn't run a Python loop — it dispatches to a C function that adds them element-by-element using SIMD instructions where possible.

The result — you write one line of Python, you get C-speed execution.

This is why every modern ML framework — PyTorch, JAX, TensorFlow — is built on the same ndarray abstraction. PyTorch tensors are essentially NumPy arrays with GPU support and automatic differentiation. JAX arrays are NumPy arrays with JIT compilation and GPU/TPU support. The shape API is the same; the underlying numerical operations are the same; only the execution backend differs.

Which means — once you internalise NumPy, you've internalised 80% of how to read PyTorch, JAX, or TensorFlow code. Same patterns. Same vocabulary. Same idioms.

The rule for the next 60 days — vectorise or perish. If you find yourself writing a Python for-loop over a NumPy array, stop. There's almost always a vectorised alternative that's 100x faster.

When NumPy can't vectorise (rare, usually involves complex stateful logic), use numba or cython to JIT-compile the loop. Last resort, write the inner loop in C++ and bind via pybind11. Pure Python loops over numerical data are something to avoid.

Five NumPy operations that, between them, replace 90% of the loops you'd be tempted to write. Memorise them. Reach for them by reflex.

One — np.where(cond, a, b). Vectorised if/else. Returns a new array where each element is from a if cond is true at that position, else from b. Replaces:

result = []
for x in arr:
    if x > 0: result.append(x)
    else: result.append(0)

With:

result = np.where(arr > 0, arr, 0)

One line. Runs in C.

Two — np.clip(x, lo, hi). Bound values to a range. Anything below lo becomes lo; anything above hi becomes hi. Used in normalisation, gradient clipping, range enforcement.

Three — np.argsort(arr). Returns the indices that would sort the array. Useful for ranking, top-k selection, picking the k largest by some score (use np.argsort(-scores)[:k]).

Four — boolean indexing. arr[mask] selects only the elements where mask is true. Replaces filter loops:

result = arr[arr > threshold]

Five — reshape. arr.reshape(-1, n) flattens into n columns. arr.reshape(rows, -1) gives the flexibility to compute the other dim. Powerful for batching, reshaping inputs, undoing flattens.

The broader principle — anywhere your loop is doing 'apply a transformation to every element', NumPy probably has the vectorised version. Most for-loops over a numpy array can be eliminated with one of these primitives.

The productivity payoff is huge. Code that took 30 lines and ran in 5 seconds becomes 3 lines that run in 50 milliseconds. Both faster to write and faster to execute.

A concrete benchmark. We compute the L2 (Euclidean) distance between every pair of vectors in two batches — A is shape (n, d), B is shape (m, d). The result should be an (n, m) matrix where output[i][j] is the distance between A[i] and B[j].

Two implementations.

Pure Python — a triple nested loop. For each row of A, for each row of B, compute the distance. Each distance involves another inner loop over the d features. Total work — O(n * m * d) Python operations. For n=m=1000 and d=128, that's 128 million Python ops. Multiple seconds.

NumPy — one broadcast and one norm. A[:, None, :] has shape (n, 1, d). B[None, :, :] has shape (1, m, d). The subtraction broadcasts to shape (n, m, d). np.linalg.norm with axis=-1 collapses the last dim, giving (n, m).

One line. Same answer. ~100x faster.

Why the speedup? The NumPy version drops into a C kernel that processes all 128 million underlying float operations using SIMD instructions and tight memory access patterns. Pure Python's interpreter overhead — type checks, object creation, attribute access — is gone.

The broadcast pattern (A[:, None, :] - B[None, :, :]) is itself memory-efficient. NumPy doesn't materialise a full (n, m, d) tensor; it iterates with virtual strides. We'll cover broadcasting tomorrow in detail.

This is the canonical NumPy upgrade. The pattern is — when your computation involves many parallel operations on arrays, find the vectorised version. The savings are dramatic.

For large-scale similarity computations, this exact pattern (reshape + subtract + norm or reshape + matmul) is the foundation of vector search, RAG retrieval, recommendation systems. Now you've seen it in 4 lines.

There's a NumPy anti-pattern I see all the time, often in tutorials, occasionally in production code. It's the silent O(n²) bug.

result = np.array([])
for x in inputs:
    val = compute(x)
    result = np.append(result, val)

Looks reasonable. Is catastrophic.

np.append doesn't append in place. It allocates a new array, copies the old contents over, adds the new element, and returns the new array. Each call is O(n) where n is the current array's length. Total work for n appends — n * (1 + 2 + ... + n) / 2 ≈ O(n²).

For n = 100000, that's 10^10 operations. Minutes of waiting for what should be milliseconds.

Three fixes, in order of preference.

One — build a Python list with .append, then np.array(lst) at the end. List append is O(1) amortised. np.array() does one allocation and one copy. Total O(n).

result = []
for x in inputs:
    result.append(compute(x))
result = np.array(result)

Two — pre-allocate the array if you know the final size. result = np.empty(n); result[i] = ...

Three — use np.fromiter for generator-style construction. result = np.fromiter((compute(x) for x in inputs), dtype=float, count=n).

Four — vectorise the whole thing if compute() can run on the entire array at once. Often the right answer.

Never np.append in a loop. Build a list, materialise once. Or pre-allocate. Or vectorise. The 'innocent-looking' append is the perf bug that ships and embarrasses you in code review.

End of Day 29. Welcome to the data stack.

What we covered.

Morning, NumPy as 'C with a Python skin'. ndarray is contiguous typed memory plus C kernels. Vectorised operations run 100x faster than equivalent Python loops. Every modern ML framework (PyTorch, JAX, TensorFlow) is built on the same idea. Internalising NumPy unlocks them all.

Midday, the five NumPy operations that replace 90% of loops. np.where, np.clip, np.argsort, boolean indexing, reshape. Each one a C-implemented vectorised primitive. Reach for them by reflex.

Afternoon, the L2-distance benchmark — pure Python triple-nested-loop versus one-line NumPy broadcast. Same answer, 100x speedup. The broadcast pattern (A[:, None, :] - B[None, :, :]) is the foundation of vector search and most similarity computations.

Evening, the silent O(n²) bug — np.append in a loop. Replace with list-then-array, or pre-allocation, or fromiter, or full vectorisation. Never append-in-loop. The bug ships and embarrasses you.

A broader theme. The shift from pure Python to NumPy is a shift in mindset. Stop iterating; start operating on whole arrays. The vocabulary changes — from 'for each x' to 'for the whole batch'. Once internalised, you don't go back.

Tomorrow, Day 30, broadcasting in depth. The single mental model behind every PyTorch shape error you'll ever encounter. Pad with 1s, stretch on 1. Two rules; massive consequences.

See you in the morning.