Attention — the mechanism that ate ML

📅 Day 52. Attention is the most important architectural innovation in deep learning since the convolution. Once you grok Q, K, V, every transformer becomes readable.

🎯 The intuition. For each token in a sequence, attention asks — 'which other tokens should I pay attention to, and how much?'

It has three vectors per token, each derived from the input via a linear projection:

→ Q (query) — what am I looking for? 

→ K (key) — what do I offer?

→ V (value) — what information do I carry?

🧮 The mechanics — for each query, dot it with every key. The result is a similarity score — how much should this token attend to that token? Apply softmax to turn scores into weights (sum to 1). Take a weighted sum of all the value vectors using these weights.

Formula — Attention(Q, K, V) = softmax(QKᵀ / √d) · V. The √d in the denominator scales the dot products to prevent the softmax from saturating.

📊 Result — each token gets a context-aware representation that mixes information from all relevant tokens. The representation depends on what's around the token. The same word in different contexts gets different representations. This is the secret behind transformers' contextual understanding.

⚡ The 'in parallel' part is what made transformers replace RNNs. RNNs process tokens one at a time. Attention processes ALL token pairs at once, in a single matrix multiplication. GPU-friendly; massively parallelisable.

🌐 Same operation in vision (Vision Transformers split images into patches; each patch is a 'token'), audio (Whisper), code (Codex), proteins (AlphaFold). The Q-K-V mechanism is universal.

💡 Three letters. The foundation of everything. Internalise once.

🧠 One attention head can only learn one type of pattern. Multi-head attention runs multiple attention computations in parallel, each free to specialise.

📊 The mechanics — split Q, K, V into h chunks (h = number of heads, typically 8-96). Compute attention independently in each chunk. Concatenate the outputs and project back to the original dimension.

If the model dim is 768 and you have 12 heads, each head operates on 64-dim Q, K, V. Lower dim per head; more heads in parallel.

🎯 Why this works:

→ Each head learns a different relationship pattern. Some heads focus on syntactic relationships (subject-verb agreement). Others on semantic similarity. Others on long-range dependencies (matching opening and closing parens). The diversity emerges from training; you don't program it.

→ More expressive than a single head with the same parameter count. The split-then-concatenate structure adds capacity.

→ Stays parallelisable. Each head is independent; computed in parallel.

📈 Modern model head counts:
→ BERT-base: 12 heads.
→ GPT-3 175B: 96 heads.
→ LLama-3-70B: 64 heads.
→ Most small models: 8-16 heads.

More heads doesn't always mean more performance. Past a point (typically 64), additional heads add parameters without improving quality. Training on more data is usually a better investment.

🛠 In PyTorch — nn.MultiheadAttention(embed_dim=768, num_heads=12). One line. Handles the split, attention computation, concatenation, and output projection.

💡 You almost never set num_heads by hand for new architectures. Use the defaults from a published architecture. The 'right' number was found empirically and tweaking rarely improves things.

💻 Strip away every transformer paper. Self-attention is matmul + softmax + matmul. The hardest part is keeping shapes straight.

Look at the snippet. We implement single-head self-attention from scratch. No multi-head, no masks, no projections — just the bare math to make the operation transparent.

📐 Inputs and shapes:
→ X is the input. Shape (B, T, D) — batch size, sequence length, hidden dim.
→ Wq, Wk, Wv are weight matrices, each (D, D). They project X into Q, K, V.

🧮 Compute Q, K, V:
→ Q = X @ Wq. Shape (B, T, D).
→ K = X @ Wk. Shape (B, T, D).
→ V = X @ Wv. Shape (B, T, D).

🎯 Compute attention scores:
→ Q @ K.transpose(-2, -1). Shape (B, T, T). For each query position, gives a row of similarity scores against every key position.
→ Divide by sqrt(d). Scaling factor that keeps softmax inputs in a reasonable range. Without it, large d would saturate the softmax.

📊 Softmax:
→ F.softmax(scores, dim=-1). Each row of scores becomes a probability distribution that sums to 1. These are the attention weights.

🔄 Aggregate:
→ weights @ V. For each query position, take a weighted sum of all value vectors. Shape (B, T, D).

The output is the same shape as the input — (B, T, D). Each position is a context-aware representation.

💡 Read this 14-line implementation once. The wrapped versions in PyTorch (nn.MultiheadAttention) and HuggingFace (transformers' attention layers) all do this exact computation, plus multi-head and masking. The core math is what's here.

🚀 You've now seen attention from first principles.

💡 Pro tip for PyTorch 2.0+ — use F.scaled_dot_product_attention. It dispatches to FlashAttention on supported GPUs. Faster, less memory, exact same math.

⚡ FlashAttention. A 2022 algorithm that computes attention in a memory-efficient way. Doesn't materialise the full attention matrix (which is O(T²) memory for sequence length T). Instead, it computes attention in tiles, using the GPU's fast on-chip SRAM.

Real numbers — for sequence length 4096:
→ Naive attention — minutes per training step. Often runs out of memory.
→ FlashAttention — seconds per step. Memory bounded by sequence length, not its square.

This isn't a quality difference; it's an efficiency difference. The output is mathematically identical.

🛠 The simplest way to get it — use PyTorch's built-in.

import torch.nn.functional as F

out = F.scaled_dot_product_attention(
    Q, K, V,
    attn_mask=mask,
    is_causal=True,  # for decoder-only models
)

The function checks the GPU and dispatches to FlashAttention if supported (Ampere or newer). Falls back to standard attention otherwise.

is_causal=True applies the causal mask automatically — each position can only attend to itself and earlier positions. Used in decoder LLMs (GPT-style).

📊 Speedup vs naive attention on long sequences (T=4096+):
→ 2-4x faster.
→ Memory drops from O(T²) to O(T).
→ Enables training on longer contexts than otherwise possible.

🚨 Common mistake. People still write the manual self_attention from yesterday's snippet in production code in 2026. That code is fine for understanding; not for production. Use the built-in.

💡 Don't roll your own attention math. Use F.scaled_dot_product_attention. Half a billion lines of tutorials are doing it the slow way.

📅 End of Day 52.

✅ Recap:

🎯 QKV in 3 letters. Query asks; Key answers; Value carries the info. softmax(QKᵀ/√d) · V is the formula.

🧠 Multi-head attention — parallel perspectives. h heads, each on D/h dim, concatenated. Each head learns different patterns. 8-96 heads typical.

💻 14-line self-attention from scratch. The clearest way to understand the math. Production code uses framework versions.

⚡ F.scaled_dot_product_attention. PyTorch's built-in dispatches to FlashAttention. 2-4x faster, exact same math. Use it.

🧠 Reflection — attention is the mechanism that scaled. Convolutions worked great for vision but had limited reach. Attention works for sequences of any length, in any modality. The 2017 'Attention Is All You Need' paper started a chain reaction that produced GPT, BERT, Claude, every modern AI system. Worth understanding deeply.

🚀 Tomorrow, Day 53 — the full transformer block. Attention + MLP + LayerNorm + residual. The Lego brick that stacks to billions of parameters. The encoder vs decoder distinction. The 'don't write your own transformer' wisdom.

💼 Week 8 is past halfway. Then RAG (week 9-10) builds on top of all this.

👋 See you in the morning.