Backpropagation — the chain rule, applied

📅 Day 44. Backpropagation is how neural networks learn. It scares beginners. It shouldn't — the idea is small once you've seen it.

🔄 The training step has three phases:

1️⃣ Forward pass. Run input through the network. Compute the output. Compute the loss (how wrong the output is vs the true label).

2️⃣ Backward pass. Compute the gradient of loss with respect to every parameter. This is what tells the optimiser how to adjust each weight to reduce loss.

3️⃣ Optimiser step. For each parameter, update it using the gradient. Typically — new_weight = old_weight - lr * gradient.

🎓 The 'backward pass' is backpropagation. It's just the chain rule of calculus, applied repeatedly through the network's layers. The smart bit is reusing intermediate values from the forward pass — without that reuse, computing all gradients would be O(n²) in the number of parameters.

With reuse, it's O(n). Same complexity as the forward pass.

🤖 PyTorch's autograd does all this automatically. You define the forward pass; you call .backward() on the loss; PyTorch computes every gradient and stores them. Then optimiser.step() applies them.

You never write the gradients by hand. Even for complex networks with custom layers, autograd handles the math. You just need to make sure your forward pass uses PyTorch operations (which know their gradients), not raw NumPy (which doesn't).

💡 The mystique around backprop is undeserved. The mechanics are clean. The math (chain rule) is from first-year calculus. The implementation (autograd) is hidden from you. Your job is to define the model and the loss; the framework handles everything else.

🚀 Forward, backward, step. Three phases. Repeat until loss is small.

⚙️ Optimisers turn gradients into weight updates. There are dozens. Three cover 95% of cases.

🐎 SGD with momentum. The classical optimiser. Plain stochastic gradient descent (subtract lr * gradient) plus momentum (a moving average of past gradients to accelerate in consistent directions). Used in many computer vision papers because it generalises slightly better than Adam at the end of long training runs.

Downsides — needs careful learning rate tuning. Slow to converge in the early epochs.

When to use — image models with cosine LR schedule. Final fine-tuning of large models when you've got the time.

🎯 Adam. Adaptive Moment Estimation. Maintains per-parameter learning rates based on running estimates of gradient mean and variance. Converges fast in the early epochs without much LR tuning. The default for most NLP and many other tasks.

Downsides — sometimes generalises slightly worse than SGD. Has a subtle bug with weight decay (next).

When to use — almost any new training run. Default LR is 3e-4 for Adam.

🔧 AdamW. Adam with proper weight decay decoupling. Original Adam had a subtle interaction between weight decay and the adaptive LR — basically, weight decay scaled by the gradient size, which isn't what you want. AdamW separates them. The original 'fixed Adam'.

When to use — any time you want weight decay with an adaptive optimiser. Default for transformer fine-tuning. Default LR 1e-4 to 5e-5 for transformers.

📋 The rule:
→ Starting a new project — Adam, lr=3e-4.
→ Image model with long training — SGD + momentum, lr=0.1, cosine schedule.
→ Transformer training or fine-tuning — AdamW, lr=1e-4 to 5e-5, linear warmup + cosine decay.
→ Anything weird — start with Adam, see what happens, adjust.

💡 Optimiser choice matters less than people think. LR schedule and batch size matter more. Pick a reasonable optimiser and tune the schedule.

💻 Every PyTorch training loop has the same shape. Memorise this template; the rest is variations.

Look at the snippet.

🛠 Setup — create the optimiser (AdamW with lr=3e-4 here) by passing model.parameters() (so the optimiser knows which weights to update). Create the loss function (CrossEntropyLoss for multi-class classification).

🔄 Outer loop over epochs. Each epoch is one full pass through the training data.

🔄 Inner loop over batches. train_loader yields (X, y) pairs. Each batch goes through the same five-step ritual.

📲 Step 1 — Move X and y to the device (GPU if available). PyTorch tensors must be on the same device as the model.

🧹 Step 2 — optim.zero_grad(). Clear gradients from the previous step. PyTorch accumulates gradients by default; without zero_grad, you'd be adding new gradients to old ones.

➡️ Step 3 — Forward pass. logits = model(X). Output of the model.

📐 Step 4 — Compute loss. loss_fn(logits, y) gives a scalar measuring how wrong the predictions are.

⬅️ Step 5 — loss.backward(). Backpropagation. Computes gradients of loss with respect to every parameter, stores them in .grad attributes.

👟 Step 6 — optim.step(). Apply the update rule (using gradients) to each parameter. The model is now slightly better.

At the end of each epoch, print the loss for monitoring. In real code, you'd also evaluate on validation data and possibly save a checkpoint.

📋 This six-step ritual is the heart of every PyTorch training loop. Once you can write it from memory, you can train any model.

🚀 Memorise the shape; specialise the model and loss for your problem.

💡 Pro tip — when your loss diverges to NaN within the first 10 training steps, the cause is almost always one specific thing. Save yourself a debugging session.

🚨 The symptoms:

→ Loss goes from a normal number (say 2.3) to NaN within a few steps.
→ Or loss explodes upward (10, 100, 1000, NaN).
→ Or all your gradients become NaN and training stops working.

🔍 The likely culprits, in order of probability:

1️⃣ Learning rate too high. The optimiser is taking steps so large that weights overflow. This accounts for ~70% of NaN cases I've seen. Drop lr by 10x and try again. Often fixes it.

2️⃣ Bad inputs. Check your training data for inf, NaN, or extremely large values (> 1e10). A single bad row can poison the gradient. Use torch.isfinite() to detect.

3️⃣ Numerical instability. Some operations (exp(), log()) blow up at extreme values. Add gradient clipping — torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) — to bound the gradient magnitude before the optimiser step.

4️⃣ Mixed precision overflow. If you're training with fp16/bf16 (automatic mixed precision), some intermediate values can overflow the smaller dtype. Try disabling AMP to see if it stabilises. If it does, use a GradScaler properly.

5️⃣ Initialisation bug. Custom layers initialised wrong might produce extreme outputs immediately. Check intermediate activation magnitudes.

6️⃣ Bug in custom loss function. log() of zero, division by zero, etc. Add small epsilons (log(x + 1e-8)) to prevent.

📋 My debugging order: lr first, inputs second, gradient clip third, then everything else.

💡 The good news — NaN losses are usually fixable by step 1 or 2. Spend 5 minutes there before going deeper.

📅 End of Day 44.

✅ Recap:

🧮 Backprop = chain rule + smart caching. Forward pass computes outputs and loss. Backward pass computes gradients. Optimiser step applies updates. Three phases per training step.

⚙️ Three optimisers cover most cases — SGD+momentum (vision), Adam (NLP default), AdamW (transformers). Default LR 3e-4 for Adam.

💻 The PyTorch training loop you'll write 100 times — zero_grad, forward, loss, backward, step. Six-step ritual; the rest is variations.

🚨 NaN loss? Lower learning rate first. ~70% of NaN losses are 'lr too high'. Then check inputs, add grad clip, then dig deeper.

🧠 The bigger picture — PyTorch hides the math (autograd) so you can focus on architecture and training dynamics. The framework is doing the heavy lifting; your job is to define the model and pick reasonable hyperparameters.

🚀 Tomorrow, Day 45 — PyTorch deeper. DataLoader for data pipelines. GPU device handling done right. Save and load models the right way. The eval+no_grad rule for inference.

💼 The PyTorch core is small — Model, Loss, Optimiser, DataLoader, plus the training loop. Master these five and you can build any neural network.

👋 See you in the morning.