Regularisation — stop memorising, start generalising

📅 Day 48. Overfitting is the central failure mode in deep learning. Spotting it is the first skill; preventing it is the second.

📊 The diagnostic chart. Train your model. Plot training loss and validation loss vs epoch.

Three possible patterns:

📉 Both losses drop together → underfitting. The model needs more capacity, more training, or both. Low priority concern.

📈 Train drops, validation drops less but follows → working well. The gap between train and val tells you how much overfitting is happening, but it's manageable.

📉📈 Train keeps dropping, validation drops then RISES → overfitting. The model is memorising training data instead of learning patterns. The divergence point is when you should have stopped.

This is the chart you check after every training run. Save it. Look at it.

🛠 Six tools to combat overfitting, in order of effectiveness:

1️⃣ More data. Always the best fix. Often impossible.

2️⃣ Smaller model. Reduce parameters until train and val track each other.

3️⃣ Dropout. Randomly zero out neurons during training. Forces the model not to depend on any single neuron.

4️⃣ Weight decay. L2 penalty on weights. Keeps weights from growing too large.

5️⃣ Data augmentation. Generate more training examples by transforming existing ones (image flips, text perturbations).

6️⃣ Early stopping. Stop training when validation loss stops improving. Use the model from the best epoch, not the last one.

💡 Almost every model needs at least one of these. Most need two or three. The exact combination depends on the model size, dataset size, and task.

🎯 Tomorrow's training-tricks post wraps these into a production training loop. Today we cover them individually.

🎲 Dropout is one of those rare deep-learning tricks that almost always helps and is dirt cheap to add. Born from a 2014 paper; ubiquitous since.

🔧 The mechanism. During training, randomly set a fraction p of activations to zero. Different random subset every forward pass. Effectively trains many subnets simultaneously.

At inference, no dropout. All neurons participate. Outputs scaled by (1-p) to compensate for the magnitude difference.

PyTorch handles the inference scaling automatically when you set model.eval(). Don't manually scale.

🧠 Why it works:

→ Forces redundancy. Since any neuron might be dropped, the model can't depend on any single neuron. It learns robust, distributed representations.

→ Implicit ensembling. Each forward pass uses a different subnet. The trained model behaves like an ensemble of many partial models. Ensembles generalise better than individual models.

→ Reduces co-adaptation. Without dropout, neurons can co-evolve to compensate for each other's quirks (you scratch my back). Dropout breaks this dependency.

📊 Typical p values:

→ 0.1-0.2 in CNNs (after activation in conv blocks).

→ 0.1-0.3 in transformers (attention dropout + residual dropout, both small).

→ 0.5 in older MLPs and pre-2015 architectures (less common now).

⚠️ Where to NOT use dropout:

→ Output layer. Adds noise where you want a clean prediction.

→ The very last hidden layer before output. Same reason.

→ Batch normalisation does some of dropout's job; combining can be over-regularising.

→ Very small models (<100k params) — they need all the capacity they have.

🎯 In PyTorch — nn.Dropout(p=0.2) added between layers. Activates only during model.train(); silent during model.eval(). The framework handles the mode-switch.

💡 Add dropout. Tune p between 0.1 and 0.3. Move on.

⏱ Early stopping is the cheapest, most-effective regularisation technique. Save compute AND prevent overfitting.

🎯 The idea — track validation loss every epoch. If it doesn't improve for N epochs in a row, stop training. Reload the model from the best epoch.

Most training frameworks (HuggingFace Trainer, PyTorch Lightning, fastai) have this built in as a callback. For plain PyTorch, here's the 8-line version.

Look at the snippet.

📊 Variables:
→ best_val = float('inf'). Best validation loss seen so far. Lower is better.
→ patience = 5. How many epochs to wait without improvement before stopping.
→ bad = 0. Counter for consecutive non-improving epochs.

🔄 Each epoch:
→ Train one epoch.
→ Evaluate on validation set.
→ If val improved (val < best_val) — update best_val, save the model checkpoint, reset bad to 0.
→ Otherwise — increment bad. If bad >= patience, stop training.

After training stops, load the best.pt checkpoint. That's your trained model.

📋 Why this works so well:

→ Prevents overfitting. The model from the best epoch is the one that generalises best, even if you trained for many more epochs.

→ Saves compute. No point continuing to train if validation is degrading. Stop early; iterate faster.

→ Eliminates the 'how many epochs' hyperparameter. Just set max_epochs to a generous number; let early stopping figure out the right point.

🎯 Tuning patience:

→ Small patience (3-5) — stops training quickly. Use when training is fast and you want quick iteration.

→ Large patience (10-20) — gives the model more chances to recover. Use when validation loss is noisy or you have time for long runs.

💡 Add early stopping to every training script. The 8 lines pay back enormously.

🎁 Bonus — also save model EMA (exponential moving average of weights). Often performs better than the snapshot at peak validation. Modern training tricks.

💡 Pro tip with subtle math — Adam + weight_decay= isn't actually L2 regularisation, despite looking like it. Use AdamW instead.

🤔 The history. Adam was published in 2014 with a weight_decay parameter. It worked... mostly. Researchers noticed that Adam-with-weight-decay didn't generalise as well as SGD-with-weight-decay. The community shrugged and said 'Adam is just like that'.

In 2017, a paper called 'Decoupled Weight Decay Regularization' (Loshchilov & Hutter) figured out why.

🧮 The math. L2 regularisation adds (lambda * w^2) to the loss. The gradient of this is (2 * lambda * w). So in plain SGD, weight decay subtracts (lr * 2 * lambda * w) from each weight per step. The decay scales with lr.

Adam, however, scales gradients by an adaptive learning rate (depending on past gradient magnitudes). When you add the L2 gradient term, IT ALSO gets scaled by the adaptive rate. The net effect — weight decay strength varies per parameter, depending on each parameter's gradient history. Often inconsistent and worse.

✅ The fix — AdamW. Decouples weight decay from the gradient. Subtracts (lr * lambda * w) from each weight directly, BEFORE the adaptive scaling. Net effect — weight decay is consistent and applies as you'd expect.

AdamW was retrofitted into PyTorch as torch.optim.AdamW. Drop-in replacement. Same hyperparameters; subtly better behaviour.

📊 Default weight decay values:
→ Transformers — 0.01 (this is the value used in BERT, GPT, etc).
→ CNNs — 5e-4 (smaller because CNNs have fewer parameters and less risk of weight explosion).
→ MLPs — typically 1e-4 to 1e-3.

🎯 The takeaway:

→ If you're using Adam with weight decay, switch to AdamW. Free upgrade.
→ If you're using SGD, weight_decay= works correctly. No change needed.

💡 The bug was real but small. The fix is one character — Adam → AdamW. Use it.

📅 End of Day 48.

✅ Recap:

📊 Overfitting in one chart. Train loss drops; val loss starts rising. The divergence point is where you should stop.

🎲 Dropout. Random zeroing during training. Forces redundancy and acts as implicit ensemble. p = 0.1-0.3 in modern architectures.

⏱ Early stopping in 8 lines. Track validation loss; stop after N epochs without improvement; reload best checkpoint. Saves compute and prevents overfitting.

🔧 AdamW for weight decay. Adam + weight_decay= is subtly buggy due to adaptive LR interaction. AdamW fixes it. Drop-in replacement.

🧠 Reflection — generalisation is the central problem of deep learning. Models with millions of parameters can memorise their training data perfectly; the goal is to constrain them so they learn patterns instead. Dropout, weight decay, augmentation, early stopping — each one a soft constraint that says 'don't memorise'.

🚀 Tomorrow, Day 49 — production training tricks that put it all together. LR schedules (warmup + cosine). Mixed precision (free 2x speedup). Gradient clipping. The 'use Lightning or HF Trainer instead of writing your own' wisdom. Then we wrap week 7.

💼 Two days to the end of week 7. Then NLP and transformers (week 8), the architecture under every modern AI system you've used.

👋 See you in the morning.