Production training tricks + week 7 wrap

📅 Day 49. Last day of week seven.

📈 Constant learning rate is fine for tutorials. For real training, learning rate schedules give you better final loss and faster convergence.

🎯 The two-phase modern recipe — linear warmup followed by cosine decay.

🔥 Phase 1 — linear warmup. First 5-10% of training steps. Start lr near zero; ramp linearly up to your target lr. Why — Adam's adaptive learning rates are unreliable in the first few steps (variance estimates haven't stabilised). Starting with a tiny lr prevents the optimiser from blowing up immediately.

🌊 Phase 2 — cosine decay. Rest of training. Gradually drop lr from target down to near zero following a cosine curve. Why — the optimiser benefits from large steps early (find the rough region of optimum) and small steps late (refine the optimum). Cosine gives a smooth transition.

Math — lr(t) = lr_max * 0.5 * (1 + cos(pi * t / T)) where t is current step, T is total steps. Simple to implement; effective in practice.

📊 Alternative — ReduceLROnPlateau. When validation loss stops improving, drop lr by a factor (typically 0.5 or 0.1). Simpler than cosine; works when you don't know total steps in advance.

Used when — you're not sure how long training will take, you want robust 'just works' behaviour without scheduler tuning.

🎯 The choice in 2026:

→ Transformer training/fine-tuning — warmup + cosine. The de facto standard.
→ CNN training — warmup + cosine, OR step decay (drop lr at fixed milestones).
→ Quick experimentation — constant lr is fine. Schedules matter for final-quality runs.

💡 The single biggest improvement most teams don't make — adopt warmup + cosine. Often beats months of architecture tuning. PyTorch has CosineAnnealingLR + LinearLR built in; combine with SequentialLR.

🚀 LR schedule is a free upgrade. Use it.

⚡ Free 2x speedup on modern GPUs (NVIDIA Volta, Ampere, Hopper) — train in fp16 or bfloat16 instead of fp32.

🧮 The math. fp32 (32-bit floats) is the default. fp16 / bfloat16 are 16-bit alternatives. Smaller representation, faster operations.

Why it speeds things up:

→ Tensor cores on modern GPUs do mixed-precision matrix multiplication 8-16x faster than full fp32. Same hardware; way more throughput.

→ Memory bandwidth is half. Loading half the bytes per parameter; faster everywhere.

→ Memory footprint drops 50%. You can train larger batches or larger models on the same GPU.

🛡 Numerical stability is preserved by keeping a master copy of weights in fp32. Forward and backward passes happen in fp16; the gradient updates are applied to the fp32 master copy. The fp32 master is then copied to fp16 for the next forward pass.

Gradient scaling — fp16 has limited range. Small gradients (common late in training) can underflow to zero. The trick — multiply the loss by a scale factor before backprop; gradients are scaled up; divide back before optimiser step. PyTorch's GradScaler handles this automatically.

📦 Code:

from torch.amp import autocast, GradScaler
scaler = GradScaler()

with autocast(device_type='cuda'):
    out = model(x)
    loss = loss_fn(out, y)

scaler.scale(loss).backward()
scaler.step(optim)
scaler.update()

Three extra lines. 2x speedup. Half the memory. Same final accuracy on most tasks.

🚨 When NOT to use AMP:

→ Tasks with extreme dynamic range (some scientific computing). Test first.

→ Custom layers that don't have fp16 implementations. Modern PyTorch is fine for almost everything.

→ Old hardware (pre-Volta). The speedup comes from tensor cores; without them, no benefit.

💡 If you're not using mixed precision in 2026, you're leaving 2x on the table. Try it.

💻 Putting it all together — the production training loop with every trick from this week.

Look at the snippet.

🛠 Setup:
→ AdamW optimiser (proper weight decay).
→ CosineAnnealingLR scheduler (lr decays cosinely over epochs).
→ GradScaler for mixed precision.

🔄 Per epoch, per batch:
→ Move X, y to device.
→ Zero gradients.
→ With autocast — forward pass and loss in fp16/bf16.
→ scaler.scale(loss).backward() — gradients computed in scaled fp16, then reverse-scaled.
→ scaler.unscale_(opt) — undo the scale before clipping.
→ Gradient clipping — clip total gradient norm to 1.0. Prevents catastrophic updates from outlier gradients.
→ scaler.step(opt) — optimiser step (skipped if gradients are NaN).
→ scaler.update() — update the scale factor for next iteration.

📊 At end of each epoch — sched.step() updates the learning rate.

This is the modern training loop. AMP (2x speedup), AdamW (correct weight decay), cosine decay (better convergence), gradient clipping (stability), early stopping (would be added on top).

🚀 Real numbers — same model, same data, same final accuracy:
→ Naive PyTorch loop — 60 minutes per epoch.
→ With this production loop — ~25 minutes per epoch.
→ Plus less hyperparameter sensitivity (LR scheduler reduces dependence on initial lr).

📋 Things this loop is missing for completeness:
→ Validation evaluation per epoch.
→ Logging to TensorBoard / Weights & Biases.
→ Model checkpointing.
→ Multi-GPU support (DDP).
→ Early stopping callback.

Rather than add them by hand, use a framework. That's tonight's tip.

💡 Memorise this loop shape. It's the foundation of every serious training run you'll write.

💡 Pro tip — past your first 10 training loops, stop writing them by hand. Use a framework.

📦 Three frameworks worth knowing:

🤗 HuggingFace Trainer. Best for transformer fine-tuning. Handles AMP, gradient accumulation, distributed training, checkpointing, evaluation, logging — all with sensible defaults. Most NLP work in 2026 uses Trainer or its extensions (TRL, PEFT).

⚡ PyTorch Lightning. Clean abstraction over plain PyTorch. You define a LightningModule (model + train/val/test steps); Lightning handles the loop, callbacks, checkpointing, multi-GPU, mixed precision. Used by many vision and audio teams.

🚀 HuggingFace Accelerate. Lower-level than Trainer. Lets you keep your training loop's structure while abstracting the device/distributed-training plumbing. Used when you need custom training logic but want easy multi-GPU.

🎯 Why use a framework:

→ Battle-tested. Edge cases (mixed precision overflow, gradient sync across GPUs, checkpoint resume) are handled correctly. Your hand-rolled version probably has bugs you haven't found.

→ Multi-GPU for free. PyTorch Lightning's 'accelerator' or HF Trainer's 'fsdp' enables multi-GPU with one config flag. Implementing DDP correctly by hand is non-trivial.

→ Logging baked in. TensorBoard, Weights & Biases, MLflow — all integrated. One log call sends to all configured backends.

→ Saves time. The boilerplate (loop, checkpointing, early stopping, eval) is hundreds of lines. Frameworks reduce it to a couple dozen.

🛠 When to write your own loop:

→ First few times you train, to understand the mechanics.

→ Highly custom training (RL, GANs, complex multi-task setups).

→ Embedded/edge deployments where framework overhead matters.

💡 For 95% of production training, frameworks win. Don't reinvent the loop.

🚀 The energy you save goes to data, architecture, and evaluation — the parts that actually matter.

📅 End of week 7. 49 days. 245 posts. We're 54% through the sprint.

✅ Week 7 in seven topics:

🧠 Neurons + activations. f(Wx + b). ReLU/GELU for hidden, sigmoid/softmax for output, tanh for RNNs. PyTorch's defaults are good.

🔄 Backprop + optimisers. Forward, backward, step. Three optimisers (SGD+momentum, Adam, AdamW) cover most cases. NaN loss = lower lr first.

📦 PyTorch hygiene. Dataset + DataLoader. Device handling. Save state_dict. eval + no_grad at inference.

🖼 CNNs. Convolutions = sliding dot product + parameter sharing. ResNet's skip connections (x + f(x)) unlocked deep networks. Always start from a pretrained model.

🔄 RNNs. Hidden state across time. Gradient issues solved by LSTMs/GRUs. Transformers won; RNNs survive on the edge.

🛡 Regularisation. Dropout, weight decay, early stopping, AdamW. Each one a soft constraint that prevents memorisation.

⚡ Production training tricks. LR warmup + cosine decay. Mixed precision (AMP) for 2x speedup. Gradient clipping for stability. Use a framework (HF Trainer, Lightning) for non-toy training.

🧠 The big takeaway from week 7 — deep learning is mostly engineering, not math. The math behind backprop is calculus from the 1960s. The hard parts are the engineering — efficient training, reproducibility, scaling, regularisation. The frameworks abstract most of this; your job is to use them well.

🚀 Next week — NLP and transformers. The architecture under every LLM you've used. From tokenisation to attention to the transformer block to BERT vs GPT. Then RAG (week 9-10) builds on top.

💼 We're past halfway. The remaining six weeks are the AI-modern stack — NLP, RAG, agents, automation, career. Strap in.

👋 See you in week 8.