Fine-tune or prompt? + week 8 wrap

📅 Day 56. Last day of week 8. Before we go into RAG next week, let's frame when to use what.

🎯 When you want an LLM to do something new, try in this order:

1️⃣ Better prompting. A clearer system prompt + few-shot examples solves 60% of cases. Cheap (no compute), fast (immediate iteration), reversible (just edit text). The first thing to try.

2️⃣ RAG (Retrieval-Augmented Generation). If it's a knowledge problem — the model lacks information about your domain or recent events — retrieve relevant docs and stuff them into the prompt. Doesn't change the model; provides it with knowledge.

3️⃣ Fine-tuning. If you need a different style, format, or specialised behaviour at scale — and prompting alone can't get you there — fine-tune. Modifies the model's weights to bake in patterns you want.

📋 Why this order:

→ Prompting is free. Zero training cost. Zero infrastructure cost. Just tokens.

→ RAG is moderate. You pay for embeddings + a vector DB + a few extra tokens per request. Knowledge updates instantly when you re-index docs.

→ Fine-tuning is expensive. GPU compute for training. New model artifact to maintain. Requires re-fine-tuning when the base model updates. Brittle — small data shifts can require retraining.

💸 Most teams jump to fine-tune too early. They have a problem prompting could solve in an afternoon; they spend two weeks fine-tuning. The result is no better, more expensive to maintain, and locked to a specific model version.

🎯 The disciplined approach — try cheap things first. Move down the cost curve only when needed.

💡 Most production LLM apps in 2026 use prompting + RAG. Fine-tuning is reserved for specific cases — distilling a large model into a small one, or training a model on private data that can't be retrieved at inference.

🚀 Start cheap. Stop early.

💰 Full fine-tuning of a 7B model needs roughly 80GB+ of GPU memory (for the weights, gradients, optimiser states, and activations). Most people don't have an A100 or H100 sitting around.

🎯 LoRA (Low-Rank Adaptation) solves this. Released in 2021; standard practice by 2023.

🧮 The idea — instead of updating all the model's weights, freeze them. Add small trainable matrices ('LoRA adapters') alongside each layer. The forward pass becomes — original_weight @ x + lora_a @ lora_b @ x. lora_a and lora_b are tiny (typically rank 8-64), much smaller than the full weight.

📊 Numbers — for Llama-3-8B:
→ Full fine-tune trains 8B parameters. ~80GB GPU.
→ LoRA fine-tune trains ~10M parameters (0.1% of the original). ~24GB GPU.

Quality is usually within 1-2% of full fine-tuning, often equivalent on most tasks.

🔧 QLoRA combines LoRA with 4-bit quantisation of the base model. The base model is loaded in 4-bit (saving 4x memory) and the LoRA adapters are trained in fp16. Now you can fine-tune Llama-3-8B on a single 16GB GPU.

📦 Libraries:
→ peft — HuggingFace's parameter-efficient fine-tuning library. Implements LoRA, prefix-tuning, IA3, and more.
→ trl — for SFT (supervised fine-tuning), DPO (direct preference optimisation), and GRPO. Uses peft underneath.
→ Axolotl — declarative YAML config for fine-tuning. Pick model, dataset, method; one command runs it.
→ unsloth — optimised LoRA training. 2x faster than standard implementations.

💼 The democratisation of fine-tuning. In 2023, fine-tuning a 7B model required cloud GPUs. By 2026, you can do it on a consumer M-series MacBook (slowly) or a single 16GB consumer GPU (fast).

💡 LoRA changed who can fine-tune. Now anyone with a modest GPU can adapt large models to specific tasks.

💻 QLoRA fine-tune of a 1B model. Twenty lines. Fits in a 16GB consumer GPU; on a Mac with unified memory, even less.

Look at the snippet.

📦 Imports — peft for LoRA, transformers for model loading, BitsAndBytesConfig for quantisation.

🔧 Quantisation config:
→ load_in_4bit=True. Load the base model in 4-bit precision.
→ bnb_4bit_compute_dtype=torch.bfloat16. Use bf16 for forward/backward computation (more numerically stable than fp16).

📦 Load base model:
→ AutoModelForCausalLM.from_pretrained(name, quantization_config=bnb).
→ The base loads in 4-bit. ~1GB memory for a 1B model (vs 4GB in fp32).

🎯 LoRA config:
→ r=16. The rank of the LoRA matrices. Higher = more parameters, more capacity, more overfitting risk. 8-32 is the typical range.
→ lora_alpha=32. Scaling factor. Common choice — alpha = 2 * r.
→ target_modules=['q_proj','v_proj']. Apply LoRA to the attention's Q and V projections. Typical for Llama-family. Some setups also add 'k_proj', 'o_proj', or all linear layers.
→ task_type='CAUSAL_LM'. We're fine-tuning for causal language modeling.

🚀 Wrap base model:
→ model = get_peft_model(base, cfg). Returns a model with LoRA adapters added. The base weights are frozen; only the LoRA adapters are trainable.

📊 model.print_trainable_parameters() — typically reports 'trainable: ~0.1% of total'. Huge savings.

💼 What's next (not in the snippet):
→ Prepare a dataset of (prompt, response) pairs.
→ Tokenise with the model's tokenizer.
→ Use trl's SFTTrainer (or HuggingFace Trainer with DataCollatorForLanguageModeling) to train.
→ At the end, save the LoRA adapter (a few MB). Load alongside the base for inference.

💡 The full training pipeline is ~50 lines with TRL. Axolotl reduces it to a YAML config + one command.

🚀 Production fine-tuning is now within reach for anyone with a $500 GPU.

💡 Pro tip — after fine-tuning, every output looks great. That's confirmation bias. ALWAYS evaluate the fine-tuned model against the base model on the same held-out test set.

🚨 The trap. You spent two weeks fine-tuning. You see the model outputs and feel pride. You ship it. Three weeks later, someone asks 'is this actually better than the base model?' You don't know.

📊 The discipline:

1️⃣ Hold out a test set BEFORE fine-tuning. 100-500 examples that the model never sees during training. Cover the use cases you care about — easy ones, hard ones, edge cases.

2️⃣ Evaluate the BASE model on the test set. Get its score. This is your floor. Document it.

3️⃣ Fine-tune.

4️⃣ Evaluate the FINE-TUNED model on the SAME test set. Get its score.

5️⃣ Compare. The fine-tune should beat the base by a meaningful margin (e.g., 5+ points on accuracy, or substantially better on qualitative review).

🎯 If it doesn't, the fine-tune isn't worth the cost. Roll it back. Use the base model. Save the maintenance burden.

📋 How to evaluate qualitative outputs (e.g., generated text quality):

→ Use a stronger LLM as judge. GPT-4o or Claude 3.5 Sonnet rates outputs on a 1-5 scale or compares pairs. Cheap, fast, reproducible.

→ Use a panel of human evaluators on a sample. Slower, more accurate, captures things LLMs miss.

→ Use task-specific metrics where they apply (BLEU for translation, accuracy for classification).

💡 The key principle — the comparison must be apples-to-apples. Same test set. Same scoring method. Different models.

🚨 Common mistake — only looking at fine-tune outputs, never comparing. Always go back to the base. It might already be good enough; you'd save weeks of effort.

🚀 Fine-tunes that don't beat baseline are technical debt. Don't ship them.

📅 End of Day 56. End of week 8. We're 62% through the sprint.

✅ Week 8 in seven topics:

📝 Tokenisation. BPE/subword units. Tokens ≠ words. Always count before API calls.

🧭 Embeddings. Coordinates of meaning. Pick by cost/dim/domain. Always normalise.

🎯 Attention. Q, K, V, softmax(QKᵀ/√d) · V. Multi-head for parallel perspectives. F.scaled_dot_product_attention for production.

🏗 Transformer block. LayerNorm + Attention + Residual + LayerNorm + MLP + Residual. Stack to depth. Don't write your own; use HF transformers.

👁 BERT and encoders. Bidirectional + MLM pretraining. Use for embeddings, small-data classification, NER. Distilled variants for production.

🎯 GPT and decoders. Autoregressive next-token prediction. Sampling dials (T, top-k, top-p). vLLM for production inference.

🛠 Fine-tune vs prompt vs RAG. Try cheap first. LoRA/QLoRA make fine-tuning accessible. Always compare fine-tune to base.

🧠 Big takeaway from week 8 — modern AI rests on the transformer architecture. Same building block (attention + MLP), different masks (encoder vs decoder), different scales (millions to trillions of parameters). Understanding the block makes every paper and every framework readable.

🚀 Next week — RAG. The architecture that powers most LLM apps you'll build in 2026. We've covered all the components — embeddings, encoders, decoders, sampling. Now we put them together.

💼 We're 62% through. Five weeks to go. RAG (week 9-10), agents (11), automation (12), career (13).

👋 See you in week 9.