GPT-style decoders — autoregressive generation

📅 Day 55. GPT-style models — the architecture behind every modern LLM you've used.

🎯 The objective is shockingly simple — given tokens t_1, t_2, ..., t_n, predict t_{n+1}. Take the predicted token, append it, predict t_{n+2}. Repeat until you have enough output or hit a stop token.

This is autoregressive generation. The model generates one token at a time, each one informed by everything before it.

📚 Training. Cross-entropy loss on the next token across a huge corpus (trillions of tokens for modern LLMs). The model learns to model P(token | context). Train for months on thousands of GPUs; you have a base LLM.

💬 Then comes instruction tuning. Take the base LLM and fine-tune on (instruction, response) pairs. The model learns to follow instructions. This is the difference between GPT-3 (good at completing text) and ChatGPT (good at answering questions).

📈 Then RLHF (Reinforcement Learning from Human Feedback). Humans rank model outputs; the model learns to produce outputs humans prefer. This is what makes Claude, ChatGPT, Llama-Instruct so useful.

🌐 Inference. Most modern chat is — start with a system prompt + user message; predict the next token; append; repeat until end-of-turn token or max length. The 'thinking' you see is just one token at a time, sampled from a distribution.

🚀 The simplicity is striking. ONE training objective (predict next token), scaled massively (trillions of tokens, hundreds of billions of parameters), gives you GPT-4, Claude 3.5, Llama-3-405B. The intelligence emerges from scale plus good training data.

💡 Understand this loop and modern AI is no longer mysterious. It's a transformer predicting tokens, one at a time, very fast.

🎲 The model outputs a probability distribution over the entire vocabulary at each step. The sampling strategy decides which token to pick.

🎯 Four common strategies:

📐 Greedy. argmax of the distribution. Always pick the most-likely token. Deterministic; given the same input, you get the same output. Often produces repetitive, robotic-sounding text. Use for tasks where determinism matters (factual QA, code generation).

🌡 Temperature. Divide logits by T before softmax. T < 1 makes the distribution sharper (more confident, closer to greedy). T > 1 makes it flatter (more random). T = 0 is greedy. T = 1 is the model's natural distribution. For creative writing, T = 0.7-1.0. For factual QA, T = 0 or 0.1.

🔢 Top-k. Keep only the top k tokens by probability; renormalise; sample from those. Limits the candidate set. k = 50 is common. Less likely to pick weird low-probability tokens.

📊 Top-p (nucleus). Keep the smallest set of tokens whose cumulative probability >= p; renormalise; sample from those. Adaptive — sometimes 5 tokens, sometimes 50, depending on how peaked the distribution is. p = 0.9-0.95 is the modern default.

🎯 Combinations:

→ For factual answers — T = 0 or T = 0.1. Greedy or near-greedy.

→ For creative writing — T = 0.7-1.0 with top_p = 0.95. Diverse but not random.

→ For code generation — T = 0 to 0.2. Want correct, not creative.

→ For RAG/Q&A from docs — T = 0. You want the model to ground in the docs, not invent.

→ For brainstorming / idea generation — T = 0.9-1.2 with top_p = 0.95. More diversity.

💡 The 'temperature' parameter people see in OpenAI's playground is exactly this T. Drop it for grounded tasks. Raise it for creative ones.

💻 Twenty lines to load a small open LLM and generate text locally. transformers + a 1B-3B parameter model is the perfect size for laptops with 16GB RAM.

Look at the snippet.

📦 Imports — AutoTokenizer, AutoModelForCausalLM, torch.

🔧 Load model:
→ name = 'meta-llama/Llama-3.2-1B-Instruct'. The Instruct variant is fine-tuned for chat. The 1B size runs on a laptop.
→ tok = AutoTokenizer.from_pretrained(name).
→ model = AutoModelForCausalLM.from_pretrained(name, torch_dtype='auto'). torch_dtype='auto' picks the best precision (bf16 on supported hardware, fp32 fallback).

📝 Prompt — 'Explain RAG in two sentences.'.

🔢 Encode — tok(prompt, return_tensors='pt').input_ids.

🎯 Generate:
→ model.generate(ids, max_new_tokens=80, temperature=0.7, top_p=0.95).
→ max_new_tokens=80 limits the output length.
→ temperature=0.7 + top_p=0.95 — moderate creativity.

📝 Decode — tok.decode(out[0], skip_special_tokens=True). Get back text. The model's response will be appended to your prompt; you typically slice off the input portion.

🚀 The whole loop runs in seconds for 80 tokens on a modern laptop. Great for prototyping, evaluation harnesses, and learning.

📊 Model size guide for laptops:
→ 1B params — fits in ~4GB memory. Runs comfortably on most laptops. Quality good for simple tasks.
→ 3B params — ~6GB memory. Better quality. Borderline on 8GB laptops.
→ 7-8B params — ~16GB. Needs a decent GPU or 32GB+ system RAM. Quality competitive with hosted models for many tasks.
→ 70B+ — needs server-class hardware. Use cloud or APIs.

💡 For local production deployment (more than just generating one prompt at a time), use vLLM (next post). The naive .generate() loop is fine for tutorials, awful for serving.

💡 Pro tip — for serious local LLM inference, use vLLM (or llama.cpp). transformers' .generate() is fine for tutorials but terrible for production.

⚡ vLLM. State-of-the-art LLM inference engine. Open source, pip install vllm. Serves models with an OpenAI-compatible API.

📊 What it does:

→ Paged attention. Memory-efficient KV cache that allows much higher batch sizes. The KV cache is divided into pages and managed like virtual memory; less wasted memory.

→ Continuous batching. Doesn't wait for a fixed batch to fill before processing. New requests join the batch in flight. Throughput stays high regardless of request timing.

→ FlashAttention integration. Fastest attention implementations.

→ Quantisation support — INT8, INT4, AWQ, GPTQ. Run larger models with less memory.

Real numbers — for Llama-3-8B on an A100 GPU:
→ Naive .generate() — ~50 tokens/sec, single request only.
→ vLLM — ~1500 tokens/sec across 32 concurrent requests. 30x throughput.

🛠 Quick setup:

from vllm import LLM

llm = LLM(model='meta-llama/Llama-3.2-3B-Instruct')
outputs = llm.generate('Explain RAG in two sentences.')
print(outputs[0].outputs[0].text)

Or serve as an OpenAI-compatible API:
vllm serve meta-llama/Llama-3.2-3B-Instruct

Then point your OpenAI client at http://localhost:8000.

🦀 llama.cpp. Alternative for CPU and consumer GPUs. C++ implementation; supports GGUF quantised models. Best for laptops, edge devices, no-GPU setups.

💻 Use llama.cpp on a MacBook M-series, Windows laptop without a discrete GPU, or anywhere you need to run LLMs without cloud infrastructure. Surprisingly fast on Apple Silicon thanks to unified memory.

📋 The decision tree:
→ Have a NVIDIA GPU, want max throughput → vLLM.
→ Want CPU/laptop inference → llama.cpp.
→ Just prototyping → transformers .generate().

💡 Don't ship .generate() to production. The paths above are easier and orders of magnitude faster.

📅 End of Day 55.

✅ Recap:

🎯 Decoder LLMs predict the next token autoregressively. Train on trillions of tokens. Add instruction tuning + RLHF. You have ChatGPT/Claude.

🎲 Four sampling dials — greedy, temperature, top-k, top-p. T=0 for factual; T=0.7 + top_p=0.95 for creative.

💻 20-line local LLM with transformers + Llama-3.2-1B-Instruct. Runs on a laptop.

⚡ Use vLLM for serious local inference. 10-30x throughput vs naive .generate(). OpenAI-compatible API for easy integration.

🧠 Reflection — the gap between 'how it works' and 'how to use it well' is huge for LLMs. The architecture is simple. The engineering of training and serving is enormous. Most ML engineers spend their time on the engineering side; the architecture is library code.

🚀 Tomorrow, Day 56 — fine-tune vs prompt vs RAG. The decision tree for adapting LLMs to your problem. LoRA and QLoRA for cheap fine-tuning. Always-evaluate-fine-tunes-vs-base. Then we close week 8.

💼 One more day in week 8. Then RAG (week 9-10).

👋 See you in the morning.