RNNs & LSTMs — the predecessor to transformers

📅 Day 47. Recurrent Neural Networks (RNNs) ruled sequential data from 2010 to 2018, then transformers replaced them. They're worth understanding for context.

🔄 The core idea — process a sequence one element at a time, maintaining a hidden state that 'remembers' what's been seen so far.

At each step t:
h_t = f(W_xh · x_t + W_hh · h_{t-1} + b)

Where:
→ x_t is the input at this step.
→ h_{t-1} is the hidden state from the previous step.
→ h_t is the new hidden state.
→ f is a non-linearity (typically tanh).
→ W_xh and W_hh are weight matrices (shared across all time steps).

The hidden state acts as the model's memory. As you process the sequence, h_t accumulates information about everything seen so far.

🎯 Useful for sequential data — text, audio, time-series, DNA sequences.

⚠️ Big problem — vanishing/exploding gradients. When you backprop through many time steps, the gradient is multiplied by W_hh each step. If |W_hh| < 1, the gradient vanishes (model can't learn long-term dependencies). If |W_hh| > 1, it explodes (training diverges).

In practice, vanilla RNNs struggle to learn dependencies more than ~10 steps apart.

🛠 The fix — LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit). Both add explicit 'gates' that decide what to remember, forget, and write to the hidden state. The gating mechanism enables long-range learning that vanilla RNNs can't do.

LSTMs ruled NLP from 2014 to 2018. Then attention/transformers came.

💡 In 2026, you'll write RNNs rarely. They survive in resource-constrained settings (mobile speech recognition, embedded systems) and as a teaching example. For everything else — transformers.

🔄 The RNN's biggest limitation isn't the math — it's the architecture. RNNs are inherently sequential.

🐌 At step t, you can't compute h_t until you've computed h_{t-1}. There's no way to parallelise across the sequence. On a GPU with thousands of cores, an RNN uses one core for each sequence position; the others sit idle.

For a sequence of length 1000, an RNN takes 1000 sequential operations. On a transformer (next week's deep dive), the same sequence can be processed in ONE parallel operation.

📈 Concrete numbers. Same hardware, same model size:

→ RNN training on 1000-token sequences: limited by single-core speed. ~100 sequences/sec.

→ Transformer training on 1000-token sequences: limited by GPU parallelism. ~1000 sequences/sec (10x faster).

Multiply across millions of sequences for a full training run. Days vs weeks.

🎯 Beyond speed, transformers also win on quality:

→ Long-range dependencies. Attention can directly link any two positions in the sequence (O(1) path length). RNNs need to propagate information through many steps (O(n) path length), which decays.

→ Better gradient flow. Attention has direct paths; RNNs have sequential paths that compound vanishing/exploding.

→ More parameters per FLOP. Transformer compute scales with attention + feed-forward, both highly parallel. RNNs scale with sequence length.

🏆 Result — transformers replaced RNNs across the board. GPT, BERT, Claude, Llama, Gemini — all transformers. Speech-to-text models — transformers (Whisper). Image transformers (ViT) compete with CNNs.

📋 Where RNNs still survive in 2026:

→ Mobile/edge devices where transformer compute is too expensive.
→ Streaming/online tasks where you need to process input as it arrives (some real-time speech).
→ Specific niche tasks where the sequential bias is helpful.
→ Teaching, because they're conceptually simpler.

💡 Know the history. Use transformers.

💻 PyTorch's nn.LSTM does the gating internally. You don't implement gates by hand; you just declare input size, hidden size, number of layers.

Look at the snippet.

🏗 LSTMClassifier:
→ self.emb = nn.Embedding(vocab, d_emb) — turns integer token IDs into dense vectors.
→ self.lstm = nn.LSTM(d_emb, d_hid, batch_first=True) — processes the embedded sequence.
→ self.head = nn.Linear(d_hid, n_classes) — maps the final hidden state to class logits.

📊 batch_first=True changes input shape from (seq, batch, features) to (batch, seq, features), which matches every other layer in PyTorch. Always pass batch_first=True.

🔄 forward(x):
→ e = self.emb(x) — embed the input. Shape (batch, seq, d_emb).
→ out, (h, c) = self.lstm(e) — run through LSTM.
  → out is the per-step output (batch, seq, d_hid). Useful for tasks where you predict at each step (sequence labeling).
  → h is the final hidden state (1, batch, d_hid). Useful for sequence classification.
  → c is the final cell state (1, batch, d_hid). LSTMs have both.
→ self.head(h.squeeze(0)) — project the final hidden state to logits. Squeeze removes the leading dim of 1.

🎯 The output — (batch, n_classes) of logits. Pair with CrossEntropyLoss for training.

📋 Variants:
→ Stacked LSTM — pass num_layers=2 or more. Multiple LSTM layers on top of each other. More capacity, more compute.
→ Bidirectional LSTM — pass bidirectional=True. Processes the sequence both forward and backward; the output is the concatenation. Useful for tasks where future context matters (NER, POS tagging).
→ GRU — nn.GRU instead of nn.LSTM. Same API, slightly less compute, often equivalent quality. Use when LSTM is overkill.

💡 LSTMs are simple in PyTorch. The complexity is in WHEN to use them (rare in 2026) and HOW to handle variable-length sequences (next post).

💡 Pro tip for RNN code (rare in 2026, but useful when needed) — handle variable-length sequences with packed sequences.

🚨 The problem. Different sequences in a batch have different lengths. To batch them, you pad shorter ones to match the longest. The LSTM still iterates through all the padding, wasting compute.

📊 For a batch of [length=10, length=50, length=200] sequences, naive padding pads everything to 200. The LSTM iterates 200 times for the length=10 sequence, 190 of which are useless padding processing.

✅ The fix — torch.nn.utils.rnn.pack_padded_sequence + pad_packed_sequence.

from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

packed = pack_padded_sequence(input, lengths, batch_first=True, enforce_sorted=False)
out, (h, c) = lstm(packed)
out, lengths = pad_packed_sequence(out, batch_first=True)

Pack the input with the original lengths. The LSTM processes only the actual tokens, skipping the padding. Then unpack to get back to padded form.

📈 Speedup — proportional to the variation in lengths. If sequences vary 10x in length (10-200), you can save up to 10x compute. If they're roughly uniform, gain is smaller.

🎯 Other RNN tips for production:

→ Sort sequences by length before batching. Reduces wasted padding within batches.

→ Use bucketing — group sequences of similar length into the same batch. More uniform length per batch, less padding waste.

→ For Transformers, use attention masks instead. Same problem (padding), different solution. Attention masks tell the model to ignore padded positions.

💡 In 2026, you'll mostly write transformers, not RNNs. But if you maintain RNN code, packed sequences are the pro move. The training-loop boilerplate is small; the speedup is real on length-varying datasets.

📅 End of Day 47.

✅ Recap:

🔄 RNN = hidden state across time. h_t = f(Wx + Wh + b). Sequential by design.

🚧 Vanishing/exploding gradients limited vanilla RNNs to ~10 steps of memory. LSTMs and GRUs solved this with explicit gates.

🏆 Transformers won. Parallel attention, longer context, better gradient flow. RNNs survive on the edge and in niche cases.

💻 Tiny LSTM in PyTorch — 8 lines. nn.LSTM handles gates internally; you just declare sizes.

📦 Pack variable-length sequences for RNN production work. Or use transformers with attention masks.

🧠 Reflection — RNNs were the cutting edge in 2016. By 2018, transformers had displaced them. The history matters because the SAME architectural shifts continue to happen — Vision Transformers are displacing CNNs in some domains; Mamba and state-space models are challenging transformers in others. Stay flexible; the dominant architecture changes every 5-10 years.

🚀 Tomorrow, Day 48 — regularisation. Dropout, weight decay, early stopping. Stop your model from memorising. Then training tricks Day 49 wraps the week.

💼 Three days left in week 7. We're 53% through the sprint. Next week is NLP and transformers — the dominant architecture of the 2020s.

👋 See you in the morning.