BERT — encoders that still matter

📅 Day 54. BERT was the first transformer to dominate NLP benchmarks. Released in 2018; still in active production use seven years later.

🎯 Two ideas made BERT work:

1️⃣ Bidirectional attention. Unlike GPT (which only sees past tokens), BERT sees the entire input both ways. Each token's representation is informed by tokens before AND after it. Useful for understanding tasks where the whole context is available.

2️⃣ Masked language modeling (MLM) pretraining. Take a sentence, randomly mask 15% of tokens, train the model to predict the masked tokens. The model learns rich contextual representations from massive unlabeled text.

📚 BERT was pretrained on Wikipedia + BooksCorpus (~16GB of text). After pretraining, fine-tune for any downstream task — classification, NER, QA, similarity.

Fine-tuning costs are low — a few hours on a single GPU for most tasks. The 'transfer learning' from pretraining provides 90% of the value; fine-tuning specialises the last 10%.

🌐 In 2026, BERT-class models dominate:

→ Sentence embeddings. sentence-transformers, BAAI/bge, Jina embeddings — all BERT-family architectures.

→ Retrieval for RAG. The 'R' in RAG uses encoder embeddings. BERT and descendants.

→ Small-data classification. Fine-tune BERT on 1000 labeled examples; often beats a hand-crafted feature pipeline.

→ Named entity recognition (NER), token classification, sentence-pair tasks.

💡 GPT-style models excel at generation; BERT-style at understanding. Both are transformers; the architectural difference is the attention mask. The right choice depends on whether you're producing text or understanding it.

🚀 BERT's descendants — RoBERTa (better training), DistilBERT (smaller, faster), DeBERTa (improved attention), MPNet — push the original further. All share the bidirectional encoder design.

🤔 LLMs (GPT, Claude, Llama) can do most NLP tasks now. So when is BERT-family still the right choice in 2026?

✅ Use encoder models when you need:

📐 Sentence/document embeddings. For semantic search, RAG, similarity computations. Sentence-transformers and BAAI/bge are BERT-family. Faster, smaller, more cost-effective than running a large LLM for each embedding.

📊 Small-data classification. If you have 1000 labeled examples for a classification task, fine-tuning BERT typically beats few-shot prompting an LLM. The fine-tuned BERT is also faster at inference and easier to deploy.

🏷 NER and information extraction. Token-level tasks like 'find all person names' or 'extract dates'. Encoder models with token classification heads are the standard.

🔍 Retrieval inside RAG. Both as embedders (encode docs into vectors) and as cross-encoder rerankers (score query-doc pairs).

❌ DON'T use encoders when you need:

📝 Open-ended generation. BERT can't generate text. Use a decoder LLM.

💬 Multi-turn chat. Encoders aren't designed for stateful conversation.

🧠 Long-form reasoning, math, code. Decoders win at multi-step reasoning.

🎯 Few-shot or zero-shot tasks where you can't fine-tune.

💰 Cost comparison:

→ Embedding 1M docs with BAAI/bge-small (encoder) — minutes on a GPU, free.

→ Embedding 1M docs with OpenAI text-embedding-3-small (encoder via API) — $20.

→ Generating answers for 1M queries with a fine-tuned BERT classifier — minutes, runs anywhere.

→ Generating answers for 1M queries with GPT-4 — $1000s, slower, requires API.

For classification + retrieval, encoder + classical model is dramatically cheaper and faster. Use the right tool.

💡 Encoders are not legacy. They're the right choice for retrieval and small-data classification.

💻 HuggingFace Trainer makes BERT fine-tuning trivial. Twenty lines from import to trained model.

Look at the snippet.

📦 Imports — AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments. The 'Auto' classes pick the right architecture based on the model name.

🔧 Load:
→ tok = AutoTokenizer.from_pretrained('bert-base-uncased'). Standard BERT tokenizer.
→ model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2). Loads BERT with a fresh classification head (2 output classes for binary classification).

🔄 Tokenize:
→ Define a tokenize function that takes a batch and returns token IDs.
→ ds = ds.map(tokenize, batched=True). Apply to the dataset (assumed to be a HuggingFace Datasets object).

⚙️ Training arguments:
→ output_dir='out' — where to save checkpoints.
→ per_device_train_batch_size=16 — fits on most GPUs for BERT-base.
→ num_train_epochs=2 — usually enough for BERT fine-tuning.

🚀 Train:
→ Trainer(model=model, args=args, train_dataset=ds['train'], eval_dataset=ds['val']).train().

One line of training. Trainer handles AMP, gradient accumulation, evaluation, checkpointing, logging — all by default.

📊 Inference after training:
→ outputs = model(**tok('your text', return_tensors='pt'))
→ predictions = outputs.logits.argmax(dim=-1)

Or use the pipeline API for simpler inference:
→ from transformers import pipeline
→ classifier = pipeline('text-classification', model='./out')
→ classifier('your text')

💡 Twenty lines covers the entire fine-tuning workflow for binary classification. For multi-class, change num_labels. For multi-label, use AutoModelForSequenceClassification with problem_type='multi_label_classification'. For NER, use AutoModelForTokenClassification.

🚀 Trainer is the modern standard for most fine-tuning. Learn it once; use it forever.

💡 Pro tip — full BERT-large is overkill for most production tasks. Use distilled or small variants.

📦 The ecosystem of small encoder models in 2026:

🔹 DistilBERT (66M params). 40% smaller than BERT-base, 60% faster, retains 97% of accuracy. Ideal for production where latency matters.

🔹 MiniLM / all-MiniLM-L6-v2 (22M params). Tiny but excellent for sentence embeddings. Industry standard for free, self-hosted embeddings. Runs on CPU at decent speed.

🔹 BAAI/bge-small (33M params, 384 dim). Modern lightweight embedding model. Strong on MTEB. Fast.

🔹 BAAI/bge-base (110M, 768 dim). Mid-size. Best balance for most retrieval.

🔹 BAAI/bge-large (335M, 1024 dim). Highest quality from this family. Slower; use when quality is critical.

🔹 RoBERTa-distill, MobileBERT, ALBERT — other distillation lineages.

📊 Why this matters in production:

→ Latency. Smaller models = faster inference. For real-time applications (search, chatbots), the 2-3x speedup of distilled models is huge.

→ Cost. Smaller models fit on CPUs. No GPU needed. CPU instances are 5-10x cheaper than GPU instances.

→ Memory. Distilled models can fit in browser-side WebAssembly. Edge inference becomes possible.

→ Throughput. Same hardware can serve more requests with smaller models. Better cost-per-prediction.

🎯 The trade-off — typically 1-3% accuracy drop. For most production tasks, that's a fine trade for the speed and cost benefits.

💡 The rule — start with the smallest model that meets your accuracy requirements. Don't default to the largest. The marginal accuracy gain rarely justifies the cost overhead.

🚀 For embeddings specifically — bge-small is hard to beat for the cost. Try it first.

📅 End of Day 54.

✅ Recap:

👁 BERT = bidirectional encoder + masked language modeling pretraining. 2018 release; still in production use in 2026.

🎯 Use encoders for embeddings, small-data classification, NER, RAG retrieval. Use decoders for generation.

💻 20-line fine-tune with HuggingFace Trainer. Tokenize, set arguments, train. Done.

📦 For production, use distilled / small variants — DistilBERT, MiniLM, bge-small. 1-3% accuracy drop for 2-3x speedup and CPU deployability.

🧠 Reflection — encoders are the unsung workhorse of modern NLP. Every search bar with semantic understanding, every recommendation system, every retrieval-augmented chat — they're all running encoder models in the background. GPT and Claude get the press; BERT family does the heavy lifting.

🚀 Tomorrow, Day 55 — GPT-style decoders. Autoregressive generation. Sampling strategies (greedy, top-k, top-p, temperature). Local LLM with vLLM. Then Day 56: fine-tune vs prompt vs RAG. Then we close week 8.

💼 Three days left in week 8.

👋 See you in the morning.