Rerankers — the second pass that fixes everything

Two-stage retrieval is the modern default:

1. Retrieve top-50 with cheap embedding similarity (millisecond, scalable)
2. Rerank to top-5 with a cross-encoder (slower, more accurate)

A cross-encoder reads query + document together and outputs a relevance score. Way more accurate than separate embedding cosine. Too slow for the full corpus, perfect for the top-50.

Net: 5-30% relevance improvement, often more than switching the LLM. Most underrated upgrade in 2026 RAG.

Open:
- BAAI/bge-reranker-v2-m3 — multilingual, strong default
- mixedbread-ai/mxbai-rerank-large-v1
- jina-reranker-v2

API:
- cohere rerank-3.5 — fastest, paid by query
- voyage rerank-2 — strong English

For self-hosted: bge-reranker-v2-m3 is hard to beat for the cost. Runs on a 8GB GPU. ~50ms per query at top-50.

For speed: cohere. For privacy + cost: open self-hosted.

Use sentence-transformers' CrossEncoder for an open reranker. Pass (query, doc) pairs; get scores. Sort; take top-5. That's the second pass.

Reranking 1000 candidates is wasteful. Past ~50, gains plateau and latency explodes (cross-encoder is O(n × seq_len²)).

Sweet spot:
- Retrieve top-50 cheaply
- Rerank → top-5
- 30-150ms total reranker time on a small GPU

If your reranker is too slow, reduce candidates first, optimise model second. 50 is a great default; tune from your latency budget.

Day 65 done.

- Two-stage retrieve + rerank is default
- Open and API rerankers
- 12-line implementation
- Cap at 50 candidates

Tomorrow (Day 66): query rewriting. Most retrieval failures are query failures, not retrieval failures.