Top-k retrieval — the parameters that matter

Top-k = how many chunks you retrieve and stuff into the prompt.

- k=1 — minimal context, brittle
- k=3-5 — typical, balances context with noise
- k=10+ — risk of polluting prompt with irrelevant chunks

LLMs don't read perfectly through long context (look up 'lost in the middle'). More chunks isn't always better. The sweet spot depends on chunk size and document density.

Start at k=4. Tune up if recall is bad, down if answers get noisy.

Most embedding models are trained with cosine similarity. Use that.

If the model docs say 'inner product' or 'dot product', use dot. Match the metric the model was trained with.

Euclidean (L2) is rare for text but common for images. If your DB lets you choose, ask 'what was the model trained with?' first.

Using the wrong metric silently degrades retrieval. The model still returns ranked results — just from a slightly distorted similarity space.

If you know the user is asking about 'product X' or 'docs from 2025', filter on metadata BEFORE the ANN. Smaller candidate set, faster search, and irrelevant docs can't pollute results.

Pure vector search misses exact-keyword matches (product codes, names, IDs). BM25 catches those but misses semantic similarity.

Hybrid search runs both, normalises scores, and combines. Reciprocal Rank Fusion (RRF) is the simplest blend that works.

Qdrant, Weaviate, OpenSearch all support hybrid native. For pgvector + Postgres, run BM25 (full-text search) and vector queries in parallel and RRF in code.

Typically a 5-15% relevance lift over either alone. Free win.

Day 61 done.

- Start at k=4
- Match metric to model
- Pre-filter with metadata
- Hybrid > vector alone

Tomorrow (Day 62): augmenting prompts with retrieved context — the part that often surprises engineers.