Evaluating RAG + week 9 wrap

RAG quality has two stages. Eval them independently or you'll attribute bugs incorrectly.

1. Retrieval — given a query, did we fetch the right chunks?
   - Metrics: recall@k, MRR, hit@k
   - Needs (query, expected_chunk) pairs

2. Generation — given the chunks, did the LLM answer well?
   - Metrics: faithfulness (does the answer use the context?), correctness, citation accuracy
   - Needs (query, answer, context) triples + a judge

Fix retrieval first. A great LLM can't recover from bad retrieval.

Two libraries automate the boring parts:

- Ragas — focused on RAG specifically. Faithfulness, answer correctness, context relevance. Uses LLMs as judges.
- TruLens — broader LLM observability + RAG eval. Trace and grade in one.

Both need:
1. Eval queries
2. Ground-truth answers (or ground-truth contexts)
3. An LLM as judge (GPT-4o or Claude is typical)

Start with 30 hand-curated queries. Run ragas. The numbers tell you which step (retrieval, prompt, model) to fix.

Build a Dataset, run ragas evaluate(), get back faithfulness + answer_correctness + context_recall numbers per query.

Iterate on chunking / retrieval / prompts. Keep ragas as your CI gate. Quality regressions show up before users see them.

Treat eval scores as commit-by-commit metrics, not one-off reports.

- Run ragas on every PR that touches RAG code
- Fail CI if faithfulness drops > 5%
- Plot trends in a dashboard

A new chunking strategy that improves average looks great until you see p20 dropped. Distribution > average. Ship like a real engineering team — measure regressions before users do.

Week 9 done. 63 days, 315 posts in.

This week:
- Why RAG
- Chunking honestly
- Embedding choice
- Vector DBs without hype
- Top-k tuning + hybrid
- Prompt augmentation
- Eval as CI

Next week: production RAG. Reranking, query rewriting, agentic RAG, GraphRAG, the hard ones.