Chunking — the deceptively hard step

If your chunks split mid-sentence or fragment a thought, no embedding model can rescue it.

The goal: each chunk should be self-contained on its topic. Retrieving it should give the LLM enough context to answer without other chunks.

Three common chunkers:
- Fixed-size — naive, breaks sentences. Avoid.
- Recursive (split by ¶, then sentence, then word) — LangChain default.
- Semantic (split where embedding similarity drops) — newer, sometimes better.

Your first improvement on a RAG that's misbehaving is almost always better chunking.

Two parameters dominate:

- Chunk size — too small = no context; too big = retrieval gets noisy. 256-512 tokens is the sweet spot for most docs.
- Overlap — let consecutive chunks share 10-20% of tokens. Captures context that crosses boundaries.

For structured content (tables, code, headings), chunk by structure — keep a section together. For prose, recursive character splitter on sentence boundaries.

Log the average chunk length. If most chunks are tiny or huge, your splitter isn't tuned for the content.

LangChain's RecursiveCharacterTextSplitter tries paragraph splits, then sentence, then word, then char. Almost never breaks mid-sentence. Tune chunk_size and chunk_overlap to your docs.

A chunk on its own is just text. With metadata, it's filterable, traceable, and rerankable.

Attach:
- Source URL or filename
- Section / heading
- Date / version
- Document type

Now you can filter retrieval to one product family, one date range, or one section. You can show the user exactly where each answer came from. You can debug 'why was this retrieved' instantly.

Metadata is free; chunks without it are wasted.

Day 58 done.

- Bad chunks ruin RAG
- 256-512 tokens, 10-20% overlap
- Recursive splitter as default
- Metadata everywhere

Tomorrow (Day 59): embedding choice. Why your embedding model matters more than your LLM in many RAG apps.