The RAG Playbook: Retrieval Patterns That Survive Real Data

Retrieval-augmented generation is the workhorse pattern of enterprise AI. It's also the pattern most likely to look great in week one and fall apart in week six, when the real corpus lands and "answers" start turning into confidently-wrong paragraphs stitched from three unrelated policy documents.

This is the playbook we now use on every knowledge-heavy engagement.

Stop chunking. Start segmenting.

Fixed-size chunking is the original sin of RAG. It cuts contracts mid-clause, splits tables from their headers, and cheerfully separates a question from its answer. We segment semantically — by heading, by section, by table — and store the segment tree, not a flat list. Retrieval then returns segments plus their parents, which restores the context a naive chunker throws away.

Hybrid search is not optional

Dense embeddings are wonderful at meaning and terrible at part numbers, SKU codes, statute numbers, and any of the other identifiers your business actually runs on. BM25 (or its modern cousin, SPLADE) finds those instantly. Every one of our retrieval services runs dense + sparse in parallel and fuses with reciprocal rank fusion. It is a five-hour change with an outsized impact on quality.

Rerank ruthlessly

A cross-encoder reranker on the top-50 candidates is the single highest-leverage quality improvement we deploy. Cohere's Rerank 3, or a fine-tuned bge-reranker for cost-sensitive workloads, both punch far above their weight. Precision at 5 typically doubles.

Design for freshness, not just accuracy

• Every document has a source-of-truth timestamp indexed as metadata.
• The generator is prompted with the retrieved timestamps and instructed to prefer newer material on conflict.
• A nightly job re-embeds anything touched by an upstream change — not the whole corpus.
• Deletions propagate. This sounds obvious. It is almost never done.

Measure what you claim

"The AI is smarter now" is not a metric. We ship every RAG system with an eval set built from real user questions, scored on faithfulness, relevance and citation accuracy. If we can't move those numbers, we don't ship the change.