RAG that survives production: grounding, evals & guardrails
Asad Siddiq4 min read · Jun 24, 2026
Retrieval-augmented generation demos beautifully and breaks quietly. The gap between a weekend prototype and a system that answers thousands of real questions a day is almost entirely about grounding, evaluation, and guardrails, not the model.
A RAG pipeline has four moving parts: chunking, retrieval, generation, and verification. As NIST defines it, RAG pairs a generative model with a separate information retrieval system that supplies relevant context for each query. Most teams obsess over the model and ignore the other three parts. In production, the model is rarely the bottleneck. The document ingestion and chunking layer and unmeasured drift are.
Why Chunk for Meaning Instead of Token Count?
Splitting documents into fixed 500-token windows is the single most common cause of wrong answers. It cuts tables in half and separates a claim from its caveat. Chunk on semantic boundaries (sections, list items, Q&A pairs) and attach metadata (source, date, section title) so the model can cite and the retriever can filter.
Is Retrieval a Ranking Problem or a Search Problem?
It's a ranking problem. Treating it as plain search is where most RAG systems lose accuracy.
- Use hybrid retrieval: combine dense vector similarity with keyword (BM25) so exact terms and acronyms aren't lost.
- Re-rank the top candidates with a cross-encoder before they reach the model. It's the cheapest accuracy you'll ever buy.
- Always retrieve more than you need, then filter by metadata (recency, permissions, source trust).
Why Can't You Improve a RAG System Without Evaluating It?
Before launch, build a golden set of 100–300 real questions with verified answers. Score every change against it for faithfulness (is the answer supported by the retrieved context?) and relevance (did it answer what was asked?). This is the discipline a real RAG and knowledge systems engagement is built around: ship an eval, then ship the feature, never the other way around.
If your only test for a RAG system is 'it looked right when I tried it', you don't have a system. You have a slot machine.
How Do Guardrails Stop a RAG System From Hallucinating?
The safest answer is often 'I don't have enough information to answer that.' Add a guardrails and safety check that compares the generated claim against the retrieved passages and abstains when support is weak. Log every abstention. It's a map of exactly where your knowledge base has holes.
Frequently asked questions
Why does RAG work well in a demo but break in production?
Demos are tested on a handful of friendly questions. Production traffic hits edge cases in chunking, retrieval ranking, and content drift that a demo never exercises. Those three, not the model, are where real RAG systems usually fail.
What's wrong with fixed-length chunking?
Fixed 500-token windows split mid-table and mid-claim, separating a statement from the caveat that qualifies it. Semantic chunking (splitting on sections, list items, and Q&A pairs) keeps related content together so the model can retrieve and cite it correctly.
Do I need a reranker if I already use vector search?
Yes, if accuracy matters. Vector similarity alone misses exact terms and acronyms; hybrid retrieval (vector + keyword) plus a cross-encoder reranker on the top candidates is one of the cheapest accuracy gains available in a RAG pipeline.
How big does a golden eval set need to be?
100–300 real questions with verified answers is enough to catch regressions reliably. The set should be scored for both faithfulness (is the answer supported by retrieved context) and relevance (did it answer what was asked) before every change ships.
Should a RAG system ever refuse to answer?
Yes, by design. When retrieved context doesn't support a confident answer, the system should say so rather than guess. Every abstention should be logged; the pattern of abstentions maps exactly where the knowledge base has gaps.
Want this built for your team?
We ship production-grade AI like this across every industry — in weeks, not months.