RAG systems that survive contact with real data
12 June 2026 · 6 min read · Nintech AI
Retrieval-augmented generation has a reputation problem: it demos brilliantly and disappoints quietly. The pattern is almost always the same — the team upgraded the model when they should have fixed retrieval, chunked documents by page when the meaning lived in tables, and never measured whether users actually accepted the answers.
Retrieval quality beats model size
If the right passage is not in the context window, no model can cite it. In our engagements the single highest-leverage change is rarely the LLM — it is the retriever: better indexing, query rewriting, hybrid keyword-plus-vector search, and reranking. A mid-tier model over excellent retrieval reliably beats a frontier model over naive retrieval, at a fraction of the cost.
Chunking is a product decision
How you split documents decides what questions the system can answer. Split a rate card by paragraph and you can answer 'what is the rate' but not 'how did rates change across versions'. The right chunking follows the questions users ask — which means someone has to read real queries before choosing a splitter. Treating chunking as an infrastructure default, instead of a product decision, is the most common silent killer of RAG quality.
Citations are a guardrail, not a garnish
Per-answer citations do two jobs. For users, they turn a black-box answer into a checkable claim. For the system, they impose a discipline: if the model cannot point at a retrieved passage, the answer should degrade to 'I don't know' rather than improvise. Systems that enforce citation-or-abstain hallucinate an order of magnitude less than systems that merely encourage citations.
Measure answer acceptance
The metric that matters is not retrieval precision or a benchmark score — it is whether the human on the other end used the answer. Instrument accept/edit/reject on every response and review the rejects weekly. That single loop, run for a quarter, took one client's acceptance from 55% to 92% — and told us exactly which fixes mattered, in which order.
Working on something like this? Talk to an engineer.