What I changed after my first RAG system

I was debugging the wrong component

When an answer was wrong, my instinct was to change the prompt. That is the most visible lever and the least effective one. In practice, the majority of bad answers came from the model being handed passages that did not contain the answer — at which point no prompt saves you.

The diagnostic that fixed my workflow was cheap: before looking at the generated answer, look at what was retrieved. If the answer is not in the retrieved context, the generation step is not the problem, and tuning it is wasted effort.

Chunk by answerable unit, not by document

My first chunking strategy split documents by length because that was the default. It produced chunks that started mid-table and ended mid-sentence, and embeddings for those chunks describe nothing in particular.

Splitting instead along the natural boundaries of the content — one record, one procedure, one specification — produced chunks whose embedding actually corresponds to a question someone might ask. Retrieval quality improved more from this than from any model change I made.

You need an evaluation set before you need a better model

For an embarrassing amount of time, my evaluation process was asking the system questions I had thought of and judging the answers by eye. That method cannot detect a regression, and it flatters the system, because you unconsciously ask questions you know it handles.

A modest set of twenty to thirty real questions with known correct answers is enough to change the character of the work. It turns 'this feels better' into 'retrieval hit rate went from 60% to 85%', and it makes it obvious when a change that improved one class of question broke another.

• Measure retrieval separately from generation: did the right chunk make it into the context?
• Include questions the system should refuse — a RAG system that confidently answers from nothing is worse than one that says it does not know.
• Keep the questions from real users. The ones you invent are always the ones you already handle.

Grounding is a UI concern too

An answer that cites which record it came from is worth more to an operator than a better-worded answer that does not, because it is checkable. When the system is wrong, a citation turns a mysterious failure into an obvious one.

That also changes the trust dynamic in the operator's favour. They stop asking 'is this thing right' and start asking 'is this source right', which is a question they are actually equipped to answer.