RAG is mostly a search problem

A client came to us in July with a familiar complaint. They'd built a chatbot over their internal policies. It "hallucinated". Could we switch it to a better model?

We looked at the logs before touching the model. For the questions it got wrong, the right paragraph was almost never in what it had been given to read. The model wasn't making things up out of nowhere. It was doing its best with the wrong pages.

That's most RAG projects I've seen. The "G" gets the attention. The "R" does the damage.

Look at what you retrieved

The first thing we do on any of these projects is boring: take 30 to 50 real questions, run retrieval only, and read what comes back. No model involved.

For this client, two problems jumped out in about an hour.

Chunks were cut in silly places. Their splitter cut every 500 tokens, regardless of headings. A rule and its exception often landed in different chunks. The bot would confidently quote the rule.

Exact terms didn't match. Someone would ask about "form 12B" and pure vector search would return paragraphs about forms in general. Embeddings are great at meaning and surprisingly bad at codes, IDs and product names.

What fixed it

None of this is new. It's just often skipped.

• Split on structure. Headings and sections first, size second. Keep the heading path with each chunk ("Leave policy > Sick leave > Exceptions") so the chunk makes sense on its own.
• Hybrid search. Run keyword search (BM25) next to vector search and combine the results. Codes and names come back through keywords. Fuzzy questions come back through vectors.
• Rerank. Pull 30 candidates, then use a reranker to pick the best 5. This one change helped more than anything else for this client.
• Show sources. Every answer links to the paragraphs it came from. Users trust it more, and when it's wrong, they can tell you which page was wrong.
• Let it say "I don't know". If nothing relevant comes back, say so and point to a person. A bot that admits it doesn't know is far more useful than one that guesses.

We kept the same model

The client's answer quality went from "people stopped using it" to "people use it daily", and we didn't change the model at all. We did eventually move them to a cheaper one, because once the right text was going in, a smaller model was plenty.

The questions I ask now

When someone says their RAG system hallucinates, I ask:

1. For a wrong answer, was the right text in the context?
2. If not, why not? Chunking, matching, or the document just isn't indexed?
3. If yes, is the prompt telling the model to stick to the sources?

Nine times out of ten we never get to question 3.

Written by Balaji B at eStreetz. Questions or disagreements: naveen@estreetz.com