How to Evaluate a Production RAG System Before Deployment

Most RAG demos work. You upload a few clean documents, ask the questions you already know the answers to, and the system looks finished.

Production is a different test. Users ask vague questions, in mixed languages, about documents that contradict each other, and some of what they ask has no answer in the corpus at all. A system can pass every demo and still fail on the first day.

I architected and built the MVP of Auto-RAG, which went to production for 7 to 10 enterprise customers. Each one brought a different corpus, a different language mix, and a different tolerance for a wrong answer. That taught me to stop asking "is this RAG system good?" The useful question is "is it good enough for this customer, on this corpus, for these questions, and how do we know?"

Here is how I would answer that before deployment.

Step 1: Build the question set from real demand

Do not start by generating questions from your documents. A model that reads a passage and writes a question produces a question that passage answers perfectly. That tests almost nothing.

Start with what people ask: support tickets, search logs, emails to the team that owns the documents, or interviews with the people who will use the system. Then make sure the set covers these groups:

The second-to-last row matters most, and teams tend to skip it. Include plenty of questions with no valid answer. A system never tested on those has an unknown refusal rate.

A smaller set that a domain expert has reviewed is worth more than a large generated one. Freeze it, version it, and treat changes to it as seriously as changes to the code.

Step 2: Score retrieval and generation separately

When an answer is wrong, you need to know which half failed. End-to-end scores hide that.

Retrieval: for each question, record which passages should be found, then measure whether they came back and how highly they ranked. If the right passage is not in the context, no model can save the answer.

Generation: given the correct context, did the answer use it? Check three things: it is correct, every claim is supported by the supplied passages, and it did not leave out what matters.

This split changes where you spend time. If retrieval is failing, a better model will only write better-sounding wrong answers. So I would look first at the unglamorous layers: chunking, metadata, query handling and search configuration.

Step 3: Treat refusal as a feature

A RAG system has two ways to be wrong on any question. It can invent an answer when it should refuse, or refuse when the answer was available. These fail differently, and customers feel them differently.

For a government or regulated customer, an invented answer may be far worse than a refusal. For an internal helpdesk, constant refusals make people stop using the tool. So measure both rates, and agree the balance with the customer before launch. That conversation is part of the requirements, and it is much easier before the first bad answer than after it.

Step 4: Compare configurations by experiment

RAG has many settings: chunk size, overlap, embedding model, keyword or hybrid search, reranking, number of passages, prompt template. People argue about these for weeks.

On Auto-RAG I contributed to automated experimentation across retrieval strategies and prompt templates, so tuning became a measured run rather than an argument. You do not need a platform for this. A script that runs your frozen set against a list of configurations and writes one results table is enough to start. Different customers may land on different settings, which is another reason to automate it.

Step 5: Assert behaviour, not wording

A test that compares an answer to a stored paragraph will fail good answers. Models rephrase, and so they should.

Write checks about facts instead. Does the answer contain the correct figure? Does it cite the governing policy? Does it avoid a claim the source does not make? Does it refuse? When I built the evaluation harness for Career Rafeeq, I used assertion-based ground truth with several runs per case aggregated statistically, because a single run of a stochastic system tells you very little.

LLM judges can help with volume. I would keep them advisory, and audit a sample by hand, since even public benchmark verifiers can give the wrong verdict.

Step 6: Test the parts that are not the model

• Permissions. Can a user retrieve a passage they should not see? Test with accounts at different levels. Filtering must happen at retrieval, not in the prompt.
• Freshness. Change a document. How long until answers reflect it? What happens to the old version?
• Latency. Measure under realistic concurrency, including the slowest few percent of requests.
• Cost per answer. Include retries, reranking and long contexts.
• Injected content. Put instructions inside a document. The system should treat them as text.

Step 7: Write the release gate down

Evaluation only changes outcomes if it can stop a release. Before launch, agree on thresholds with the customer, such as minimum correctness on the frozen set, maximum invented answers on unanswerable questions, and zero permission leaks.

Then run the same gate on every change to the prompt, model, chunking or index. A model upgrade is a release. So is a re-index.

After launch

Pre-deployment evaluation is a baseline, not a guarantee. Sample real conversations, review the failures, and add them to the question set. Over time your test set stops reflecting what you imagined and starts reflecting what happens.

If you do only one thing from this list, add fifty questions that have no answer in your documents, and see what your system does. It is the cheapest honest test in RAG, and the result is usually informative.

Written by Eslam Hamed, a Technical Lead and AI Solutions Architect working on enterprise RAG, agentic AI, and model evaluation.