How to Evaluate RAG Retrieval Quality Before You Tune the Model
Evaluate retrieval separately from generation. Build a versioned, representative query set with the relevant source documents identified, measure whether the right evidence is retrieved and ranked, then evaluate whether the answer uses that evidence correctly. A single final answer score cannot tell you whether the problem is a missing source, a bad chunk, a weak filter, an incorrect rank, or a model that ignored good context.
This distinction matters because retrieval-augmented generation (RAG) is a pipeline, not a single model setting. The original RAG research paper describes combining a generator with a non-parametric memory. In a production application, the quality of that memory depends on source preparation, indexing, search, access filters, ranking, context assembly, and answer verification.
Start with a labeled evaluation set
Do not begin by tuning chunk size or swapping embedding models. First write down what a good result means for the task. For each query, record the expected answerability, the source documents that should be retrieved, any required passage or field, the access scope, and what the system should do when the corpus does not contain an answer.
Use a mixture of synthetic and representative business documents without copying confidential customer material into a test fixture. Include terminology variants, long questions, short keywords, misspellings, table lookups, conflicting revisions, and questions whose correct response is that the evidence is unavailable. A benchmark that contains only easy answerable questions can reward confident guessing.
Where did the supporting passage go?
Track the same passage through each layer. A fluent answer cannot show where it was lost.
-
01Source inventory
Source preparation
The passage exists
Failure: Missing document
Confirm the expected source was ingested and is current.
-
02Passage + heading
Indexing
Meaning survives chunking
Failure: Lost context
Inspect chunk boundaries, headings, IDs, and metadata.
-
03Filtered candidates
Retrieval
The passage is returned
Failure: Evidence not found
Inspect the query, access filters, and relevant returned sources.
-
04Final assembled context
Ranking and context
The passage reaches the model
Failure: Evidence dropped
Check rank, truncation, duplicates, and conflicting revisions.
-
05Claim + cited source
Answer support
The claim uses the passage
Failure: Unsupported answer
Check that the citation supports the claim; abstain when evidence is insufficient.
Measure retrieval before judging the answer
For a labeled query set, retrieval metrics can show whether relevant evidence appears in the returned set and where it appears. Recall@k asks what fraction of the known relevant sources appears in the first k results. If a query has exactly one relevant source, that reduces to a yes-or-no result; otherwise it is different from hit rate. Precision@k asks what fraction of the returned set is relevant. Hit rate, also called success@k in some evaluations, asks the simpler yes-or-no question of whether at least one acceptable source appears. Mean reciprocal rank or nDCG can be useful when the position and ordering of relevant results matter.
These metrics are not interchangeable and their thresholds are not universal. A support workflow that needs one authoritative passage may care about top-rank precision. An exploratory research workflow may value broader recall. Define the metric and threshold from the task, then keep the labels and evaluation set stable enough to compare changes.
| Layer | Question | Evidence to inspect |
|---|---|---|
| Source preparation | Did the system receive the right documents and fields? | Source inventory, parser output, permissions, freshness, and version. |
| Indexing | Were useful passages created with the metadata needed for filtering? | Chunk boundaries, IDs, metadata, embeddings, and index counts. |
| Retrieval | Did the search return relevant evidence for the query? | Recall@k, precision@k, hit rate, and filtered result samples. |
| Ranking and context | Did the best evidence arrive in a usable order and fit the context? | Rank positions, reranker changes, truncation, duplicates, and conflicts. |
| Answer support | Did the response use the evidence without adding unsupported claims? | Citation correctness, completeness, abstention, and human review. |
Debug the five common failure layers
Suppose a synthetic policy question requires two passages: A defines eligibility and B lists the exceptions. The first three results are A, C, and D. Only A is relevant. Count unique passage IDs against a fixed set of relevance labels.
One hit. Half the evidence.
The question needs eligibility and exceptions. Retrieval returns only one of them.
Required evidence
Returned ranking
- 01ARelevant
- 02CNot relevant
- 03DNot relevant
- Recall@3
- 50%1 of 2 required passages
- Precision@3
- 33.3%1 of 3 returned passages
Hit@3 = 1. Reciprocal rank = 1. Both look successful, but neither detects the missing exception. MRR averages reciprocal rank across queries.
What changes if B replaces C at rank 2?
Recall@3 becomes 100% (2/2); precision@3 becomes 66.7% (2/3). Hit@3 and reciprocal rank stay at 1. Complete retrieval still does not prove the answer uses the evidence correctly.
These are illustrative calculations, not Software Sushi benchmark results or recommended thresholds. For metric foundations, see Stanford's Introduction to Information Retrieval: ranked evaluation. Decide whether your labels describe documents or passages before comparing runs. Multiple chunks from the same document should not silently inflate document-level recall.
1. The source never entered the system
A missing document, failed parser, stale sync, or incorrect permission filter cannot be repaired by a better prompt. Compare the expected source inventory with what the index actually contains. Test updates and deletions as well as initial ingestion.
2. The source entered, but the index made it hard to find
Bad chunk boundaries can separate a heading from the content that gives it meaning. Missing metadata can make it impossible to filter by product, customer, date, or access scope. Inspect parsed text, chunk IDs, overlap, metadata, and the exact text sent to the embedding or indexing step.
3. Relevant text was retrieved, but the rank or filter was wrong
Query rewriting, hybrid search, access filters, reranking, and freshness policies can change which passages reach the model. Save the query, candidates, scores, filters, and final order for a test run. Otherwise a team may tune the generator to compensate for a retrieval or ranking problem.
4. Good evidence was retrieved, but context assembly weakened it
Truncation, duplicates, contradictory revisions, a poor order, or an overfilled context window can keep useful evidence from being usable. Inspect the final context sent to the generator, not only the retrieved candidates, and test the token budget and assembly rules with long and conflicting examples.
5. The answer did not stay within the evidence
The generator can ignore relevant context, combine conflicting passages, overstate a partial source, or answer an unanswerable question instead of abstaining. Evaluate support at the claim level and inspect whether citations actually entail what the response says. Citation presence alone is not citation correctness.
Our guide to RAG versus fine-tuning addresses the higher-level architecture choice. Retrieval evaluation answers the next question: whether the chosen RAG path is returning evidence that the application can use.
Improve the weakest layer in small, comparable changes
Once the failure layer is known, change one meaningful variable at a time and rerun the same evaluation set. A source or parser fix should be visible in source coverage. A chunking change should be tested against passage boundaries and metadata. A retrieval change should be judged on relevant results and access filters. A prompt or model change should be judged on evidence use and abstention, not only on fluency.
Keep latency and cost beside quality. A reranker may improve ordering while adding a step to every request. A larger context may increase answer support while raising runtime cost or making conflicting evidence harder to manage. A smaller model may meet the task's quality threshold with a simpler operating profile, but that is a result to demonstrate on the benchmark, not a general rule.
For an LLM application, production LLM development should connect retrieval, permissions, evaluation, feedback, and monitoring. An AI/ML consulting review can help teams decide which layer to fix first when the evaluation results are mixed.
Use human review and automated judges carefully
Keep a development set for tuning and a separate held-out set for release evaluation. Repeatedly adjusting the system to the same questions can overfit the benchmark. Report results by query type and access scope, alongside the overall average, so a gain on common questions cannot hide failures on unanswerable or restricted ones.
Automated evaluators can make a large test set easier to triage, but they are not ground truth. A judge can prefer a fluent answer, miss a subtle unsupported claim, or apply the wrong standard to an ambiguous question. Keep a human-reviewed sample with explicit criteria: is the answer supported, complete for the task, correctly cited, appropriately uncertain, and within the user's access scope?
Use disagreement as information. If a judge and a reviewer diverge, add the example to the evaluation set or clarify the rubric. Over time, the benchmark should capture the failure modes that matter to the workflow, including when the correct behavior is to say that the evidence is insufficient.
Every material change to the corpus or retrieval pipeline should produce a comparable evaluation result before release, with a rollback path when regressions appear.
Turn evaluation into a release habit
Track the versions that produced every result: query set, source corpus, parser, chunking rules, embedding model, index, filters, reranker, prompt, generator, and application code. Store the retrieved IDs and answer evidence for failed cases. This makes a regression explainable and gives the team a way to compare a fix with the baseline.
After release, monitor what can be measured immediately, such as latency, empty retrievals, filter failures, citation coverage, and user corrections. When labels arrive later, connect them back to the original query and answer. A feedback loop is useful only when the team can act on it: update the corpus, repair the pipeline, adjust the rubric, or roll back a change.
Compare evidence. Then decide.
- Comparable versions
- Run the same labeled queries against baseline and candidate. Record source, index, and model versions.
- Quality criteria
- Review missing evidence, permissions, unsupported claims, and human-reviewed disagreements against the agreed thresholds.
- Operating budget
- Compare latency and cost. Name the release owner and confirm the rollback path.
Questions to ask before approving a RAG system
Can the team show the expected source for a test query?
If nobody can identify what evidence should support an answer, the system cannot be evaluated at the retrieval layer. Start by labeling a small but representative set.
Does a retrieval score include permissions and freshness?
A result is not relevant if the user cannot access it or if it is superseded by a newer source. Keep access scope and source version in the benchmark.
What happens when the corpus has no answer?
Define an abstention or clarification path. Do not reward an answer that sounds complete when the expected behavior is to surface missing evidence.
Can you tell whether a regression came from the index or the model?
Version the retrieval and generation layers independently and retain retrieved source IDs. A final answer score alone cannot localize the regression.
Is the evaluation tied to an owner and release decision?
Someone should own the benchmark, review failures, approve changes, and decide when rollback or further investigation is required.
Frequently Asked Questions
What is the difference between retrieval quality and answer quality?
Retrieval quality asks whether the system found and ranked relevant evidence. Answer quality asks whether the generator used that evidence accurately, completely, and appropriately. They influence each other but should be measured separately.
Is recall@k enough to evaluate a RAG system?
No. Recall@k is one retrieval signal. Ranking, access filters, freshness, context assembly, answer support, abstention, latency, cost, and human review can all matter depending on the workflow.
Should we switch embedding models first?
Not by default. First identify whether the source, parser, chunking, metadata, filters, ranking, or generation layer is failing. Change a relevant variable and compare it against the same labeled evaluation set.
Can an LLM judge replace human evaluation?
No. It can help triage or scale a rubric, but it can miss unsupported claims and inherit the wrong standard. Keep a human-reviewed sample and investigate disagreement.
How often should a RAG benchmark run?
Run it whenever a material change affects sources, parsing, indexing, retrieval, ranking, prompts, models, permissions, or answer handling. Monitor production feedback as well, because the benchmark is a baseline rather than a complete picture.
Make Retrieval Quality Visible Before It Reaches Users
RAG becomes easier to improve when the team can show what should have been retrieved, what was actually returned, and whether the final answer stayed within that evidence.
If a RAG system is hard to evaluate or its failures are difficult to localize, explore our LLM development services or book a discovery call.