Retrieval Evaluation Metrics: What to Measure and When

Retrieval Evaluation Metrics: What to Measure and When

Retrieval evaluation metrics score how well a search system, or the retrieval step of a RAG pipeline, finds the right documents for a query. The core set is Precision@k, Recall@k, MRR, MAP and NDCG. Pick them by the way your users or your language model consume results: if one good hit is enough, track MRR. If missing a document is costly, track Recall@k. If order across the whole list matters, track NDCG. Below you get each metric defined, one worked example with real arithmetic, the data you need, and a way to choose.

TL;DR

• Retrieval metrics (Precision@k, Recall@k, MRR, MAP, NDCG) measure whether the right documents were found and how high they ranked. They do not measure whether the final answer is good.
• Precision@k and Recall@k ignore order inside the top k. MRR, MAP and NDCG reward putting relevant items early.
• Every metric needs a labeled set: queries paired with the documents or chunks that truly answer them. NDCG also needs graded labels.
• Evaluate retrieval and generation separately first, then test end to end, so you can see where a pipeline fails.
• Choose metrics by failure cost, and treat any offline score as a proxy for real outcomes.

Table of Contents

• What retrieval metrics measure, and what they leave out
• The five metrics you will actually use
• A worked example with real numbers
• Metrics at a glance
• What data you need before any score is trustworthy
• Choosing metrics for your use case
• Our perspective
• Putting a retrieval scorecard in place
• FAQ
• Sources

What retrieval metrics measure, and what they leave out

A retrieval metric takes a query, the ranked list your system returned, and a set of relevance judgments, then outputs a number. That number answers one question: did the system surface the documents a person or a model needed?

In a RAG system this is only half the picture. Cohere’s evaluation guide treats the two halves separately: retrieved documents are compared against a set of golden documents using precision, recall and MAP, while generated answers are judged on faithfulness, correctness and coverage of the claims in the answer. Anyscale’s documentation makes the same split and adds that BLEU and ROUGE cannot measure retrieval quality or whether an answer stays faithful to the retrieved context. It recommends testing retrieval and generation on their own before running end-to-end tests.

The reason to start with retrieval is simple: it sets the ceiling. Anyscale’s documentation says retrieval performance is the upper bound on final answer quality. If the right passage never reaches the model, no prompt can recover it. If you want the background on the architecture itself, our definition of retrieval-augmented generation covers it.

Retrieval metrics will not tell you whether the model invented something. That is a generation problem, covered by faithfulness checks, and it is the territory of hallucination. Keep the two scorecards apart and you can tell a search problem from a model problem.

The five metrics you will actually use

Below, k is the number of results you look at (the cutoff), and a result is “relevant” if your labels say it answers the query.

Precision@k

Precision@k is the number of relevant items in the top k divided by k. It answers: of what I showed, how much was useful? It is order-unaware, so a relevant item at rank 1 and one at rank k count the same. Use it when each returned item costs something, for example when every chunk you retrieve consumes context window and adds noise.

Recall@k

Recall@k is the number of relevant items found in the top k divided by the total number of relevant items that exist for that query. It answers: of everything that mattered, how much did I catch? It is also order-unaware. Use it when missing information is expensive, such as policy lookups, contract clauses or clinical guidance, where one absent document changes the answer.

There is a practical catch. Recall depends on knowing all the relevant documents, which is rarely true for a large corpus. The Webis information retrieval lecture notes point out that pooling, a common way to build judgments, has limits including self-selection bias and unknown recall for documents ranked below the pooling depth. Treat recall numbers as measured against your labeled set, not against the universe of your documents.

Mean Reciprocal Rank (MRR)

The Stanford CS276 handout defines MRR as the average, across queries, of the reciprocal of the rank of the first relevant result. If the first relevant item is at rank 1 the score is 1, at rank 2 it is 0.5, at rank 4 it is 0.25. Scores run from 0 to 1, and higher is better, as the Harvard IACS lecture notes put it. MRR fits cases where one correct answer is enough: a lookup assistant, a “find the policy” box, or a pipeline that only passes the top chunk to the model. It ignores everything after the first hit.

Mean Average Precision (MAP)

MAP is the mean, across queries, of average precision. According to the same Stanford handout, it rewards systems that rank relevant items early while measuring the full ranked list. Where MRR cares about the first hit, MAP cares about all of them. It uses binary relevance. Use it when queries have several relevant documents and you want one score that reflects both coverage and order.

Normalized Discounted Cumulative Gain (NDCG)

NDCG supports graded relevance, for example 0 for irrelevant, 1 for partly useful, 2 for exactly right. The Stanford handout describes it as normalizing DCG by the DCG of the ideal ranking. Gains are discounted by position, so burying a perfect document low in the list costs you. Dividing by the ideal ordering gives a score from 0 to 1 and, as the Harvard lecture notes explain, removes the problem of scale varying from query to query. Use it when some documents are clearly better than others and ranking order across the list matters.

A worked example with real numbers

Take one query. Your labels say three chunks are relevant: A, B and C. Your retriever returns five chunks. Only two of them are relevant, at ranks 2 and 4.

Here is how each metric scores this list (the example is illustrative, built to show the arithmetic):

The DCG line works like this: relevant items at ranks 2 and 4 contribute 1/log2(3) and 1/log2(5), about 0.631 and 0.431, for 1.062. The ideal list would place all three relevant items at ranks 1 to 3: 1 + 0.631 + 0.5 = 2.131.

Notice what each number tells you. Precision says you returned noise. Recall says a relevant chunk was missed entirely. MRR says the first useful chunk came second. MAP is lowest because it punishes both the missing chunk and the late arrivals. Reading them together beats reading any one alone.

In practice you average each metric over your whole query set, and you report it at the cutoff your pipeline really uses. If your generator receives five chunks, report at k = 5, not k = 100.

Metrics at a glance

What data you need before any score is trustworthy

The metrics are easy. The labels are the work. The University of Basel multimedia retrieval lecture lists what an effective benchmark needs: a document collection, queries, a method for judging relevance, and clear performance goals. The Webis notes describe the same trio for a lab experiment: a corpus, a set of information needs, and relevance judgments, which can be binary or graded.

For a RAG team, that translates into a concrete checklist:

1. Collect real queries. Pull them from support tickets, search logs, or interviews with the people who will use the system. Invented questions drift away from real phrasing.
2. Define the unit of retrieval. Decide whether you judge documents, pages or chunks, and give each a stable ID. Metrics only work if every returned chunk can be matched to a label.
3. Label relevance. For each query, list the IDs that genuinely answer it. Binary labels are enough for Precision, Recall, MRR and MAP. NDCG needs a grade.
4. Judge sparse labels with care. The Basel lecture notes that sparse judgments can affect precision at the top of the list, and that metrics relying on full recall may be less reliable. Unlabeled does not mean irrelevant.
5. Keep a frozen test set. Do not tune your chunking and embedding settings against the same queries you report. Hold some back.
6. Version everything. Record the index build, chunk size, embedding model and query set with each score so changes are comparable.

A synthetic start is legitimate when you have no logs yet: take a passage from your corpus, extract the facts it states, and generate questions those facts answer. The source passage becomes the labeled relevant chunk. Review a sample by hand, because generated questions are often easier than real ones.

Choosing metrics for your use case

There is no single best metric. The Basel lecture says retrieval goals range from answering fact-checking questions in milliseconds to surfacing long reports for expert analysts, and that a collection built for web search will not capture the challenges of legal discovery or medical literature review. Start from the failure you cannot afford.

• A miss is the expensive error (compliance lookups, clinical or legal knowledge bases): lead with Recall@k, with k set to the number of chunks you pass to the model.
• Noise is the expensive error (small context windows, per-token cost): lead with Precision@k.
• Success depends on the first hit (a single-answer assistant): lead with MRR.
• Several documents matter and order matters (research or analyst workflows): lead with MAP or NDCG, and invest in graded labels for NDCG.

Then add one guardrail metric from the generation side. Faithfulness, whether an answer is consistent with the retrieved context, is the one most teams add first, since low faithfulness signals the model is inventing content, as Anyscale’s documentation describes in its generator evaluation section.

Limits you should plan around

Offline metrics are proxies. Microsoft Research’s paper “What Matters in a Measure?” argues that metric choice needs both scientific criteria, such as sensitivity and validity, and practical ones, such as efficiency, robustness and debuggability. It also warns that improving a proxy metric might not translate into real-world gains, and that a metric lacking sensitivity can be flat and hide real improvements.

In practice that means three habits. Pair the offline score with a small human review of failures each cycle. Track metrics per query segment, not only as one average, so a weak category does not hide behind a strong one. And ask whether a metric moved because the system improved or because the test set got easier.

Our perspective

Most teams we meet start by tuning the language model and discover later that the answers were wrong because the right passage never arrived. We start the other way: build the labeled query set first, score retrieval alone, and only then look at generation.

Our view is that a retrieval scorecard belongs to the business, not the vendor. The query set, the labels and the metric definitions should be owned by your team and run against your own data, so you can re-run them after every change to the index. That is how we approach ownership and the absence of vendor lock-in in our RAG and knowledge systems work, and it is why we favor weekly demos on real data over slide decks.

We also choose metrics by consequence. In a regulated workflow, a missed document is a different kind of failure than a noisy one, so the metric and the cutoff follow from the workflow, not from a leaderboard.

arosplatforms team

Putting a retrieval scorecard in place

If you are building or inheriting a RAG system, a reasonable first pass looks like this:

1. Gather 50 to 100 real queries from the people who will use the system.
2. Label the relevant chunk IDs for each, with a subject-matter expert reviewing a sample.
3. Report Recall@k and MRR at your real cutoff, plus Precision@k if context cost matters.
4. Add NDCG once you have graded labels and a ranked list that users or models actually read in order.
5. Review the worst-scoring queries by hand before changing anything. The pattern in the misses, such as tables, acronyms or stale documents, usually points to the fix.
6. Re-run the same set after every change to chunking, embeddings, filters or reranking.

If you want a team to design the evaluation set and the pipeline together, our RAG and knowledge systems service covers answers grounded in your own data, and an AI readiness assessment is a sensible first step to scope the data, the use case and the success measures. You can also see how we keep enterprise knowledge bases fresh without retraining, which matters because a stale index quietly lowers every retrieval metric above.

FAQ

What is the difference between retrieval metrics and RAG evaluation metrics?

Retrieval metrics, such as Precision@k, Recall@k, MRR, MAP and NDCG, score whether the right documents were found and how they were ranked. RAG answer metrics, such as faithfulness and correctness, score the generated response. Cohere’s guidance evaluates the two separately, which makes it easier to locate a failure.

Which retrieval metric should I start with?

Start with Recall@k at the number of chunks your model actually receives, and add MRR if one good result is usually enough. Both work with simple binary labels. Move to NDCG when you have graded relevance judgments and users or models read the list in order.

How many labeled queries do I need?

There is no universal number. A few dozen real queries with careful labels are enough to find obvious problems, and more are needed to detect small differences between configurations. Microsoft Research’s work on metric sensitivity is a reminder that a coarse test set can miss real improvements.

Do BLEU and ROUGE work for retrieval?

No. Anyscale’s documentation states that these text-generation metrics cannot measure retrieval quality or whether an answer is faithful to retrieved context. Use ranking metrics for retrieval and faithfulness-style checks for the answer.

Can a good retrieval score still produce a bad answer?

Yes. Strong retrieval only raises the ceiling on answer quality. The model can still ignore or misread good context, which is why generation needs its own checks alongside the retrieval scorecard.

Sources

• Stanford CS276 evaluation handout
• Harvard IACS lecture: Introduction to RAG evaluation
• University of Basel: Multimedia Retrieval, performance evaluation
• Webis lecture notes: laboratory experiments in information retrieval
• Microsoft Research: What Matters in a Measure?
• Cohere: Deep Dive Into Evaluating RAG Outputs
• Anyscale: RAG evaluation

Recommended

• Retrieval-Augmented Fine-Tuning: Definition
• Evaluation: Definition
• Retrieval-Augmented Generation (RAG): Definition
• RAG & Knowledge Systems: Answers Grounded in Your Data