What does the MRR metric represent?

Mean reciprocal rank (MRR) represents the average of the reciprocal ranks of the first relevant result for each evaluated query. For a query where the first relevant item appears at position r, the contribution is 1/r. These per-query values are then averaged across the test queries you evaluate (Elastic ranking evaluation).

How MRR is computed

To calculate MRR you need:

• A set of queries you explicitly evaluate.
• Relevance judgments for results, plus a threshold that determines which results count as “relevant.” In tooling like Elasticsearch’s rank evaluation, this threshold is configurable as relevant_rating_threshold and you can also limit evaluation to the top k results (Elastic ranking evaluation).

For each query, find the rank of the first result that meets your relevance threshold, take its reciprocal (1/r), and average those reciprocals across all evaluated queries. If no relevant result is retrieved for a query, its reciprocal rank contribution is 0 in the average. If a query fails to run or cannot be scored, report it separately rather than treating it as 0; APIs such as Elasticsearch’s return a failures section to record per-query errors (Elastic ranking evaluation).

A worked example

Suppose you evaluate five queries against the top 10 results returned by your system, using binary relevance (relevant if rating ≥ 1):

{
  "example_type": "hypothetical",
  "evaluation_cutoff": 10,
  "evaluated_queries": [
    {
      "id": "Q1",
      "first_relevant_rank": 1
    },
    {
      "id": "Q2",
      "first_relevant_rank": 3
    },
    {
      "id": "Q3",
      "first_relevant_rank": null
    },
    {
      "id": "Q4",
      "first_relevant_rank": 2
    },
    {
      "id": "Q5",
      "first_relevant_rank": 5
    }
  ],
  "evaluated_query_count": 5,
  "mrr_rounded_to_4_decimal_places": 0.4067
}

For this hypothetical set, the calculation is (1 + 1/3 + 0 + 1/2 + 1/5) / 5 = 61/150, approximately 0.4067. Q3 completed but had no relevant result in the top ten, so it contributes zero. MRR is an average of reciprocals, not an average result position; taking its inverse does not recover an average rank. A query that failed or could not be evaluated must be reported separately, with the denominator stated explicitly.

What MRR does and does not measure

What it captures

• Speed to first relevant result: Only the rank of the first relevant item affects the score (Elastic ranking evaluation).
• Binary relevance by threshold: Graded judgments can be mapped to “relevant” using a threshold such as relevant_rating_threshold (Elastic ranking evaluation).
• Evaluation scope: You can cap evaluation to the top k results, reflecting your product’s UI or user behavior (Elastic ranking evaluation).

What it does not capture

• All relevant results: MRR ignores results after the first relevant one. If multiple relevant results matter, distinguish recall, which measures retrieved coverage, from DCG/NDCG, which evaluate relevance-weighted ordering (Elastic ranking evaluation).
• Fact or citation truth: MRR reflects ranking behavior given your relevance labels; it does not validate whether a citation or claim is actually correct. For how to think about citation accuracy, see How accurate are AI citations?
• Ground-truth quality: MRR’s meaning depends on the quality of your judgments. For guidance on creating reliable labels, see What is ground truth in the context of AI?

Using MRR in tooling

Many search systems provide an evaluation API that computes MRR from your test suite. For example, Elasticsearch’s _rank_eval endpoint lets you submit representative queries, relevance ratings per query, and choose mean_reciprocal_rank as the metric. You can set the evaluation cutoff k and a relevance threshold for binary decisions:

The following is a metric-selection fragment, not a complete _rank_eval request. A complete evaluation also needs the actual queries and relevance ratings.

{
  "metric": {
    "mean_reciprocal_rank": {
      "k": 10,
      "relevant_rating_threshold": 1
    }
  }
}

The API’s response includes an overall metric score, a per-query breakdown, and a failures section for queries that could not be evaluated, which should be reported rather than silently treated as zero (Elastic ranking evaluation).

Checklist: compute and report MRR correctly

• Assemble a representative set of explicit queries and per-query relevance judgments.
• Decide a relevance threshold and an evaluation cutoff k.
• For each query, find the first relevant result’s rank r and compute 1/r; use 0 if none is retrieved.
• Average these values across all evaluated queries; state the denominator.
• Log and report failed or unavailable measurements separately; do not silently substitute zeros.
• Pair MRR with recall or DCG/NDCG if you also care about multiple relevant results.

FAQ

Does MRR require binary relevance?
MRR uses the rank of the first result that meets your definition of “relevant.” If you collect graded judgments, you can apply a threshold (for example, rating ≥ 1) to decide relevance before computing MRR (Elastic ranking evaluation).

How is MRR different from reciprocal rank fusion (RRF)?
MRR is an evaluation metric per query set. RRF is a method to combine multiple ranked lists into a new ranking. For a primer on RRF vs reranking, see reciprocal rank fusion vs reranking.