Context Recall
Outputs on this page
Outputs of LLM calls on this page come from earlier runs, some with models that have since been retired, and the code now names current models. Scores and generated text from your run will differ.
Context Recall measures how many of the relevant documents (or pieces of information) were successfully retrieved. It focuses on not missing important results. Higher recall means fewer relevant documents were left out. In short, recall is about not missing anything important.
Since it is about not missing anything, calculating context recall always requires a reference to compare against. The LLM-based Context Recall metric uses reference as a proxy to reference_contexts, which makes it easier to use as annotating reference contexts can be very time-consuming. To estimate context recall from the reference, the reference is split into sentences, and the judge LLM classifies each sentence as attributable to the retrieved context or not. It must return exactly one classification per sentence. If its answer can't be matched to the sentences after one retry, the sample scores NaN rather than a ratio over a partial answer. In an ideal scenario, every statement in the reference is attributable to the retrieved context.
The formula for calculating context recall is as follows:
Example
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextRecall
# Setup LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-6-luna", client=client)
# Create metric
scorer = ContextRecall(llm=llm)
# Evaluate
result = await scorer.ascore(
user_input="Where is the Eiffel Tower located?",
retrieved_contexts=[
"The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France."
],
reference="The Eiffel Tower is located in Paris. It was completed in 1889.",
)
print(f"Context Recall Score: {result.value}")
Output:
The reference has two sentences. The retrieved context supports the first, that the tower is in Paris, but says nothing about 1889, so one of two sentences is recalled. A context like "Paris is the capital of France." supports neither sentence and scores 0.0: being about Paris is not enough.
Synchronous Usage
If you prefer synchronous code, you can use the .score() method instead of .ascore():
Non LLM Based Context Recall
NonLLMContextRecall metric is computed using retrieved_contexts and reference_contexts, and the values range between 0 and 1, with higher values indicating better performance. This metrics uses non-LLM string comparison metrics to identify if a retrieved context is relevant or not. You can use any non LLM based metrics as distance measure to identify if a retrieved context is relevant or not.
The formula for calculating context recall is as follows:
Example
from ragas.metrics.collections import NonLLMContextRecall
scorer = NonLLMContextRecall()
result = await scorer.ascore(
retrieved_contexts=["Paris is the capital of France."],
reference_contexts=[
"Paris is the capital of France.",
"The Eiffel Tower is one of the most famous landmarks in Paris.",
],
)
print(f"Score: {result.value}")
Output:
No LLM is involved, so scorer.score(...) gives the same result synchronously. distance_measure (default Levenshtein) and threshold (default 0.5) control when a retrieved context counts as matching a reference.
ID Based Context Recall
IDBasedContextRecall provides a direct and efficient way to measure recall by comparing the IDs of retrieved contexts with reference context IDs. This metric is particularly useful when you have a unique ID system for your documents and want to evaluate retrieval performance without comparing the actual content.
The metric computes recall using retrieved_context_ids and reference_context_ids, with values ranging between 0 and 1. Higher values indicate better performance. It works with both string and integer IDs.
The formula for calculating ID-based context recall is as follows:
Example
from ragas.metrics.collections import IDBasedContextRecall
scorer = IDBasedContextRecall()
result = await scorer.ascore(
retrieved_context_ids=["doc_1", "doc_2", "doc_3"],
reference_context_ids=["doc_1", "doc_4", "doc_5", "doc_6"],
)
print(f"Score: {result.value}")
Output:
Of the 4 reference IDs, only "doc_1" was retrieved, so recall is 1/4.