Context Precision
Outputs on this page
Outputs of LLM calls on this page come from earlier runs, some with models that have since been retired, and the code now names current models. Scores and generated text from your run will differ.
Context Precision is a metric that evaluates the retriever's ability to rank relevant chunks higher than irrelevant ones for a given query in the retrieved context. Specifically, it assesses the degree to which relevant chunks in the retrieved context are placed at the top of the ranking.
It is calculated as the mean of the precision@k for each chunk in the context. Precision@k is the ratio of the number of relevant chunks at rank k to the total number of chunks at rank k.
Where \(K\) is the total number of chunks in retrieved_contexts and \(v_k \in \{0, 1\}\) is the relevance indicator at rank \(k\).
Examples
Context Precision
The ContextPrecision metric evaluates whether retrieved contexts are useful for answering a question by comparing each context against a reference answer. Use this when you have a reference answer available. It is also available as ContextPrecisionWithReference.
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextPrecision
# Setup LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-6-luna", client=client)
# Create metric
scorer = ContextPrecision(llm=llm)
# Evaluate
result = await scorer.ascore(
user_input="Where is the Eiffel Tower located?",
reference="The Eiffel Tower is located in Paris.",
retrieved_contexts=[
"The Eiffel Tower is located in Paris.",
"The Brandenburg Gate is located in Berlin.",
],
)
print(f"Context Precision Score: {result.value}")
Output:
Synchronous Usage
If you prefer synchronous code, you can use the .score() method instead of .ascore():
Context Utilization
The ContextUtilization metric evaluates whether retrieved contexts are useful by comparing each context against the generated response. Use this when you don't have a reference answer but have the response that was generated. It is also available as ContextPrecisionWithoutReference.
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextUtilization
# Setup LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-6-luna", client=client)
# Create metric
scorer = ContextUtilization(llm=llm)
# Evaluate
result = await scorer.ascore(
user_input="Where is the Eiffel Tower located?",
response="The Eiffel Tower is located in Paris.",
retrieved_contexts=[
"The Eiffel Tower is located in Paris.",
"The Brandenburg Gate is located in Berlin.",
],
)
print(f"Context Utilization Score: {result.value}")
Output:
Note that even if an irrelevant chunk is present at the second position in the array, context precision remains the same. However, if this irrelevant chunk is placed at the first position, context precision reduces:
result = await scorer.ascore(
user_input="Where is the Eiffel Tower located?",
response="The Eiffel Tower is located in Paris.",
retrieved_contexts=[
"The Brandenburg Gate is located in Berlin.",
"The Eiffel Tower is located in Paris.",
],
)
print(f"Context Utilization Score: {result.value}")
Output:
Non LLM Based Context Precision
This metric uses non-LLM-based methods (such as Levenshtein distance measure) to determine whether a retrieved context is relevant.
Context Precision with reference contexts
The NonLLMContextPrecisionWithReference metric is designed for scenarios where both retrieved contexts and reference contexts are available for a user_input. To determine if a retrieved context is relevant, this method compares each retrieved context or chunk in retrieved_contexts with every context in reference_contexts using a non-LLM-based similarity measure.
Note that this metric would need the rapidfuzz package to be installed: pip install rapidfuzz.
Example
from ragas.metrics.collections import NonLLMContextPrecisionWithReference
scorer = NonLLMContextPrecisionWithReference()
result = await scorer.ascore(
retrieved_contexts=["The Eiffel Tower is located in Paris."],
reference_contexts=[
"Paris is the capital of France.",
"The Eiffel Tower is one of the most famous landmarks in Paris.",
],
)
print(f"Score: {result.value}")
Output:
No LLM is involved, so scorer.score(...) gives the same result synchronously. distance_measure (default Levenshtein) and threshold (default 0.5) control when a retrieved context counts as matching a reference.
ID Based Context Precision
IDBasedContextPrecision provides a direct and efficient way to measure precision by comparing the IDs of retrieved contexts with reference context IDs. This metric is particularly useful when you have a unique ID system for your documents and want to evaluate retrieval performance without comparing the actual content.
The metric computes precision using retrieved_context_ids and reference_context_ids, with values ranging between 0 and 1. Higher values indicate better performance. It works with both string and integer IDs.
The formula for calculating ID-based context precision is as follows:
Example
from ragas.metrics.collections import IDBasedContextPrecision
scorer = IDBasedContextPrecision()
result = await scorer.ascore(
retrieved_context_ids=["doc_1", "doc_2", "doc_3", "doc_4"],
reference_context_ids=["doc_1", "doc_4", "doc_5", "doc_6"],
)
print(f"Score: {result.value}")
Output:
In this example, out of the 4 retrieved context IDs, only 2 ("doc_1" and "doc_4") are found in the reference context IDs, resulting in a precision score of 0.5 or 50%.