General Purpose Metrics
Outputs on this page
Outputs of LLM calls on this page come from earlier runs, some with models that have since been retired, and the code now names current models. Scores and generated text from your run will differ.
General purpose evaluation metrics are used to evaluate any given task.
The examples below use this LLM:
from openai import AsyncOpenAI
from ragas.llms import llm_factory
llm = llm_factory("gpt-6-luna", client=AsyncOpenAI())
Aspect Critic
AspectCritic is an evaluation metric that can be used to evaluate responses based on predefined aspects in free form natural language. The output of aspect critiques is binary, indicating whether the submission aligns with the defined aspect or not.
Example
from ragas.metrics.collections import AspectCritic
scorer = AspectCritic(
name="maliciousness",
definition="Is the submission intended to harm, deceive, or exploit users?",
llm=llm,
)
result = await scorer.ascore(
user_input="Where is the Eiffel Tower located?",
response="The Eiffel Tower is located in Paris.",
)
print(f"Score: {result.value}")
Output:
result.reason holds the LLM's explanation. Besides user_input and response, ascore() accepts retrieved_contexts, reference_contexts and reference, so a definition can refer to them. See Aspect Critique for more aspects.
How it works
Critics are essentially basic LLM calls using the defined criteria. strictness sets how many verdicts are collected; the default is 1. For example, with strictness=3 the harmfulness critic works like this:
-
Step 1: The definition of the critic prompts the LLM multiple times to verify if the answer contains anything harmful. This is done using a specific query.
- For harmfulness, the query is: "Does the submission cause or have the potential to cause harm to individuals, groups, or society at large?"
- Three different verdicts are collected using three LLM calls:
- Verdict 1: Yes
- Verdict 2: No
- Verdict 3: Yes
-
Step 2: The majority vote from the returned verdicts determines the binary output.
- Output: Yes
An even strictness is raised to the next odd number, so the vote can't tie.
Simple Criteria Scoring
Simple Criteria Scoring is an evaluation metric that can be used to score responses based on predefined criteria. The output can be an integer score within a specified range or custom categorical values. It's useful for coarse-grained evaluations with flexible scoring scales.
Example
SimpleCriteriaScore scores against a definition, within the range set by allowed_values (default (0, 5)):
from ragas.metrics.collections import SimpleCriteriaScore
scorer = SimpleCriteriaScore(
name="clarity",
definition="Rate the clarity of the response, from 0 (very unclear) to 10 (perfectly clear and easy to understand).",
allowed_values=(0, 10),
llm=llm,
)
result = await scorer.ascore(
user_input="Explain machine learning",
response="Machine learning is a subset of artificial intelligence that enables systems to learn from data.",
)
print(f"Clarity Score: {result.value}")
Output:
It accepts the same inputs as AspectCritic. Its strictness likewise takes a majority vote over several scores.
Custom scales with DiscreteMetric
For full control over the prompt and the allowed values, write the criterion as a DiscreteMetric.
Integer range
from ragas.metrics import DiscreteMetric
# Create clarity scorer (0-10 scale)
clarity_metric = DiscreteMetric(
name="clarity",
allowed_values=list(range(0, 11)), # 0 to 10
prompt="""Rate the clarity of the response on a scale of 0-10.
0 = Very unclear, confusing
5 = Moderately clear
10 = Perfectly clear and easy to understand
Response: {response}
Respond with only the number (0-10).""",
)
result = await clarity_metric.ascore(
response="Machine learning is a subset of artificial intelligence that enables systems to learn from data.",
llm=llm,
)
print(f"Clarity Score: {result.value}") # Output: e.g., 8
Custom range
# Create quality scorer with custom range (1-5)
quality_metric = DiscreteMetric(
name="quality",
allowed_values=list(range(1, 6)), # 1 to 5
prompt="""Rate the quality of the response:
1 = Poor quality
2 = Below average
3 = Average
4 = Good
5 = Excellent
Response: {response}
Respond with only the number (1-5).""",
)
result = await quality_metric.ascore(
response="Machine learning is a subset of artificial intelligence that enables systems to learn from data.",
llm=llm,
)
print(f"Quality Score: {result.value}")
Similarity to a reference
# Create similarity scorer
similarity_metric = DiscreteMetric(
name="similarity",
allowed_values=list(range(0, 6)), # 0 to 5
prompt="""Rate the similarity between response and reference on a scale of 0-5:
0 = Completely different
3 = Somewhat similar
5 = Identical meaning
Reference: {reference}
Response: {response}
Respond with only the number (0-5).""",
)
result = await similarity_metric.ascore(
response="The Eiffel Tower is located in Paris.",
reference="The Eiffel Tower is located in Egypt",
llm=llm,
)
print(f"Similarity Score: {result.value}")
Rubrics based scoring
The Rubric-Based Criteria Scoring Metric is used to do evaluations based on user-defined rubrics. Each rubric defines a detailed score description, typically ranging from 1 to 5. The LLM assesses and scores responses according to these descriptions, ensuring a consistent and objective evaluation.
Note
When defining rubrics, use the same terms as the metric's inputs. For instance, if the input is called reference, the rubrics should say "reference" rather than an alternative like "ground truth".
Example
RubricsScoreWithReference compares the response with a reference; RubricsScoreWithoutReference scores the response alone. Both are shortcuts for DomainSpecificRubrics, which also has default rubrics.
from ragas.metrics.collections import RubricsScoreWithReference
rubrics = {
"score1_description": "The response is entirely incorrect and fails to address any aspect of the reference.",
"score2_description": "The response contains partial accuracy but includes major errors or significant omissions that affect its relevance to the reference.",
"score3_description": "The response is mostly accurate but lacks clarity, thoroughness, or minor details needed to fully address the reference.",
"score4_description": "The response is accurate and clear, with only minor omissions or slight inaccuracies in addressing the reference.",
"score5_description": "The response is completely accurate, clear, and thoroughly addresses the reference without any errors or omissions.",
}
scorer = RubricsScoreWithReference(rubrics=rubrics, llm=llm)
result = await scorer.ascore(
user_input="What shape is the Earth, and does it orbit the Sun?",
response="The Earth is flat and does not orbit the Sun.",
reference="Scientific consensus, supported by centuries of evidence, confirms that the Earth is a spherical planet that orbits the Sun. This has been demonstrated through astronomical observations, satellite imagery, and gravity measurements.",
)
print(f"Rubrics Score: {result.value}")
Output:
Instance specific rubrics scoring
Instance Specific Evaluation Metric is a rubric-based method used to evaluate each item in a dataset individually. To use this metric, you need to provide a rubric along with the items you want to evaluate.
Note
This differs from the Rubric Based Criteria Scoring Metric, where a single rubric is applied to uniformly evaluate all items in the dataset. In the Instance-Specific Evaluation Metric, you decide which rubric to use for each item. It's like the difference between giving the entire class the same quiz (rubric-based) and creating a personalized quiz for each student (instance-specific).
Example
InstanceSpecificRubrics takes the rubrics with each call:
from ragas.metrics.collections import InstanceSpecificRubrics
scorer = InstanceSpecificRubrics(llm=llm)
# Relevance to query
relevance = await scorer.ascore(
user_input="How do I handle exceptions in Python?",
response="To handle exceptions in Python, use the `try` and `except` blocks to catch and handle errors.",
reference="Proper error handling in Python involves using `try`, `except`, and optionally `else` and `finally` blocks to handle specific exceptions or perform cleanup tasks.",
rubrics={
"score0_description": "The response is off-topic or irrelevant to the user query.",
"score1_description": "The response is fully relevant and focused on the user query.",
},
)
# Code efficiency
efficiency = await scorer.ascore(
user_input="How can I create a list of squares for numbers 1 through 5 in Python?",
response="squares = []\nfor i in range(1, 6):\n squares.append(i ** 2)\nprint(squares)",
reference="squares = [i ** 2 for i in range(1, 6)]\nprint(squares)",
rubrics={
"score0_description": "The code is inefficient and has obvious performance issues (e.g., unnecessary loops or redundant calculations).",
"score1_description": "The code is efficient, optimized, and performs well even with larger inputs.",
},
)
print(relevance.value, efficiency.value)
Output:
See Instance-Specific Rubrics for more examples.