Semantic Similarity
Semantic Similarity
The Semantic Similarity metric evaluates the semantic resemblance between a generated response and a reference (ground truth) answer. It ranges from 0 to 1, with higher scores indicating better alignment between the generated answer and the ground truth.
This metric uses embeddings and cosine similarity to measure how semantically similar two answers are, which can offer valuable insights into the quality of the generated response.
Example
from openai import AsyncOpenAI
from ragas.embeddings import OpenAIEmbeddings
from ragas.metrics.collections import SemanticSimilarity
# Setup embeddings
client = AsyncOpenAI()
embeddings = OpenAIEmbeddings(model="text-embedding-3-small", client=client)
# Create metric
scorer = SemanticSimilarity(embeddings=embeddings)
# Evaluate
result = await scorer.ascore(
reference="The Eiffel Tower is located in Paris. It has a height of 1000ft.",
response="The Eiffel Tower is located in Paris.",
)
print(f"Semantic Similarity Score: {result.value}")
Output:
Synchronous Usage
If you prefer synchronous code, you can use the .score() method instead of .ascore():
How It's Calculated
Example
Reference: Albert Einstein's theory of relativity revolutionized our understanding of the universe.
High similarity response: Einstein's groundbreaking theory of relativity transformed our comprehension of the cosmos.
Low similarity response: Isaac Newton's laws of motion greatly influenced classical physics.
Let's examine how semantic similarity was calculated for the high similarity response:
- Step 1: Vectorize the reference answer using the specified embedding model.
- Step 2: Vectorize the generated response using the same embedding model.
- Step 3: Compute the cosine similarity between the two vectors.
- Step 4: The cosine similarity is the final score. Cosine similarity can range from -1 to 1, and unrelated texts occasionally score slightly below 0 with some embedding models, so the score is clipped to 0-1. With
thresholdset, the score is instead1if the similarity is at or above the threshold and0otherwise.