How to estimate Cost and Usage of evaluations and testset generation
Outputs on this page
Outputs of LLM calls on this page come from earlier runs, some with models that have since been retired, and the code now names current models. Scores and generated text from your run will differ.
When using LLMs for evaluation and test set generation, cost will be an important factor. Ragas provides you some tools to help you with that.
Implement TokenUsageParser
Ragas records token usage only when you ask for it, because providers report it in different shapes. To get usage data, collect it with a TokenUsageCollector built from a TokenUsageParser.
A TokenUsageParser is a function that takes the provider's raw completion and returns the TokenUsage that Ragas expects. Ragas ships parsers for OpenAI-shaped (get_token_usage_for_openai), Anthropic, Bedrock and Azure AI responses. LLMs created with llm_factory() call through LiteLLM, which returns OpenAI-shaped responses, so get_token_usage_for_openai works for them whatever the provider.
For example, here is the OpenAI parser applied to a completion:
from openai import OpenAI
# lets import a parser for OpenAI
from ragas.cost import get_token_usage_for_openai
client = OpenAI()
completion = client.chat.completions.create(
model="gpt-6-luna",
messages=[{"role": "user", "content": "hai there"}],
)
get_token_usage_for_openai(completion)
You can define your own or import parsers if they are defined. If you would like to suggest parser for LLM providers or contribute your own ones please open an issue 🙂.
Token Usage for Evaluations
Let's use the get_token_usage_for_openai parser to calculate the token usage for an evaluation.
from datasets import load_dataset # pip install datasets
from ragas import Dataset
from ragas.dataset_schema import SingleTurnSample
df = load_dataset("vibrantlabsai/amnesty_qa", "english_v3")["eval"].to_pandas()
df = df.astype(object).where(df.notna(), None) # NaN -> None
eval_dataset = Dataset.from_pandas(
df, "amnesty_qa", "inmemory", data_model=SingleTurnSample
)
Pass a TokenUsageCollector with the parser to metrics_experiment(usage=...). Every LLM call the run makes records its usage into it.
from ragas import metrics_experiment
from ragas.cost import TokenUsageCollector, get_token_usage_for_openai
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextRecall
evaluator_llm = llm_factory("gpt-6-luna", client=client)
usage = TokenUsageCollector(get_token_usage_for_openai)
run = metrics_experiment([ContextRecall(llm=evaluator_llm)], usage=usage)
exp = run.run(eval_dataset, name="context_recall_cost")
You can compute the cost of the run by passing the cost per token to usage.total_cost(). Use a new collector for each run you want to price separately.
In this example the model costs $5 per 1M input tokens and $15 per 1M output tokens. Use your own model's prices.
Output
Token Usage for Testset Generation
You can use the same parser for testset generation: pass token_usage_parser to generate(), generate_with_docs() or generate_with_chunks(). With the generate_with_* methods, the total includes building the knowledge graph (the transforms), which is usually most of the cost. generate() on an existing knowledge graph counts only generation, since the graph is already built. Only LLMs created with llm_factory() report usage.
For an example let's load an existing KnowledgeGraph and generate a testset. If you want to know more about how to generate a testset please check out the testset generation.
from ragas.testset.graph import KnowledgeGraph
# loading an existing KnowledgeGraph
# make sure to change the path to the location of the KnowledgeGraph file
kg = KnowledgeGraph.load("../../../experiments/scratchpad_kg.json")
kg
Output
Choose your LLM
Install the OpenAI SDK:
Set your API key:
Build the evaluator LLM with llm_factory:
Set your API key:
No Anthropic SDK is needed: the call goes through LiteLLM.
Embeddings default to LiteLLM, which resolves credentials from the environment:
For Vertex AI, use llm_factory("vertex_ai/gemini-3.6-flash") with your
Google Cloud credentials (Application Default Credentials, or
vertex_project= / vertex_location=). No Google SDK is needed.
Embeddings default to LiteLLM, which resolves credentials from the environment:
Install the OpenAI SDK, which ships the Azure client:
from openai import AzureOpenAI
from ragas.llms import llm_factory
client = AzureOpenAI(
api_version="2024-02-01",
azure_endpoint="https://<your-resource>.openai.azure.com",
)
generator_llm = llm_factory("gpt-6-luna", client=client)
gpt-6-luna here is your deployment name, which need not match the model name.
Embeddings default to LiteLLM, which resolves credentials from the environment:
Bedrock is reached through LiteLLM:
Credentials come from the usual AWS chain — environment, profile or instance role:
from ragas.llms import llm_factory
# the model ID from the Bedrock console, e.g. an Anthropic Claude model
generator_llm = llm_factory("bedrock/<bedrock-model-id>")
Embeddings default to LiteLLM, which resolves credentials from the environment:
LiteLLM reaches 100+ providers — Ollama, vLLM, Groq, Mistral, Cohere, Together, Vertex AI and more — behind one interface, so ragas is not tied to any single vendor's SDK.
from ragas.llms import llm_factory
# any model string LiteLLM understands; credentials come from the environment
generator_llm = llm_factory("ollama/llama3")
See the LiteLLM provider list for the model string to use. Credentials are read from the environment variables each provider expects.
Embeddings default to LiteLLM, which resolves credentials from the environment:
Migrating from LangchainLLMWrapper
ragas no longer depends on LangChain, and LangchainLLMWrapper has been
removed. Replace LangchainLLMWrapper(ChatOpenAI(model="gpt-6-luna")) with
llm_factory("gpt-6-luna", client=OpenAI()) as above, and
LangchainEmbeddingsWrapper(...) with embedding_factory(...). See the
Migrating off LangChain guide under How-to Guides > Migrations.
from ragas.llms import llm_factory
from ragas.testset import TestsetGenerator
tg = TestsetGenerator(llm=llm_factory(), knowledge_graph=kg)
# generating a testset
testset = tg.generate(testset_size=10, token_usage_parser=get_token_usage_for_openai)
# total cost for the generation process
testset.total_cost(cost_per_input_token=5 / 1e6, cost_per_output_token=15 / 1e6)
Output