Behavior changes from ragas
ragas-ng fixes bugs in legacy ragas even where code that depended on the old behavior has to change. This page lists those changes, apart from score changes, which are listed in Metric score changes. Each entry says what changed and what, if anything, you need to do.
Entries marked ⚠ silent don't raise an error, so read them first. Your code keeps running, but it may now do something different.
Running sync APIs: no more nest_asyncio (#80)
ragas no longer depends on or imports nest_asyncio. Every sync entry point
(evaluate(), metric.score(), batch_score(), @metric metrics, the
executor and testset transforms) runs on one shared background event loop. They
work inside Jupyter and async frameworks without patching the running loop.
Repeated calls also reuse one loop, so async clients such as AsyncOpenAI
stay bound to it. A second score() used to fail with "bound to a different
event loop".
| Before | Now | What to do |
|---|---|---|
evaluate(..., allow_nest_asyncio=False) |
accepted, ignored, DeprecationWarning |
remove the argument |
ragas.async_utils.apply_nest_asyncio() |
no-op returning False, DeprecationWarning |
remove the call |
score() inside a running loop raised "Cannot call sync score() from an async context" |
works | nothing |
One case to avoid: in a notebook, if you await an async client yourself
on the notebook's loop and then pass the same client to sync evaluate(), the
client is bound to the notebook's loop, not ragas'. Use await aevaluate(...)
in async code.
evaluate() leaves your objects alone (#76)
evaluate() works on copies of your metrics and never sets anything on your
LLM. The run_config and token counting apply only to calls made during that
run, including jobs from return_executor=True.
- ⚠ silent:
metric.llmandmetric.embeddingsstay as you left them (oftenNone) afterevaluate(llm=...). Code that readmetric.llmafter a run to find the LLM it used now getsNone. - ⚠ silent: without
run_config=,evaluate()leaves your LLM's ownset_run_config(...)in effect. It used to overwrite it with defaults. - Two concurrent
evaluate()calls on shared metrics or a shared LLM no longer mix up their LLMs, settings or token counts. TestsetGeneratorworks the same way, and no longer leaves its token counter attached to your LLM.
evaluate() inputs and results (#77, #78, #79)
| Input or call | Before | Now |
|---|---|---|
dataset=[SingleTurnSample(...), ...] or a list of dicts |
failed later | converted to an EvaluationDataset |
| an empty list, or a list mixing types | failed later | ValueError / TypeError before the run |
a raw provider client as llm= / embeddings= |
failed during the run | TypeError before the run, pointing to llm_factory() / embedding_factory() |
metrics=(m1, m2) (a tuple) |
rejected | accepted |
| a prompt that runs more than once per sample in traces | only the last run was kept | name, name (2), name (3), ... |
total_tokens() / total_cost() with no usage recorded |
IndexError |
zero, with a warning |
Testset generation: size, failures and personas (#81, #82, #84)
- ⚠ silent:
testset_sizeis an upper bound. Testsets used to overshoot it, for example 12 samples for 10, or 3 for 1. The size is now split exactly across the synthesizers. You can get fewer samples if the knowledge graph is too small, and a warning then gives the count and the reason. - With
raise_exceptions=False, a failed scenario or sample is left out, with a warning. It used to crash generation and lose all the completed work. - Persona names returned by the LLM match ignoring case and spacing. Names
that match no persona are skipped instead of raising
KeyError. - Personas generated on the first
generate()call are still saved on the generator aspersona_listand reused, but a list you supplied is no longer shuffled in place. New:testset.personasholds the personas used. -
Repeatable personas across runs. Persona generation runs at temperature 1.0 by design, so each new generator or script run gets a different set (see Prompt output handling). To get the same personas, and so the same mix of questions, every time, save them once and pass them back:
-import json from ragas.testset.persona import Persona # first run with open("personas.json", "w") as f: json.dump([p.model_dump() for p in testset.personas], f) # later runs with open("personas.json") as f: personas = [Persona(**p) for p in json.load(f)] generator = TestsetGenerator(llm=llm, embedding_model=emb, persona_list=personas)with_debugging_logs=Truenow turns on theragas.testset.*loggers. It had no effect before.
Testset generation: building the knowledge graph (#83, #85–#88)
- ⚠ silent: every document gets a chunk.
HeadlineSplitterturns a document it doesn't split into one CHUNK node. It used to add the document a second time under the same id, and that document never got a chunk to generate questions from. If you load a knowledge graph saved with an older version, its unsplit documents still have no chunks. default_transformshandles mixed-length corpora:- The long-document pipeline no longer crashes on documents under 500 tokens.
- The medium pipeline is used whenever any document is over 100 tokens.
- An empty list raises
ValueError. - If every document is 100 tokens or shorter, the error points to
generate_with_chunks(). CustomNodeFilterscores chunks you supplied yourself using their own summary. It used to skip them with a warning.apply_transformsstops at the first failure. It cancels the step's remaining tasks, doesn't run later steps, and raises the error. It used to run every task in the step first. There is no rollback, so retry on a freshKnowledgeGraph.
Testset generation: cost and metadata (#90, #91)
- ⚠ silent:
testset.total_tokens()/total_cost()from thegenerate_with_*methods now include building the knowledge graph, usually most of the cost, so the totals are higher. The transforms also get the run'sRunConfigtimeouts and retries. generate_with_llamaindex_docswithoutrun_config=keeps your LLM's own settings. It used to apply defaults.- New field
SingleTurnSample.reference_contexts_metadata: the source document's metadata for each reference context, kept as JSON-safe values. It is a new column into_pandas()output. If you write CSV from datasets where only some samples have the field,to_csv()can fail on the mixed columns.
Few-shot examples (#92, #93)
- ⚠ silent: the similarity threshold applies.
FewShotPydanticPrompt.threshold_for_examples, whichDemonstrationConfig.thresholdsets duringmetric.train(...), never reached the example store, so its default of 0.7 always applied. It does now. - ⚠ silent:
DynamicFewShotPromptshows no examples when none reachsimilarity_threshold. It used to fall back to the most recent examples. See Metric score changes. - Asking for 0 examples or history entries returns none. This applies to
top_k=0,max_similar_examples=0and DSPyinspect_history(0).[-0:]used to return them all. A negative count raisesValueError. metric.train(...)with annotations whose output fails validation no longer shifts the remaining examples. Before, each later input was paired with the previous annotation's output.- Custom example stores: the abstract
ExampleStore.get_examplesandSimpleExampleStore.get_examplesnow takethreshold. A store without that parameter still works, but the threshold is ignored and a warning says so once. Addthreshold: float = 0.7to your store to use it.
Prompt output handling (#94)
This applies to PydanticPrompt and ImageTextPrompt: the prompt classes
behind testset generation, the private legacy metric modules, and metrics with
customised prompts.
- Error type: parse failures raise
RagasOutputParserException. This covers aValidationErrororJSONDecodeError, and instructor's retries-exhausted errors with anllm_factory()LLM. Some of these used to escape as their raw types. The original is kept as__cause__, and the message includes it. If you caughtpydantic.ValidationErroraround a prompt call, catchRagasOutputParserExceptioninstead. Provider errors such as timeouts, rate limits and connection errors still pass through unchanged. - Every failure is reported to callbacks with
on_chain_error, so traces no longer leave a failed prompt run open. retries_leftis the number of fix attempts. When the LLM's output doesn't parse, ragas asks the LLM to fix it up toretries_lefttimes (default 3). Each fixed output is parsed like the original, including JSON inside code fences or surrounding text. Before, there was one fix attempt, and its result failed if it came back fenced.- Output that is already valid JSON is parsed as is. JSON with a fenced block inside one of its string values used to be cut down to that inner block.
- ⚠ silent:
llm_factory()LLMs run prompts concurrently. The prompt called the blockinggenerate()for an LLM on a sync client, so concurrent prompts, such as every testset transform, ran one at a time. It now awaitsagenerate(), which runs sync clients in a thread. Expect faster testset generation and more parallel requests to your provider. SetRunConfig(max_workers=...)if you need to limit them. - ⚠ silent:
nandtemperatureapply withllm_factory()LLMs. generate_multiple(n=3)makes three calls at a sampling temperature, 0.3 unless you pass one. It used to make one call and warn.- A
temperature=passed to a prompt is now used. It used to be ignored. - Testset personas vary between runs again. Persona generation has
asked for temperature 1.0 since it was added, so that personas come out
varied. That worked with the legacy LangChain-style LLM wrappers, which
honour the temperature. The code path for
llm_factory()LLMs, added later, dropped it, so those users got nearly the same personas every time without anyone choosing that. The original design applies again: a new generator or script run gets a different set of personas. To keep them fixed, see Repeatable personas across runs. - See Metric score changes.
@metric function arguments (#95)
This applies to metrics created with @discrete_metric, @numeric_metric,
@ranking_metric or create_metric_decorator().
- ⚠ silent: arguments arrive as the objects you passed. The decorator
validates inputs with pydantic and then called your function with
model_dump(), which turned pydantic-model arguments, such as aSingleTurnSampleor your ownBaseModel, into plain dicts. Your function now receives the objects themselves. Coercion still applies, so"2.5"becomes2.5for afloatparameter. If your function worked around this by indexing a dict (sample["user_input"]), switch to attribute access (sample.user_input). - ⚠ silent: concurrent calls no longer share inputs. The validated inputs
were stored on the metric object between validation and the call. Two calls
on one metric from different event loops, such as
aevaluate()on your loop alongsidescore(), could run with each other's inputs and return the wrong score. Each call now keeps its own inputs. - If the inputs can't be validated, for example because of a type hint
pydantic can't resolve, the function is called with your arguments and a
warning is issued once. Before, it either crashed
score()or called the function with no arguments, and the error was returned asMetricResult(value=None).
CSV files: every row's columns, JSON cells, typed loads (#100)
This applies to LocalCSVBackend ("local/csv") and
EvaluationDataset.to_csv().
- Rows may differ. The header is the union of every row's keys, in
first-seen order, and a missing value is an empty cell. Before, any row with
a key the first row lacked raised
ValueError: dict contains fields not in fieldnames. Because@experimentsaves its results at the end, that lost the whole run's results. - ⚠ silent: lists and dicts are JSON text. Cells used to hold Python
repr, such as['a', 'b']. They now hold JSON, such as["a", "b"]. If you parse these cells yourself withast.literal_eval, usejson.loads. - Typed loads restore types.
Dataset.load(..., data_model=Row),reload()andvalidate_with(Row)decode each cell using the field's type before validation, so lists, dicts, nested models, numbers, booleans, dates and empty optional fields load back. Before, any list, dict orOptional[...] = Nonefield failed validation. strfields are never changed. CSV can't tell""fromNone, so anOptional[str]saved asNoneloads as"". Without adata_model, CSV loads still return strings. Use"local/jsonl"to keep types without a model.Dataset.save()writes model rows withmodel_dump(mode="json"), so values such as UUIDs, enums and paths are saved as strings, and the model parses them back. Before, they could fail to save to JSONL.
Backends load what was stored (#101, #102)
Backends no longer guess types on load. Types come from the data_model you
load with.
- ⚠ silent: JSONL keeps strings as strings.
LocalJSONLBackend("local/jsonl") turned any string that looked like an ISO date, such as"2024-01-02", into adateordatetimeon load. That changed your text, and a model'sstrfield then rejected it. Now:- Loading with a
data_model:date/datetimefields parse their ISO strings andstrfields keep their text, so typed datasets round-trip exactly. - Loading without one: dates you saved come back as ISO strings, such as
"2024-01-15T10:30:00". Usedatetime.fromisoformat(...), or load with a model.
- Loading with a
- A malformed JSONL line raises
ValueErrornaming the line and file. It used to be skipped with aprint(), so rows disappeared silently. Blank lines are still skipped. - ⚠ silent: Google Drive cells stay text.
GDriveBackendturned numeric-looking cells into numbers on load, so"007"became7. Cells are now strings unless adata_modeltypes them, as with CSV. Also:Noneis saved as an empty cell instead of the text"None".- Lists and dicts are saved as JSON text.
- Loads read every column. Only columns A to Z were read before, so wider datasets silently lost columns.
- Dataset files are UTF-8.
to_csv(),to_jsonl(),from_jsonl(),MetricAnnotation.from_json()andSingleMetricAnnotation.from_json()read and write UTF-8 and close their files. They used the platform encoding, which breaks non-ASCII text on Windows.from_jsonl()ignores blank lines.
Message converters: AG-UI snapshots and OpenAI-format histories (#96, #98)
- ⚠ silent: AG-UI
MessagesSnapshotEventtool calls keep their name and arguments.convert_messages_snapshot()readname/argsfrom the wrong place: AG-UI nests them underfunction, with arguments as a JSON string. So every snapshot tool call came out asunknown_toolwith{}, and tool-call metrics such asToolCallAccuracyscored those calls as wrong. Scores for snapshot-based samples change. Arguments that aren't valid JSON are kept as{"raw_args": "..."}, as in the streaming path. - AG-UI message content:
Noneis"". It used to be the text"None". A list of content parts keeps its text parts, joined by newlines; image, audio, video, document and binary parts are skipped. It used to be the pydantic repr of the whole list. ragas.integrations.swarm.convert_to_ragas_messages()takes any OpenAI-format chat history, not only Swarm's:- An assistant message without
tool_calls(or withNone) no longer raisesKeyError. - Arguments that aren't a JSON object are kept as
raw_argsinstead of raising. - List-of-parts content keeps its text parts.
systemanddevelopermessages are skipped instead of raisingValueError.- An unknown or missing role still raises.
- An assistant message without
Langfuse helper removed (#97)
ragas.integrations.tracing.langfuse is removed, and importing it raises
ModuleNotFoundError. It never worked in ragas-ng: sync_trace() returned a
placeholder trace with no observations, and LangfuseTrace.filter() always
returned []. Langfuse's own SDK and OpenTelemetry cover both of its jobs
without vendor-specific code in ragas, so it is removed rather than ported to
the Langfuse SDK.
| Before | Now |
|---|---|
from ragas.integrations.tracing.langfuse import observe (or from ragas.integrations.tracing import observe) |
from langfuse import observe |
await sync_trace(trace_id) / LangfuseTrace(...) |
from langfuse import get_client; get_client().api.trace.get(trace_id) |
trace.filter("span_name") |
[o for o in trace.observations if o.name == "span_name"] on the Langfuse SDK's trace |
add_query_param(url, name, value) |
urllib.parse: urlencode / urlunparse |
pip install "ragas[tracing]" installed langfuse |
it installs only mlflow; pip install langfuse if you use it |
To see ragas runs in Langfuse, export ragas' OpenTelemetry spans to it. The
Langfuse SDK's default export filter drops spans from unknown
instrumentations, ragas among them, so allow the ragas scope with
should_export_span. The Langfuse guide shows
both set-ups and how to score your app's traces with ragas.
The MLflow helper, ragas.integrations.tracing.mlflow, is unchanged.
Adapted testset prompts write in the target language (#106)
adapt_prompts(language, ...) recorded the target language on each prompt
but never put it in the rendered prompt. An adapted prompt carried only a
translated instruction (with adapt_instruction=True) and translated
examples, so the generated queries followed the language of the documents
and personas. A French corpus adapted to "spanish" produced a mix of
French, English and Spanish queries.
- The single-hop and multi-hop query/answer generation prompts now tell the model to write their output in the adapted language. Testset queries and reference answers come out in that language, whatever the corpus's language is.
- This is opt-in per prompt, through the
states_output_languageclass attribute onPydanticPrompt. Other prompts don't get the line: persona matching, entity extraction and theme combination must copy values from their input exactly, and translating those values (for example, a persona named "curious student" coming back as "Estudiante curioso") drops every scenario. - Metric prompts are unchanged. Adapting a metric still translates its instruction and examples. It doesn't promise the language of the metric's output (reasons, extracted statements), and scores are unaffected.
- Prompts in English, and prompts never adapted, render exactly as before.
Experiment backends by name (#165)
@experiment(backend=...) and arun(..., backend=...) accepted a backend name
such as "local/jsonl", but created it with no constructor arguments, so it
raised TypeError for the missing root_dir. Only "inmemory" worked by
name. Both now take backend_kwargs:
@experiment(Result, backend="local/jsonl", backend_kwargs={"root_dir": "./data"})
async def run(row): ...
await run.arun(dataset, backend="local/csv", backend_kwargs={"root_dir": "./data"})
backend_kwargsgiven toarunreplaces the decorator's.backend_kwargswith a backend instance, or with no backend, raisesValueError. Before, there was no such parameter, so this breaks nothing.backend_kwargsis keyword-only and reserved, likenameandbackend. An experiment function with a parameter of that name can't receive it througharun.
Experiments: concurrency, threads, order and failures (#117)
ExperimentWrapper.arun (await my_experiment.arun(dataset)) changed in four ways:
- At most 16 rows run at once by default. Before, every row started at
once, so a large dataset sent all its LLM calls together and hit provider
rate limits. Set
@experiment(max_workers=...)orarun(..., max_workers=...).-1restores the old unlimited behavior. Runs with many fast rows may take longer than before. - Sync experiment functions run in worker threads, up to
max_workersat once. Before, they ran one after another on the event loop, blocking it. A function that isn't thread-safe needsmax_workers=1. Calling the decorated function directly (await my_experiment(row)) still runs it inline. - ⚠ silent: results are in dataset order. Before, they were stored in the order rows finished, so row i of the results was usually not row i of the dataset.
- Failed rows are logged. A row whose function raises is still left out
of the results. It's now a warning on the
ragas.experimentlogger, with the row index and error, instead of aprint(), and a final warning gives "N of M rows failed and were not saved".
max_workers is a keyword-only parameter of arun, like name, backend
and backend_kwargs, so an experiment function can't receive a parameter of
that name through arun.
Retired model IDs replaced (#176)
Docs, examples, ragas quickstart templates and docstrings named models that their providers have shut down, or will shut down soon:
- gpt-3.5-turbo, gpt-4, gpt-4.1-nano and gemini-2.0-flash, which fail today;
- gpt-5-mini and gpt-5-nano (2026-12-11), and claude-sonnet-4-5 (2026-11-30);
- gpt-4o and gpt-4o-mini (2027-01-14).
They now use each provider's listed replacement:
| Was | Now |
|---|---|
gpt-4o, gpt-4o-mini, gpt-4.1, gpt-4.1-mini |
gpt-6-luna |
gpt-3.5-turbo, gpt-5-mini |
gpt-5.6-terra |
gpt-4.1-nano, gpt-5-nano |
gpt-5.6-luna |
gpt-4, gpt-4-turbo, gpt-5, o1/o3 |
gpt-5.6-sol |
claude-sonnet-4-5, Claude 3.x Sonnet |
claude-sonnet-5-5 |
gemini-2.0-flash, gemini-1.5-flash |
gemini-3.6-flash |
gemini-1.5-pro, gemini-2.5-pro |
gemini-3.8-flash |
text-embedding-004 (Gemini) |
gemini-embedding-2 |
- The default judge model changed: see Score changes.
- The new OpenAI models are all reasoning models.
llm_factory()already maps parameters for them. Code that calls the OpenAI client directly can't sendtemperature(other than 1) ormax_tokens(usemax_completion_tokens). On Chat Completions,gpt-6-lunasupports tool calls only withreasoning_effort="none". Theworkflow_evalandagent_evalstemplates were updated accordingly. - Docs output: pages that show LLM output from earlier runs say so at the top. That output was produced with the old models.
- Your own code: if it names any of the old IDs, switch it as well. See your provider's deprecations page.
Claude: sampling parameters (#177)
llm_factory(..., provider="anthropic", client=Anthropic())no longer sends ragas' defaulttemperature/top_p. Every call with anthropic SDK 2.6 used to fail withTypeError: ... unexpected keyword argument 'temperature'.- A
temperatureortop_pyou pass (llm_factory(..., temperature=0.2)) is still sent. On current Claude models it fails, and the provider's error says why: onlytemperature=1is accepted on older API versions, and none on API2026-10-21. - A per-call temperature ragas chooses for sampling (
strictness, legacy prompts withn > 1) is not sent to Claude. Through LiteLLM it is sent withdrop_params=True, so LiteLLM drops it where a model rejects it. - The planned move of every call onto LiteLLM's parameter handling is #182.
Claude: JSON mode by default (#184)
Without a mode, llm_factory() now asks Claude for structured output as JSON:
- Mode.ANTHROPIC_JSON with an Anthropic client (provider="anthropic", client=Anthropic());
- Mode.JSON for Claude through LiteLLM ("anthropic/claude-...", including Bedrock's anthropic.claude-... IDs).
Instructor's defaults (ANTHROPIC_TOOLS / TOOLS) force a tool call. claude-sonnet-5-5 rejects that with 400 tool_choice: type "tool" and "any" are not supported for this model, so every structured call failed on both paths.
- Verified live with
claude-sonnet-5-5andclaude-haiku-5-5on both paths: generation,AspectCritic(with and withoutstrictness), andAnswerAccuracy. - A
modeyou pass is kept, somode=instructor.Mode.ANTHROPIC_TOOLSrestores the old behavior for models that accept it. - A LiteLLM
Routeror module passed asclientkeeps instructor's default, because the model isn't known when it's wrapped. Passmode=there.
NVIDIA collections metrics raise on LLM errors (#185)
AnswerAccuracy,ResponseGroundednessandContextRelevanceused to catch every exception from the judge call and turn it intonan. A misconfigured judge (wrong model, bad key, a provider rejecting a parameter) scorednanand the error was lost.- Provider and transport errors now propagate, as in other collections metrics. With
evaluate(raise_exceptions=False)the row is still missing, and the error is logged. - Only an invalid rating (outside the metric scale after
max_retriesattempts) still givesnan. DataCompyScoreis unchanged: itsexceptcovers CSV parsing, not an LLM call.
Model names in examples and hints (#181)
- Example templates name each model once, near the top of the module that uses it:
RAGAS_EXAMPLE_APP_MODELsets the model the app or agent under test calls;RAGAS_EXAMPLE_JUDGE_MODELsets the model that judges.
Unset, each template uses the same default as before.
- Error messages and deprecation hints that suggest a model use ragas.llms.base.DEFAULT_LLM_MODEL.
- Metric docstrings show llm_factory("<model>", client=client) instead of a specific model.
- Model retirements are tracked in scripts/model_registry.toml, and a unit test fails when a retired model is named in code, docs or examples. See scripts/README.md.
- text2sql/analyze_errors.py used gpt-5 (shutting down 2026-12-11) and now uses gpt-5.6-sol. A source comment and two doc lines still named gpt-5 or the shut-down o1; they were updated.
One path for every LLM: LiteLLM (#182)
llm_factory() returns an InstructorLLM, and every call goes through LiteLLM. ragas no longer wraps each provider's own SDK.
Breaking: native SDK clients of providers other than OpenAI are rejected with ValueError. That covers Anthropic(), genai.Client() and GenerativeModel, Mistral(), Groq(), Cohere(), boto3 Bedrock and xAI, and instructor-patched clients. The error names the model string to use:
| Before | Now |
|---|---|
llm_factory("claude-sonnet-5-5", provider="anthropic", client=Anthropic()) |
llm_factory("anthropic/claude-sonnet-5-5") |
llm_factory("gemini-3.6-flash", provider="google", client=genai.Client(...)) |
llm_factory("gemini/gemini-3.6-flash") |
| a google-genai Vertex client | llm_factory("vertex_ai/gemini-3.6-flash", vertex_project=..., vertex_location=...) |
| Mistral / Groq / Cohere clients | llm_factory("mistral/<m>") / "groq/<m>" / "cohere_chat/<m>" |
| a boto3 Bedrock client | llm_factory("bedrock/<model id>", aws_region_name=...) |
mode=instructor.Mode.ANTHROPIC_JSON / ANTHROPIC_TOOLS (and other native modes) |
Mode.JSON / Mode.TOOLS; native modes raise |
The provider's key comes from its environment variable (ANTHROPIC_API_KEY, GOOGLE_API_KEY or GEMINI_API_KEY, ...), or pass api_key=. To keep your own instructor client, construct InstructorLLM(client=..., model=..., provider=...) directly. Its per-provider parameter rules are unchanged.
Unchanged, but routed differently:
- OpenAI / AsyncOpenAI and AzureOpenAI / AsyncAzureOpenAI clients still work, through LiteLLM, as openai/<model> or azure/<deployment>.
- LiteLLM gets a copy of the client, because it writes max_retries and organization onto the object it receives. Your client is left untouched.
- evaluate() still builds its default embeddings from your client.
- A litellm.Router as client works as before.
Deprecated:
- llm_factory(adapter=...) is ignored, with a DeprecationWarning.
- ragas.llms.adapters (get_adapter, auto_detect_adapter, InstructorAdapter, LiteLLMAdapter) still imports, and both adapters call llm_factory().
- Constructing LiteLLMStructuredLLM warns. llm_factory() no longer returns it.
All three are removed in the next release.
Parameters (one set of rules for every llm_factory() LLM):
- ragas' defaults temperature=0.01 and top_p=0.1 are sent with LiteLLM's drop_params, so LiteLLM drops them for models that reject them.
- Values you set are sent as given.
- max_tokens is sent only if you set it. The default of 1024 used to truncate long and reasoning outputs on the client path.
- The OpenAI reasoning-model and Claude rules are unchanged (see LLMs (llm_factory)).
Token usage: responses are OpenAI-shaped for every provider, so use get_token_usage_for_openai.
Embeddings: the default embeddings for a Gemini LLM are now ragas' default (OpenAI's text-embedding-ada-002 through LiteLLM). They used to be Google's, when you passed a google-genai client. Pass GoogleEmbeddings(...) explicitly for a Gemini-only setup. GoogleEmbeddings still takes a google-genai client.
Tracing: metrics, LLM calls and experiments (#196)
Collections metrics, llm_factory() LLMs and @experiment take no callbacks, so they used to emit no OpenTelemetry spans and add nothing to the run tree. Now:
- Metrics: every metric's
score()/ascore()is anEVALUATORspan. That coversragas.metrics.collectionsmetrics andDiscreteMetric,NumericMetric,RankingMetricand decorator metrics. A metric called on its own is a root span, or a child of your application's active span. Underevaluate()it is the metric spanevaluate()opens, not a second one. - LLM calls: every call to an
llm_factory()LLM is anLLMspan named after its response model. It records the prompt, the validated response,llm.model_name, the messages, the invocation parameters and token counts. It nests under the metric, legacy prompt or experiment row that made it. - Prompts reach your tracing service, truncated by default. LLM spans carry the judge's prompt, which holds your evaluation data. Their messages, like every span's
input.value/output.value, are cut to 4,096 characters. SetRAGAS_TRACE_FULL_CONTENT=trueto send them in full. See Observability. - Experiments:
arun()is anexperiment <name>span, with onerow Nspan per row. A failed row is an error span. Metrics scored in the experiment function nest under their row. result.traceshas each LLM call's input and output again. Since metrics imported fromragas.metricsstarted handing off to collections,result.traces[i][metric]had been empty forllm_factory()LLMs. It now maps each call to{"input": {"prompt": ...}, "output": <response model>}, keyed by the response model's name, e.g.AspectCriticOutput.- A legacy prompt run without
callbacksnests under the run in progress instead of starting a new trace. - Span payloads render pydantic models as JSON fields instead of
repr().
Without an OpenTelemetry SDK, spans are still no-ops. The LLM call is unchanged unless a usage collector or a recording span needs the provider's raw response for token counts.
FaithfulnesswithHHEM loads a pinned model commit (#195)
The HHEM model is loaded with trust_remote_code=True, which runs the model repo's modeling_hhem_v2.py. Both FaithfulnesswithHHEM metrics (collections and legacy) used to load it from the repo's main, so a new push there changed the code ragas ran with no ragas change. They now load commit de9a5e6 ("Support transformers 5", 2026-10-06), which was reviewed. The collections metric takes revision= to load another. Scores are unchanged: on the model card's examples, this commit gives the card's published scores under both transformers 4.57 and 5.17.
The reported 0.0 for "The first superbowl was held on Jan 15, 1967" is HHEM working as designed. The example's context names only the "First AFL–NFL World Championship Game", and HHEM, a strict entailment model, does not credit what the context leaves to outside knowledge. The docs example now uses a context that states it.
Metric reasons: the verb agrees with the count (#200)
The first line of MetricResult.reason for Faithfulness, FaithfulnesswithHHEM, FactualCorrectness, NoiseSensitivity and ContextRecall always said "are" after the count, as in "1 of 1 statement in the response are entailed". The verb is now "is" when one item is counted or there is only one ("1 of 2 statements … is supported", "0 of 1 statement … is"), and "are" otherwise. Scores are unchanged. Code that matches on the exact reason text may need updating.
evaluate(): default embeddings follow a LiteLLM LLM's provider (#190)
When a metric needs embeddings and none are passed, evaluate() used to give every client-less llm_factory() LLM OpenAI's text-embedding-ada-002, so llm_factory("gemini/...") with AnswerRelevancy or AnswerCorrectness needed OPENAI_API_KEY. The embedding model now comes from the LLM's prefix:
gemini/...getsgemini/gemini-embedding-2, using the LLM'sapi_keyif it was given one.vertex_ai/...getsvertex_ai/gemini-embedding-2, with the LLM'svertex_project,vertex_locationandvertex_credentials.- Bare and
openai/names keeptext-embedding-ada-002. - Other providers (
ollama/,anthropic/,bedrock/, ...) also keep it, but now with aUserWarningthat says to passembeddings=.
⚠ Scores change for Gemini and Vertex users of embedding metrics who relied on the default, since a different embedding model gives different similarities. Pass embeddings= to keep the old model. LLMs built with an OpenAI or Azure client are unchanged: they still take embeddings from that client.
Experiments: raise_exceptions stops on the first failed row (#170)
@experiment(raise_exceptions=True) or arun(..., raise_exceptions=True) makes arun stop at the first row whose function raises, like evaluate(raise_exceptions=True). The rows still running are cancelled, rows not yet started never run, nothing is saved, and that row's exception is raised from arun, with a note naming the row index. The default, False, is unchanged: the row is logged and left out, and the run goes on.
Add save_partial=True (decorator or arun) to save the rows that finished before the stop instead; the error is still raised, and its note says how many rows were saved. Without raise_exceptions it has no effect.
raise_exceptions and save_partial are now arun's own keyword arguments, like name, backend, backend_kwargs and max_workers. An experiment function that took a keyword argument of either name through arun no longer receives it.
Experiments: a sync run() (#172)
@experiment functions now have run(dataset, ...), the synchronous form of await arun(dataset, ...). It takes the same arguments and returns the same Experiment, so a script no longer needs asyncio.run(fn.arun(...)). Like evaluate() and metric.score(), it runs on ragas' shared background event loop (#74, #80), so it also works inside a running loop such as Jupyter. Ctrl-C cancels the rows still running.
evaluate() / Executor: stopping stops the jobs (#207)
raise_exceptions=Trueno longer leaves jobs running. The first error used to be raised while the other jobs ran on in the background: forevaluate(), LLM calls still being made, and paid for, after the error. Now the jobs still running are cancelled and awaited before the error propagates, and no job starts after the failing one (withmax_workers=1, the rest never run).Executor.cancel()takes effect within about 50 ms. It used to wait for the next job to finish, then cancel only what hadn't finished yet. Jobs still running are cancelled;results()returns what had finished.- The cancel path no longer leaves a coroutine un-awaited (
RuntimeWarning: coroutine ... was never awaited).
ragas.async_utils.as_completed is unchanged apart from that warning; ragas itself now uses concurrent_results.
Requests and tokens per minute: RateLimit (#171)
New: llm_factory(..., rate_limit=RateLimit(rpm=..., tpm=...)) (from ragas.rate_limit) paces every request the LLM sends, waiting until the limit has room instead of hitting a 429. max_workers only limited jobs at once. Pass the same RateLimit to every LLM that shares an API key; LiteLLMEmbeddings(rate_limit=...) takes one too. Without rate_limit nothing changes. It adds one dependency, pyrate-limiter (pure Python, no dependencies of its own). See Requests and tokens per minute.
EvaluationResult.check(): thresholds for CI (#169)
New: result.check(min_mean=..., max_mean=..., allow_missing=0) raises ThresholdError (an AssertionError, from ragas.exceptions) unless each metric's mean is within its bounds and no row is unscored. The message is a per-metric table of mean, bounds, count and missing rows. A bound for a metric the result doesn't have raises ValueError. See Adding to your CI pipeline.
Copying LLMs, embeddings and metrics (#211)
copy.deepcopy() of an llm_factory() LLM raised RecursionError, and of LiteLLMEmbeddings, of a metric holding either, or of a string-distance metric (NonLLMStringSimilarity, NonLLMContextRecall, ...) raised TypeError: cannot pickle 'module' object. They now copy. A copy has its own settings (model_args, system_prompt, run config, ...) and shares the provider clients, the cache and the RateLimit: a copied LLM goes to the same provider, under the same rate limit.
A copy (deep or shallow) of an LLM or embeddings with a cache used to keep the original's cached generate / embed_text, so its calls ran on the original object, with the original's settings. They now run on the copy.
distance_measure_map on the string-distance metrics and OverlapScoreBuilder is now a read-only property rather than an instance attribute.
Prompt optimizers rank losses by their direction (#216)
Loss now declares its direction with a higher_is_better class attribute: False for MSELoss (an error), True for BinaryMetricLoss (accuracy or F1). Loss.fitness(predicted, actual) returns a value where higher is always better, and both optimizers rank by it.
- ⚠ silent:
GeneticOptimizerkept the prompt with the highest MSE, and passed(actual, predicted)to the loss. It now keeps the best prompt and passes(predicted, actual). Cross-over paired every candidate with the first one, including the first with itself; each candidate now crosses with its nearest other candidate. Optimized prompts will differ. - ⚠ silent: the DSPy metric negated every loss, so under
BinaryMetricLossDSPyOptimizerminimised accuracy. It now maximises it. UnderMSELossit still returns the negated error. GeneticOptimizer.evaluate_fitness()now returns fitness values (higher is better), not raw losses.MSELoss(reduction="sum"),BinaryMetricLoss(metric="f1_score")andInstructionConfig(loss=MSELoss())raisedTypeErrorand now work. An unknown option raisesValueError.- A custom
Lossthat is higher-is-better must sethigher_is_better = True; the default is lower-is-better. - Each
InstructionConfiggets its own copy ofoptimizer_config, so editing one no longer changes the others.
Collections metric constructors: unknown arguments raise (#216)
- Any
ragas.metrics.collectionsmetric now raisesTypeErrorfor a keyword argument it doesn't take. They used to be dropped, soAspectCritic(..., strictnes=3)silently built a metric withstrictness=1. Fix the spelling or remove the argument. ContextPrecision(llm, name="x")andContextUtilization(llm, name="x")raisedTypeError: multiple values for keyword argument 'name'. They now acceptname; the defaults are unchanged.- A custom
BaseMetricsubclass that doesn't defineascorecan no longer be instantiated (TypeError). It used to score0.0from a placeholder. AnswerCorrectness(embeddings=...)now checks that the embeddings are a modernBaseRagasEmbedding, as every other collections metric does; anything else raisesValueError.embeddings=Nonewithweights=[1.0, 0.0]still works.
Testset generation: scenarios, filters and personas (#216)
SingleHopScenariois now defined once. The class exported fromragas.testset.synthesizers.single_hopwas a different class from the one the synthesizer checks for, so passing a scenario you built yourself raisedTypeError: scenario type should be SingleHopScenario.- ⚠ silent:
JaccardSimilarityBuildernow honoursfilter_nodesin a transform pipeline. It compared every node in the graph, so withfilter_nodesset you now get fewer relationships, limited to the nodes you selected.SummaryCosineSimilarityBuildernow links only document nodes in a pipeline, as documented; it linked every node with a summary embedding, chunks included. Neither is in the default transforms. OverlapScoreBuilder().nameis now"OverlapScoreBuilder". It was an empty string.- ⚠ silent:
NodeFilter.transform()now removes every node its filter flags, as the pipeline does. It removed only the first. - ⚠ silent:
generate_personas_from_kg()now groups similar summaries by true cosine similarity, comparing each summary with its own node's embedding. It used an unnormalised dot product, and when some nodes had no summary it could compare a summary with another node's embedding. You may get different personas from the same graph. A node with a summary but nosummary_embeddingnow raises a clearValueError.
One deprecation warning, Vertex clients used, LiteLLM batch sizes per provider (#216)
evaluate()now emits oneDeprecationWarning, pointing at your call. It added a second one fromaevaluate()that pointed inside ragas.aevaluate()called directly still warns once.GoogleEmbeddings(use_vertex=True, client=...)now uses the client you pass. ATextEmbeddingModelis called directly, and a google-genaigenai.Client(vertexai=True, ...)goes throughmodels.embed_content. Any other client loadsmodelonce and reuses it;TextEmbeddingModel.from_pretrainedused to run on every call, ignoring the client.- ⚠ silent:
LiteLLMEmbeddingswithoutbatch_size=now picks its batch size from the model's provider: 100 for OpenAI, 96 for Cohere, 5 for Vertex AI, 32 for Hugging Face, and 10 otherwise. Every model got 10. OpenAI and Cohere users see fewer, larger embedding requests. Passbatch_size=to keep the old behaviour.
The LLM takes chat messages, and the multimodal metrics use them (#214)
generate() and agenerate() on llm_factory() LLMs now accept a list of OpenAI-format chat messages as well as a prompt string; message content can be a list of text and image_url parts. A messages list takes the same path as a string: retries, cache, rate limit, tracing, token usage, provider parameter mapping, and sync or async clients. The LLM's system_prompt is prepended unless the list already has a system message. String prompts are sent and cached exactly as before, so existing caches still hit.
MultiModalFaithfulness and MultiModalRelevance (collections) called the provider client directly, so they skipped retries, the cache, tracing, usage recording and system_prompt, and failed with a sync client such as OpenAI(). They now go through llm.agenerate(), so all of these apply and a sync client works. Scores are unchanged, except that a configured system_prompt is now sent and can change the verdict.
One prompt class (#214)
ragas.prompt.metrics.BasePrompt, the prompt class collections metrics use, is now ragas' one structured-prompt implementation, and ragas.prompt.PydanticPrompt is a subclass that only adds support for legacy BaseRagasLLM wrappers. Rendered prompts are byte-for-byte unchanged for both, so cache keys and scores don't move.
BasePromptgains whatPydanticPrompthad:generate()/generate_multiple()through anllm_factory()LLM (withnconcurrent samples,process_input/process_outputhooks and a traced prompt run),name, content-based==andhash(), andsave()/load().- ⚠ silent:
PydanticPrompt.adapt()with anllm_factory()LLM now translates with the same prompt collections metrics use, so translations of adapted testset prompts can differ. With a legacy LLM it still usesTranslateStatements. BasePrompt.adapt(..., adapt_instruction=True)now translates the instruction even when the prompt has no example strings. It returned without translating it.save()logs the saved path at INFO instead of printing it.
evaluate() runs modern metrics natively (#214)
evaluate() and aevaluate() no longer wrap ragas.metrics.collections metrics as legacy metrics. They call the metric's ascore() with the sample's fields by name. Scores, reasons, traces and token usage are the same as before.
Metrics that take the LLM per call now work in evaluate(): NumericMetric (and other prompt-template metrics) and @numeric_metric-decorated functions. They get the llm passed to evaluate(), or the default LLM if none was passed; every row used to score NaN. Passing a legacy BaseRagasLLM to such a metric now raises TypeError before the run starts, and a decorated function whose required parameters are not sample fields raises ValueError up front instead of scoring NaN on every row.
⚠ silent: evaluate(metrics=None) now runs the collections AnswerRelevancy, ContextPrecision, Faithfulness and ContextRecall, built with the run's LLM and embeddings, instead of the ragas.metrics hand-off classes. Scores should not change, since the hand-off already scored through collections. With a legacy LLM or legacy embeddings you still get the old defaults.
Prompt optimization moves to collections metrics (#214)
metric.train(), GeneticOptimizer and DSPyOptimizer now tune the metrics in ragas.metrics.collections. The legacy ragas.metrics classes are being removed (#215), and their train() is removed with this change. InstructionConfig and DemonstrationConfig now take only llm_factory() LLMs and embedding_factory() embeddings, and raise a ValidationError for legacy wrappers.
- Collections metrics gain
get_prompts()andset_prompts(**prompts). Prompts are keyed by attribute name (promptforAspectCritic;statement_generator_promptandnli_statement_promptforFaithfulness).get_prompts()returns copies, so to change a prompt you edit the copy and pass it toset_prompts().set_prompts()copies what it is given, rejects unknown names withValueError, and rejects a non-BasePrompt, or a prompt with different input or output models, withTypeError. - Annotation files key each sample's
promptsby those names. Legacy names such assingle_turn_aspect_critic_promptraise aValueErrorthat lists the metric's names. The sample file for the Vertex AI alignment guide uses the new names. metric.train(path, demonstration_config, instruction_config, *, dataset=None, run_config=None, batch_size=None, raise_exceptions=True)also accepts the annotations as an object throughdataset=. It no longer takescallbacksorwith_debugging_logs. A config withenabled=Falseis now skipped; it used to run anyway.- ⚠ silent:
train()no longer modifies the metric's prompt objects in place. Optimized prompts replace them throughset_prompts(), and candidates are scored on a copy of the metric. In-place edits used to reach every metric sharing a prompt object. Optimizer.optimize()now returns optimized copies of the prompts ({name: BasePrompt}) forset_prompts(), rather than{name: instruction}.DSPyOptimizer's copies also carry the demonstrations MIPROv2 chose, which it used to drop.- Each collections metric has an
output_type, which picks the default loss: binary forAspectCritic,AgentGoalAccuracyWithReference/WithoutReferenceandSQLSemanticEquivalence; discrete for the rubric metrics andSimpleCriteriaScore; continuous otherwise. - Few-shot demonstrations (
DemonstrationConfig) make each annotated prompt a copy of itself that, per input, renders thetop_kannotated examples at leastthresholdsimilar to it.technique="random"now works: a fixed random sample oftop_kexamples, with no embedding calls. It was ignored. DSPyOptimizernow drives DSPy through adspy.LMbuilt from the LLM's LiteLLM model name and credentials, rather than wrapping a legacy LLM. PassDSPyOptimizer(lm=dspy.LM(...))for an LLM over alitellm.Routeror an Azure client, which raiseValueError. DSPy's own cache and retries apply to its calls, andllm_factory(rate_limit=...)doesn't.DSPyOptimizerfailed with every current DSPy: MIPROv2 takes its metric in the constructor and rejectsnum_candidateswith anautolevel. It now passes them that way, runsnum_trials(fromoptimizer_config, defaultnum_candidates) whenauto=None, and doesn't stop to ask for confirmation. DSPy signatures now carry the prompt fields' types, and only prompts with annotations are optimized.- ⚠ silent: DSPy's metric compared an output field named after the metric, which no prompt has, so every prediction scored 0. It now compares that field if present, else the prompt's first numeric output field, such as
verdict. GeneticOptimizer's feedback mutation reads each prompt's input and output from the traced LLM calls of the metric. A custom LLM that doesn't trace its calls gets no feedback, with a warning, and keeps the candidate's instruction.
Testset generation takes modern LLMs, embeddings and prompts only (#214)
Testset generation (TestsetGenerator, the synthesizers, the transforms, default_transforms() and generate_personas_from_kg()) now takes only an llm_factory() LLM and a modern embedding model (BaseRagasEmbedding, such as embedding_factory() output). A legacy BaseRagasLLM or BaseRagasEmbeddings wrapper raises TypeError when you build the object or call the function, with a message that names what to use instead.
TestsetGenerator.from_llama_index()andgenerate_with_llamaindex_docs()are removed. Build the generator withllm_factory()andembedding_factory(), and pass LlamaIndex documents togenerate_with_docs()withdocument_adapter=lambda d: Document(page_content=d.text, metadata=d.metadata). See LlamaIndex.- New:
generate_with_docs()andgenerate_with_chunks()takedocument_adapter=, a function that turns each of your documents into one withpage_contentandmetadata, so documents from any framework work without ragas depending on it.generate_with_docs()also accepts plain strings now, asgenerate_with_chunks()did.generate_with_langchain_docs()stays as a deprecated alias ofgenerate_with_docs(). - ⚠ silent:
generate_with_docs()now skips documents with no text, asgenerate_with_chunks()and the removed LlamaIndex method did. Blank documents used to become empty graph nodes and counted toward the document-length shares that pick the default transforms, so a corpus with blank documents can now get a different pipeline. - The testset prompts (extractors, filters, synthesizers, personas) are now
ragas.prompt.metrics.BasePromptsubclasses, notPydanticPrompt. Their rendered text is byte-for-byte unchanged, so cache keys don't move. Theirgenerate()no longer takesstop=orretries_left=.get_prompts()andset_prompts()on testset classes find and take anyBasePrompt, including aPydanticPrompt. EmbeddingExtractoralways callsaembed_text(). With an embedding model that had nois_asyncattribute, such asLiteLLMEmbeddings, it ran the syncembed_text()on a worker thread and warned (UserWarning); it now makes the async call and doesn't warn. The embeddings are the same.
The dspy extra installs DSPy 3 with optuna
pip install "ragas[dspy]" now installs dspy[optuna]>=3 instead of dspy-ai>=2.4.0. DSPyOptimizer runs MIPROv2, which on DSPy 3 needs optuna; the old extra left it out, so the optimizer failed on import of a fresh install. DSPy 2.x is no longer supported: DSPyOptimizer already needed DSPy 3's MIPROv2 constructor (#214).
Dead code removed: ragas evals, as_completed, find_indirect_clusters (#215)
These had no callers in ragas, and the CLI command could not run.
- The
ragas evalsCLI command is removed. It called arun_asyncmethod that experiments don't have and aProjectclass that no longer exists, so every run failed. Run an experiment from Python instead:Dataset.load(...), thenmy_experiment.run(dataset)orawait my_experiment.arun(dataset).ragas quickstartandragas hello-worldare unchanged, except that thehello_world/evals.pyscript thatragas hello-worldwrites is now a working@experimentyou run withpython hello_world/evals.py. It used to be meant forragas evalsand could not run either. ragas.async_utils.as_completed()andragas.async_utils.process_futures()are removed. Useconcurrent_results(), which ragas has used since #207 and which cancels what is still running when you stop iterating, orasyncio.as_completed().KnowledgeGraph.find_indirect_clusters()is removed. Testset generation usesfind_n_indirect_clusters(), which stays. With it goes ragas' last use ofscikit-network(its Leiden clustering) and ofnetworkx.- The module
ragas.backends.utilsis removed. ItsMemorableNameswas a copy ofragas.utils.MemorableNames; import it from there.
The legacy layer is removed (#215)
The legacy metrics, LLM and embedding wrappers and prompt classes are deleted, not deprecated. Everything they did has a modern replacement, which #214 moved testset generation, the optimizers and evaluate() onto.
| Removed | Use instead |
|---|---|
The legacy metrics: from ragas.metrics import Faithfulness, faithfulness, RubricsScore, …, and the private modules ragas.metrics._* |
from ragas.metrics.collections import Faithfulness; Faithfulness(llm=llm). Collections metrics take their models at construction and score with await metric.ascore(user_input=..., response=...). |
| Renamed metrics | LLMContextRecall → ContextRecall, ResponseRelevancy → AnswerRelevancy, RubricsScore → DomainSpecificRubrics, AnswerSimilarity → SemanticSimilarity, ChrfScore → CHRFScore, InstanceRubrics → InstanceSpecificRubrics, LLMContextPrecisionWith[out]Reference → ContextPrecisionWith[out]Reference, LLMSQLEquivalence → SQLSemanticEquivalence, SummarizationScore → SummaryScore, TopicAdherenceScore → TopicAdherence |
ragas.metrics.quoted_spans |
ragas.metrics.collections.QuotedSpansAlignment |
Metric, MetricWithLLM, MetricWithEmbeddings, SingleTurnMetric, MultiTurnMetric, ModeMetric, ensembler (ragas.metrics.base) |
Subclass ragas.metrics.collections.BaseMetric, or use DiscreteMetric / NumericMetric / RankingMetric |
BaseRagasLLM, LlamaIndexLLMWrapper, HaystackLLMWrapper, OCIGenAIWrapper / oci_genai_factory, LiteLLMStructuredLLM, the ragas.llms.adapters package, ragas.llms.output (LLMResult, Generation) |
llm_factory(), which reaches providers through LiteLLM model prefixes (e.g. llm_factory("anthropic/...")) |
BaseRagasEmbeddings, HuggingfaceEmbeddings, LlamaIndexEmbeddingsWrapper, HaystackEmbeddingsWrapper |
BaseRagasEmbedding and its providers: embedding_factory(), HuggingFaceEmbeddings, LiteLLMEmbeddings, OpenAIEmbeddings, GoogleEmbeddings |
ragas.run_config.add_retry / add_async_retry, which retried every exception, a 400 included |
llm_factory() LLMs retry transient errors per RunConfig themselves |
PydanticPrompt, StringPrompt, the legacy ragas.prompt.base.BasePrompt, FewShotPydanticPrompt, ExampleStore / InMemoryExampleStore, ImageTextPrompt / ImageTextPromptValue, RagasOutputParser, FixOutputFormat, TranslateStatements, ragas.prompt.value, ragas.prompt.utils.extract_json |
ragas.prompt.BasePrompt, which is now ragas.prompt.metrics.BasePrompt (subclass it the way you subclassed PydanticPrompt); DynamicFewShotPrompt for few-shot examples; messages-based llm.agenerate for images |
ragas.validation.validate_required_columns / validate_supported_metrics, ragas.utils.get_required_columns_v1, get_metric_language, DeprecationHelper |
Nothing: they only served the legacy metrics |
The ai-frameworks (Haystack) and oci extras |
llm_factory() over LiteLLM |
from ragas.metrics import <old name>raisesImportErrorwith the replacement import, e.g.'RubricsScore' was removed from 'ragas.metrics' along with the legacy metrics. Use: from ragas.metrics.collections import DomainSpecificRubrics. It is a hint, not a shim: nothing is re-exported.DiscreteMetric,NumericMetric,RankingMetric, the*_metricdecorators,MetricResult,BaseMetric,LLMMetric,MetricTypeandMetricOutputTypestay inragas.metrics.evaluate()andaevaluate()take onlySimpleBaseMetricmetrics (collections metrics and theDiscreteMetricfamily), anllm_factory()LLM and aBaseRagasEmbedding; anything else raisesTypeErrorbefore the run starts.evaluate(metrics=None)always runs the collections defaults.ragas.integrations.llama_index.evaluate()no longer wraps LlamaIndex LLMs and embeddings. Itsllmandembeddingsare now ragas models (llm_factory(),embedding_factory()), and itsmetricsare collections metrics.PromptMixin.get_prompts()/set_prompts()(testset synthesizers and transforms) find and take anyragas.prompt.BasePrompt; the testset-only override is folded into it.ragas.metrics._collections_bridgeis gone; the sample-to-ascore()mappingevaluate()uses isragas.metrics.sample_inputs.DistanceMeasureforOverlapScoreBuildercomes fromragas.metrics.collections; the values are unchanged.
Multimodal metrics restrict where context images are loaded from (#229)
MultiModalFaithfulness and MultiModalRelevance load the images named in retrieved_contexts. They used to open any local image path and fetch any http(s) URL, following redirects to any host, internal ones included. Upstream's checks for this (vibrantlabsai/ragas#1991) were only in the legacy ImageTextPrompt. They now apply to both metrics, configured through the new ragas.metrics.collections.ImageSourcePolicy, passed as image_policy=:
- Local image paths are no longer read by default; such a context raises
ImageLoadError(#231). To read local images, passimage_policy=ImageSourcePolicy(local_dir="images/"). Paths are resolved against that directory, and a path that resolves outside it is refused, symlinks included. - A URL whose host resolves to a non-public address (private, loopback, link-local, shared, reserved or multicast, IPv4 or IPv6) is not fetched, and neither is a redirect to one; it raises
ImageLoadError(#231). For an internal image server, passImageSourcePolicy(allow_private_urls=True). - URLs with credentials, a backslash in the host part, or a host that
urllibandrequestsparse differently are refused. At most 5 redirects are followed. - The size limit (10 MB) is now
ImageSourcePolicy(max_bytes=...). - Data URIs are unchanged.
- The image helpers
process_image_to_base64()andis_image_path_or_url()moved from themulti_modal_faithfulness.utilandmulti_modal_relevance.utilmodules toragas.metrics.collections.image_sources.process_image_to_base64()takes the policy as its second argument.
Multimodal metrics raise when an image context can't be loaded (#231)
MultiModalFaithfulness and MultiModalRelevance used to send an image reference they could not load to the judge as text, such as Context 1: path/to/tesla.jpg, and still return a score. That covered missing files, failed downloads, non-image content, and everything #229 refuses. The judge never saw the image, so the score looked valid but was not.
- An image reference that is refused or fails to load now raises
ragas.exceptions.ImageLoadError, aValueError, before the LLM is called. The message gives the context's position, the reason, and the setting that would allow it. Underevaluate()this followsraise_exceptions: it is raised, or the score is NaN and the error is logged. ImageSourcePolicy(on_blocked="warn")restores the old fallback: a warning is logged and the reference is sent as text.- An image reference is an image data URI, an http(s) URL, or a path ending in an image extension (
.jpg,.jpeg,.png,.gif,.webp,.bmp). Any other context is text, as before. - ⚠ silent: under
local_dir, a file without an image extension is no longer opened. It is sent as text, even if it holds an image. - Other URL schemes, such as
ftp://, raise instead of being treated as file paths.
Custom backends implement _load, _save and _list (#217)
BaseBackend used to have six abstract methods (load_dataset, load_experiment, save_dataset, save_experiment, list_datasets, list_experiments), and every backend wrote the same six one-line wrappers. Now the six are concrete, and a backend implements three abstract methods that take the store, kind ("datasets" or "experiments"):
_load(kind, name),_save(kind, name, data, data_model)and_list(kind). A custom backend that only defines the six public methods can no longer be instantiated (TypeError: Can't instantiate abstract class); rename them to these three.- The new
ragas.backends.FileBackend(root_dir)is the base for one-file-per-name backends: setsuffixand implement_read(path)and_write(path, data).LocalCSVBackendandLocalJSONLBackendare built on it. - The private helpers
_get_data_dir(),_get_file_path()(nowFileBackend._path(kind, name)) andLocalJSONLBackend._serialize_datetime()are gone, as areGDriveBackend's_load_data_from_spreadsheet(),_save_data_to_spreadsheet()and_list_data_names(). - The registry names, on-disk formats and error messages are unchanged. See Storage backends.
run_async_tasks() returns results in order and runs on Executor (#217)
ragas.async_utils.run_async_tasks() had its own copy of Executor's batching, progress-bar and cancellation loop. It is now a thin wrapper over Executor, and @experiment's arun() runs its rows through concurrent_results() instead of its own semaphore-and-as_completed loop.
run_async_tasks()returns results in the order oftasks; they used to come back in completion order. Errors are unchanged: withfail_fast=Truethe first failure cancels the rest and is raised; without it every task runs and the first error to happen is raised at the end.- Its unused keyword argument
cancel_checkis removed. To stop a run early, useExecutorand callexecutor.cancel(). arun()'smax_workers,raise_exceptions,save_partial, ordering and cancellation behave as before.
EvaluationDataset: one rule for sample types, from_dict takes columns (#217)
Which sample class a row becomes was decided in three places. It is now ragas.dataset_schema.sample_type_for(row): a row whose user_input is a list (of messages) is a MultiTurnSample, any other row a SingleTurnSample. SAMPLE_TYPES lists the two classes.
EvaluationDataset.from_list()decides per row. It used to look only at the first row'suser_inputand build every row as that type. A list mixing single- and multi-turn rows now raisesValueError: Sample at index N is of type ...; it used to raise a pydantic validation error on the first row that didn't fit.EvaluationDataset.from_dict()takes columns,{"user_input": [...], "response": [...]}, as itsDictannotation said, and callsfrom_list(). It used to iterate its argument as a list of rows, so only a list worked and a real dict raisedTypeError. Pass a list of rows tofrom_list();from_dict()raisesTypeErrorfor one.EvaluationDataset.backendand.name, andfrom_list()'sbackend=andname=arguments, are removed. Nothing read them. For a named dataset saved to a backend, useragas.Dataset.ragas.validation.get_supported_metric_type()is removed.evaluate()reports an unsupported sample class asThe metric '<name>' does not support the sample type <class>, instead ofUnsupported sample type <class>.
Names moved out of ragas.utils (#217)
ragas.utils is split up. Nothing is re-exported from the old location:
| Was | Now |
|---|---|
ragas.utils.MemorableNames, memorable_names, create_nano_id |
ragas.names |
ragas.utils.async_to_sync |
ragas.async_utils.async_to_sync |
ragas.utils.convert_v1_to_v2_dataset, REQUIRED_COLS_v1 |
ragas.validation |
ragas.utils.convert_row_v1_to_v2, convert_v2_to_v1_dataset |
Removed: nothing used them |
Dataset.train_test_split() no longer shuffles the dataset or the global random (#217)
With random_state set, train_test_split() called random.seed(random_state) and shuffled the dataset's own rows in place. Now it shuffles a copy with random.Random(random_state).
- For a given seed and dataset, the same rows land in each split as before.
- The dataset keeps its row order, and the global
randomstate is not reseeded. Code that relied ontrain_test_split()seedingrandomfor later calls must seed it itself. - Calling it twice with the same seed now gives the same split. It used to shuffle the already-shuffled rows again, so the second call gave a different split.
- With
random_state=Nonethe rows are not shuffled, as before: the lasttest_sizeof them are the test split.
ragas.llms.base is split into modules (#217)
llm_factory(), InstructorLLM, InstructorBaseRagasLLM, InstructorModelArgs, DEFAULT_LLM_MODEL and default_llm import from ragas.llms and ragas.llms.base as before. Parameter mapping, cache keys and scores are unchanged.
is_reasoning_modelmoved toragas.llms.provider_rulesand is exported fromragas.llms: usefrom ragas.llms import is_reasoning_model.from ragas.llms.base import is_reasoning_modelraisesImportError.- The per-provider parameter rules are
ragas.llms.provider_rules.PROVIDER_RULES, a list of rule objects where the first match wins. TheInstructorLLM._map_google_params,_map_anthropic_params,_map_openai_paramsand_map_litellm_defaultsmethods are gone;_map_provider_paramsstill applies them. Claude's sampling warning is logged byragas.llms.provider_rules. InstructorLLM(..., is_async=...)says whetherclientis async.llm_factory()sets it. Without it,InstructorLLMchecks whetherclient.chat.completions.createis a coroutine function, which holds for instructor'sAsyncInstructor. It no longer looks inside instructor's closures or atclient.client. If you constructInstructorLLMaround your own client and it is detected wrongly, passis_async=.- With
provider="google",InstructorLLMcallsclient.chat.completions.create, like every other provider, instead ofclient.create. On an instructor client these are the same method.
DiscreteMetric, NumericMetric and RankingMetric: one scoring path and range checks (#217)
These metrics, and SimpleLLMMetric (LLMMetric) subclasses, were refactored. Saving, loading and alignment moved to their own modules; the public methods stay.
- ⚠ silent:
score()now runsascore(), so it calls the LLM'sagenerate(), nevergenerate().llm_factory()LLMs support both, with sync or async clients, so most code is unaffected. A custom LLM object needsagenerate().result.traces["input"]is now the formatted prompt fromscore()too; it used to hold the keyword arguments. ascore()andscore()check the LLM's value againstallowed_valuesand raiseValueErrorwhen it is outside them. That means aNumericMetricvalue outside its range, or aRankingMetriclist of the wrong length. Before, the value was returned as is. Underevaluate()and@experimentthis followsraise_exceptions, like any other metric error: it is raised, or the row has no score and the error is logged.DiscreteMetricvalues were already restricted by its response model. See Score changes.- A metric with no prompt raises
ValueErrorinstead of a bareException. load()checks the saved metric type in the base class, soLLMMetric.load()and a custom subclass'sload()reject another class's file too. TheDiscreteMetric/NumericMetric/RankingMetricoverrides are unchanged. Files saved by earlier versions load as before.- A
NumericMetricwithallowed_values=range(...)can now be saved. It used to raiseTypeError. The range is written as{"range": [start, stop, step]}, which earlier versions can't load. align()andalign_and_validate()takemax_similar_examplesandsimilarity_thresholdas named arguments. Other keyword arguments raiseTypeError; they were ignored before. String values are no longer converted to numbers. The aligner itself isragas.metrics.alignment.Aligner.validate_alignment()pairs each label with its prediction before dropping rows where either is missing. Before, missing labels and missing predictions were dropped from separate lists, so after a gap the labels were compared with other rows' predictions.correlationandagreement_ratechange for test sets with missing labels or predictions. A test set with no complete row raisesValueErrorinstead ofZeroDivisionError. Itsmappingargument defaults toNone.- Persistence lives in
ragas.metrics.serialization. The privateSimpleLLMMetrichelpers_serialize_prompt,_deserialize_prompt,_get_metric_config,_serialize_response_model_info,_read_metric_typeand_validate_metric_typeare gone. Prompt,DynamicFewShotPromptand metricsave()/load()share one gzip-aware JSON helper (ragas.prompt.utils.read_json_file/write_json_file). A file that is not valid UTF-8 now raisesValueError("Cannot load ..."), like other unreadable files, instead ofUnicodeDecodeError.
AG-UI: one tool-argument parser and a separate SSE client (#217)
- Streamed (
ToolCallStart/Args/End) and chunked (ToolCallChunk) tool calls now parse their arguments the same way asMessagesSnapshottool calls. Arguments that are valid JSON but not an object, such as[1, 2]or5, are kept as{"raw_args": "<the text>"}with a warning. They used to failToolCallvalidation and stop the conversion. Invalid JSON is kept asraw_args, as before. - With
metadata=True, aToolMessagenow gets the currentstep_name, like text messages. - The SSE client moved to
ragas.integrations.sse.post_sse_json, andcall_ag_ui_endpoint()uses it. It also acceptsdata:lines with no space after the colon, as the SSE spec allows. Without httpx, theImportErrornow comes from the SSE client.
Testset generation and optimizers: one code path per job (#217)
TestsetGenerator's three entry points now share one graph builder and one check of their inputs.
generate_with_docs()andgenerate_with_chunks()takenum_personasandbatch_size, asgenerate()does. They used to always use 3 personas and no batching.- An LLM and an embedding model are needed only to build the default transforms. With your own
transforms,generate_with_docs()andgenerate_with_chunks()no longer raiseValueErrorwhen the generator has no embedding model. generate_with_docs(transforms=[])applies no transforms, asgenerate_with_chunks(transforms=[])already did. It used to replace an empty list with the default transforms.BaseSynthesizer.is_applicable(kg)says whether a synthesizer can work on a knowledge graph. It returns True unless overridden.default_query_distribution(kg=...)calls it, where it used to call each synthesizer'sget_node_clusters(). The three built-in synthesizers give the same answers as before. The shared theme extraction, persona matching, combination sampling and sample generation moved toBaseSynthesizerhelpers. The generated testsets are unchanged.- The seven LLM extractors (
SummaryExtractor,TitleExtractor,TopicDescriptionExtractor,KeyphrasesExtractor,HeadlinesExtractor,NERExtractor,ThemesExtractor) are built on two shared bases. Their fields, prompts and outputs are unchanged.LLMBasedExtractorno longer has themerge_if_possiblefield, which nothing read. Passing it raisesTypeError. CosineSimilarityBuilderuses the baseRelationshipBuilderexecution plan, as the other relationship builders do. A node without an embedding, or embeddings of different lengths, now raiseValueErrorwhen the transform runs, not whenapply_transforms()builds the plan. The relationships built are unchanged.KnowledgeGraph.find_n_indirect_clusters()andKnowledgeGraph.find_two_nodes_single_rel()moved toragas.testset.graph_queriesas functions that take the graph first:find_n_indirect_clusters(kg, n, ...)andfind_two_nodes_single_rel(kg, ...). The methods are removed. The results are unchanged.Optimizer.optimize(),GeneticOptimizer.optimize()andDSPyOptimizer.optimize()are nowoptimize(dataset, loss, config=None, *, run_config=None, batch_size=None, callbacks=None, raise_exceptions=True).configis optional, and the run options are keyword-only. Thewith_debugging_logsargument, which no optimizer read, is removed.metric.train()is unchanged.GeneticOptimizerchecks itsmetricandllmonce, whenoptimize()starts. Its steps no longer check them again.
Shared metric helpers (#217)
The collections metrics now share helpers instead of carrying copies. Metric class names, constructor arguments and ascore() signatures are unchanged. Score effects are in Metric score changes.
Custom metrics declare their components. ragas.metrics.collections.BaseMetric validates the components a metric lists in required_components (and, if not None, those in optional_components), and takes them as keyword arguments:
class MyMetric(BaseMetric):
required_components = ("llm",)
def __init__(self, llm, name="my_metric", **kwargs):
super().__init__(llm=llm, name=name, **kwargs)
- It used to validate any
llm/embeddingsattribute that happened to be set beforesuper().__init__(). A subclass that still setsself.llmand doesn't declare it is no longer validated. Declare it. - Passing
llm=orembeddings=to a metric that doesn't declare it raisesTypeError, like any other unknown keyword argument.
One wording for a missing input. Every collections metric raises ValueError("<field> is missing. Please add <field> to the test sample.") for a missing or empty required input. Before, some said "<field> cannot be empty" (ContextPrecision, ContextRecall, AnswerRelevancy), "Please provide ..." (the multimodal metrics), "reference must be provided for AgentGoalAccuracyWithReference" or "rubrics must be provided for instance-specific evaluation". Code matching the old messages must match the new one. Type checks (e.g. "user_input must be a list of messages") and whitespace checks are unchanged.
The NVIDIA dual-judge metrics run both judges at once. AnswerAccuracy, ResponseGroundedness and ContextRelevance send their two judge requests concurrently, so a sample has two LLM calls in flight. When the provider raises, both calls have been made. A retry after an off-scale rating uses temperature 0.3 and its own sample_index (see score changes).
FaithfulnesswithHHEM inherits ascore(). Faithfulness.ascore() checks inputs, extracts statements, then calls _verify_statements(statements, context_str). HHEM overrides only that hook. A subclass of either that verifies statements differently should override the hook too.
Moved or removed names:
| Removed | Use instead |
|---|---|
StatementGeneratorInput / Output / Prompt in collections.answer_correctness.util |
the same names in ragas.metrics.collections.faithfulness.util |
NLIStatementInput / Output / Prompt, StatementFaithfulnessAnswer in collections.factual_correctness.util |
the same names in ragas.metrics.collections.faithfulness.util |
the module collections.noise_sensitivity.util (StatementFaithfulnessInput / Output / Prompt aliases) |
NLIStatementInput / Output / Prompt in ragas.metrics.collections.faithfulness.util |
collections.tool_call_f1.util.calculate_f1_score |
ragas.metrics.utils.fbeta_score |
ragas.testset.transforms.relationship_builders.cosine.cosine_similarity_matrix |
ragas.similarity.cosine_matrix |
The private methods _calculate_average_precision (ContextPrecision), _format_conversation (AgentGoalAccuracy, TopicAdherence), _compare_outcomes (AgentGoalAccuracy) and _get_judge_rating / _average_scores (the dual-judge metrics) are gone. ContextPrecision no longer logs "Invalid response format ...": the warning could not fire.
MLflowTrace.get_url() checks MLFLOW_HOST and escapes trace IDs (#243)
MLFLOW_HOSTmust be an http(s) URL with a host, with no query string or fragment; anything else raisesValueError. A port and a path prefix (https://example.com/mlflow) still work, and the URL for a valid host is unchanged.- The experiment ID and trace ID are percent-encoded, so an ID containing
/,?,&or#can no longer add path segments or query parameters to the URL.
Published to PyPI as ragas-ng (#242)
- The distributions are renamed:
ragas→ragas-ng, andragas-examples→ragas-ng-examples. The import names are unchanged:import ragasandimport ragas_examples. Extras keep their names:pip install "ragas-ng[all]". - Versions use calendar versioning,
YYYY.M.PATCH(for example2026.11.0), so they can't be confused with upstream's0.xreleases. A ragas-ng version says nothing about which upstream release it matches. Pin it like any other version:ragas-ng>=2026.11. pip install ragasstill installs upstream ragas. Both distributions provide theragasimport package, so don't install both in one environment.- The quickstart templates'
pyproject.tomlfiles depend onragas-ng[all](they hadragas[all]>=0.3.0, which resolved to upstream). - Install hints in error messages and the docs say
ragas-ng[...]. Theragas[examples]hint, an extra that never existed here, is now theragas-ng-examplespackage. ragas.experimentalno longer lazily imports upstream's separateragas_experimentalpackage. It raisesAttributeError, like any other missing attribute.
Decorator metrics raise instead of returning value=None (#239)
Metrics built with @discrete_metric, @numeric_metric and @ranking_metric now handle errors the way DiscreteMetric / NumericMetric / RankingMetric do.
- An invalid value raises
ValueError. That covers a value not in the allowed values, a number outside the range, or a ranking list of the wrong length. The message is the same one those classes use, e.g.Metric m returned 1.5 but expected value in range (0.0, 1.0). Before,score()/ascore()returnedMetricResult(value=None, reason=<that message>). - Errors from your function propagate. Before, any exception, including LLM, network and coding errors, came back as
MetricResult(value=None, reason="Error executing metric <name>: ..."). - ⚠ silent:
result.value is Noneno longer detects a failure. Code that checked for it, or that readresult.reasonfor the error, now gets the exception instead. If a failure should count as a score, catch it inside the function and return that score; for example, return"incorrect"when the model's output can't be parsed. - Under
evaluate()and@experimentthese errors followraise_exceptions, like any other metric error: raised, or the row gets no score (evaluate()) or is left out (@experiment) and the error is logged. Before,raise_exceptions=Truecould never surface them. - The private
_validate_result_value()method is gone; usevalidate_result_value(), which returns the error message orNone.
Leftover aliases removed; DSPy run options; seeded annotation sampling (#240)
Removed aliases:
| Removed | Use instead |
|---|---|
ragas.cost.CostCallbackHandler |
ragas.cost.TokenUsageCollector (the same class) |
TestsetGenerator.generate_with_langchain_docs() |
TestsetGenerator.generate_with_docs() (the same arguments) |
SingleMetricAnnotation.train_test_split() |
none: it only raised NotImplementedError |
DSPyOptimizer uses train()'s run options. It used to accept them and ignore them.
run_config: itsmax_retriesandtimeoutbecome thedspy.LM'snum_retriesandtimeout, unless thellm_factory()arguments set them.train()always passes aRunConfig, so by default DSPy's calls now retry up to 10 times (DSPy's default was 3), with a 180-second timeout. A DSPy LM passed asDSPyOptimizer(lm=...)keeps its own settings.raise_exceptionssets MIPROv2's error limit whenDSPyOptimizer(max_errors=...)is unset. WithTrue,train()'s default, DSPy's own limit applies (dspy.settings.max_errors, 10), as before.Falsenow never stops on errors. An explicitmax_errorsstill wins.batch_sizeandcallbacksraiseNotImplementedErrorinstead of being ignored. MIPROv2 sizes its own minibatches and traces its own calls.
SingleMetricAnnotation.sample(), batch() and stratified_batches() take seed.
- They draw from a private
random.Random(seed)and never use or change the globalrandomstate. - ⚠ silent:
random.seed(...)no longer makes them reproducible; passseed=instead. Without a seed, each call draws fresh. - ⚠ silent:
GeneticOptimizerpasses seeds derived fromrun_config.seed, whichtrain()defaults to 42. The annotations it samples are now the same on every run with the same seed. Before, they followed the globalrandomstate.
metrics_experiment, summarize and check (#253)
New, as the first step of replacing evaluate() (#241). Nothing is removed yet.
ragas.metrics_experiment(metrics, ...)scores metrics over aDatasetas an experiment. It maps the fields the same wayevaluate()does, and runs the same up-front checks. Each row gets a<metric>and a<metric>_reasoncolumn. See Experiments.ragas.summarize(exp)andragas.check(exp, min_mean=..., max_mean=..., allow_missing=...)give the same numbers and the sameThresholdErrorasEvaluationResult.summary()/check(), for any experiment with metric columns.- Differences from
evaluate(): - there are no default metrics and no default OpenAI LLM or embeddings;
max_workerscounts rows, and a row's metrics run concurrently;- token usage goes to a
TokenUsageCollectoryou pass asusage=; - traces come from OpenTelemetry spans.
Annotation types moved to ragas.optimizers.annotations; the genetic optimizer scores without evaluate() (#254)
Step 2 of replacing evaluate() (#241).
The annotation types moved. There is no alias at the old path.
| Removed | Use instead |
|---|---|
from ragas.dataset_schema import PromptAnnotation, SampleAnnotation, MetricAnnotation, SingleMetricAnnotation |
from ragas.optimizers.annotations import ... (the same classes) |
SingleMetricAnnotation.to_evaluation_dataset() |
[s.metric_input for s in annotations], the rows it held |
GeneticOptimizer scores candidates directly. Before, each candidate went through evaluate().
evaluate_candidate()/aevaluate_candidate(candidate=, eval_dataset=, ...)are nowscore_candidate()/ascore_candidate(candidate=, rows=, ...).rowsaremetric_inputdicts, and the result is aRowScoreswithscores(one per row, NaN for a failed row) andtraces(each row's traced prompt calls), not anEvaluationResult. Therun_idargument is gone: the scoring run nests undercallbacks.- Scores, traces, fitness ranking and feedback mutation are unchanged.
- A metric that takes its
llmorembeddingsper call (such asDiscreteMetric) raisesValueError. Before,evaluate()filled in a default OpenAI model. Optimize a metric that holds its own models.
Testset and the integrations return a Dataset (#255)
Step 3 of replacing evaluate() (#241). Score the results with metrics_experiment.
| Removed | Use instead |
|---|---|
Testset.to_evaluation_dataset() |
Testset.to_dataset(name, backend="inmemory", **backend_kwargs): a Dataset of copies of the samples, with data_model set to SingleTurnSample or MultiTurnSample. Call save() to write it. A testset that mixes the two raises ValueError. |
Testset.to_hf_dataset() / Testset.from_hf_dataset() |
to_list() / from_list(), or to_jsonl() / from_jsonl() |
ragas.integrations.llama_index.evaluate(query_engine, dataset, metrics, ...) |
evaluate_query_engine(query_engine, dataset, metrics, ...) (aevaluate_query_engine in async code) |
- Save a testset you'll score as JSONL, not CSV. CSV loads a missing text value, such as a sample with no
reference, as"".metrics_experimentscores""as a value instead of reporting the column missing. See What each backend keeps. Testsetis no longer aRagasDataset. It keepsto_list,from_list,to_pandas,to_csv,to_jsonl,from_jsonl,from_annotated,total_tokensandtotal_cost, and iterates and has a length as before.from_pandas,from_dict,features()andget_sample_type()are gone.- griptape and r2r:
transform_to_ragas_dataset()returns aDatasetofSingleTurnSamplerows, not anEvaluationDataset. It's in memory by default; the new keyword argumentsname,backendand the backend's own arguments choose where it lives. evaluate_query_engine()takes aDatasetof single-turn rows (samples or dicts) and leaves it unchanged. It runs the engine, then scores the answered rows withmetrics_experimentand returns theExperiment: useragas.summarize()/ragas.check()on it, notEvaluationResultmethods.token_usage_parseris replaced byusage=TokenUsageCollector(...).callbacksandin_ciare gone; traces come from OpenTelemetry spans.name,backendandbackend_kwargsname the experiment and choose where it's saved (in memory by default).- There's no default LLM or embeddings for metrics that take them per call.
evaluate_query_engine(): a failed query scores missing (#264)
⚠ silent. When the query engine failed on a row, the row was still scored with response=None. A metric that accepts a missing response, such as ExactMatch, gave it a real score (0.0), which lowered the mean while summarize() reported nothing missing. The old ragas.integrations.llama_index.evaluate() did the same.
- Now a row whose query failed scores NaN for every metric, and each
<metric>_reasonisquery engine failed: <ExceptionType>: <message>. summarize()counts the row as missing, andcheck()fails on it unlessallow_missingpermits it.- If every query fails, the run still completes, with every row missing. Before, the up-front column check raised.
- With
raise_exceptions=True, the first failed query still stops the run.
evaluate(), aevaluate() and EvaluationResult removed (#257)
Step 5 of replacing evaluate() (#241). See From evaluate() to experiments for a before/after of every argument and method.
| Removed | Use instead |
|---|---|
ragas.evaluate() / ragas.aevaluate(), ragas.evaluation |
ragas.metrics_experiment(metrics).run(dataset) / await ....arun(dataset) |
ragas.dataset_schema.EvaluationResult |
the returned Experiment, with ragas.summarize() and ragas.check() |
ragas.validation (remap_column_names, convert_v1_to_v2_dataset, ...) |
metrics_experiment(..., column_map={...}); rename v1 columns (question, answer, contexts, ground_truth) yourself |
ragas.exceptions.ExceptionInRunner |
nothing: only evaluate() raised it |
ragas.embeddings.base.default_embeddings_for_litellm_llm() |
pass embeddings= to the metrics that need them, e.g. embedding_factory("litellm", model="gemini/gemini-embedding-2") |
- No default metrics, LLM or embeddings.
evaluate(metrics=None)ran four default metrics with a default OpenAI LLM and embeddings matched to your LLM. Build the metrics with the models you want. default_llm()stays: testset generation still uses it.ThresholdErrorstays;ragas.check()raises it.
EvaluationDataset removed; datasets is no longer a dependency (#258)
The last step of replacing evaluate() (#241). See From evaluate() to experiments for each constructor and converter.
| Removed | Use instead |
|---|---|
ragas.EvaluationDataset, ragas.dataset_schema.EvaluationDataset |
Dataset("name", "inmemory", data_model=SingleTurnSample, data=samples) |
ragas.dataset_schema.RagasDataset |
nothing: Testset (#255) and EvaluationDataset were its only subclasses |
EvaluationDataset.from_hf_dataset(hf) / to_hf_dataset() |
Dataset.from_pandas(hf.to_pandas(), ...) / datasets.Dataset.from_pandas(dataset.to_pandas()) |
ragas.dataset_schema now holds the sample types (BaseSample, SingleTurnSample, MultiTurnSample) and sample_from_dict / sample_type_for.
Dependencies:
- Hugging Face
datasetsis no longer installed with ragas, and neither ispyarrow, which onlydatasetsneeded. To load data from the Hub,pip install datasets. pandasis now a direct dependency. It used to arrive only throughdatasets, andto_pandas()andragas hello-worldneed it.huggingface_hubis no longer guaranteed.HuggingFaceEmbeddings(use_api=True)needspip install huggingface-hub, and says so if it's missing. The localsentence-transformerspath brings its own copy.
Quickstart projects install on Python 3.13+; [all] skips r2r there (#270)
ragas-ng[all]no longer installsr2ron Python 3.13 or newer. r2r's releases support only Python <3.13 and pintiktoken<0.9, which has no wheel for 3.14. Because uv ignores upper bounds onrequires-python,ragas-ng[all]failed to install there. The r2r integration (ragas.integrations.r2r) doesn't import r2r, so it still works. If you need the r2r client itself, use Python 3.12 or older.- The
rag_evalandimprove_ragquickstart templates depend on what they import instead ofragas-ng[all].improve_ragno longer lists LangChain packages, which it never imported, and now listsopenai-agents, which its agentic mode needs. A project generated before this change can replaceragas-ng[all]in itspyproject.tomlwith the new list. - The
ragas-ng-examplespackage ships each template'spyproject.toml. In 2026.11.0 it didn't, soragas quickstartwithragas-ng-examplesinstalled created a project with nothing foruv syncto install. Withoutragas-ng-examples, the CLI downloads templates from GitHub and wasn't affected.