Skip to content

Behavior changes from ragas

ragas-ng fixes bugs in legacy ragas even where code that depended on the old behavior has to change. This page lists those changes, apart from score changes, which are listed in Metric score changes. Each entry says what changed and what, if anything, you need to do.

Entries marked ⚠ silent don't raise an error, so read them first. Your code keeps running, but it may now do something different.

Running sync APIs: no more nest_asyncio (#80)

ragas no longer depends on or imports nest_asyncio. Every sync entry point (evaluate(), metric.score(), batch_score(), @metric metrics, the executor and testset transforms) runs on one shared background event loop. They work inside Jupyter and async frameworks without patching the running loop. Repeated calls also reuse one loop, so async clients such as AsyncOpenAI stay bound to it. A second score() used to fail with "bound to a different event loop".

Before Now What to do
evaluate(..., allow_nest_asyncio=False) accepted, ignored, DeprecationWarning remove the argument
ragas.async_utils.apply_nest_asyncio() no-op returning False, DeprecationWarning remove the call
score() inside a running loop raised "Cannot call sync score() from an async context" works nothing

One case to avoid: in a notebook, if you await an async client yourself on the notebook's loop and then pass the same client to sync evaluate(), the client is bound to the notebook's loop, not ragas'. Use await aevaluate(...) in async code.

evaluate() leaves your objects alone (#76)

evaluate() works on copies of your metrics and never sets anything on your LLM. The run_config and token counting apply only to calls made during that run, including jobs from return_executor=True.

  • ⚠ silent: metric.llm and metric.embeddings stay as you left them (often None) after evaluate(llm=...). Code that read metric.llm after a run to find the LLM it used now gets None.
  • ⚠ silent: without run_config=, evaluate() leaves your LLM's own set_run_config(...) in effect. It used to overwrite it with defaults.
  • Two concurrent evaluate() calls on shared metrics or a shared LLM no longer mix up their LLMs, settings or token counts.
  • TestsetGenerator works the same way, and no longer leaves its token counter attached to your LLM.

evaluate() inputs and results (#77, #78, #79)

Input or call Before Now
dataset=[SingleTurnSample(...), ...] or a list of dicts failed later converted to an EvaluationDataset
an empty list, or a list mixing types failed later ValueError / TypeError before the run
a raw provider client as llm= / embeddings= failed during the run TypeError before the run, pointing to llm_factory() / embedding_factory()
metrics=(m1, m2) (a tuple) rejected accepted
a prompt that runs more than once per sample in traces only the last run was kept name, name (2), name (3), ...
total_tokens() / total_cost() with no usage recorded IndexError zero, with a warning

Testset generation: size, failures and personas (#81, #82, #84)

  • ⚠ silent: testset_size is an upper bound. Testsets used to overshoot it, for example 12 samples for 10, or 3 for 1. The size is now split exactly across the synthesizers. You can get fewer samples if the knowledge graph is too small, and a warning then gives the count and the reason.
  • With raise_exceptions=False, a failed scenario or sample is left out, with a warning. It used to crash generation and lose all the completed work.
  • Persona names returned by the LLM match ignoring case and spacing. Names that match no persona are skipped instead of raising KeyError.
  • Personas generated on the first generate() call are still saved on the generator as persona_list and reused, but a list you supplied is no longer shuffled in place. New: testset.personas holds the personas used.
  • Repeatable personas across runs. Persona generation runs at temperature 1.0 by design, so each new generator or script run gets a different set (see Prompt output handling). To get the same personas, and so the same mix of questions, every time, save them once and pass them back:

    import json
    from ragas.testset.persona import Persona
    
    # first run
    with open("personas.json", "w") as f:
        json.dump([p.model_dump() for p in testset.personas], f)
    
    # later runs
    with open("personas.json") as f:
        personas = [Persona(**p) for p in json.load(f)]
    generator = TestsetGenerator(llm=llm, embedding_model=emb, persona_list=personas)
    
    - with_debugging_logs=True now turns on the ragas.testset.* loggers. It had no effect before.

Testset generation: building the knowledge graph (#83, #85–#88)

  • ⚠ silent: every document gets a chunk. HeadlineSplitter turns a document it doesn't split into one CHUNK node. It used to add the document a second time under the same id, and that document never got a chunk to generate questions from. If you load a knowledge graph saved with an older version, its unsplit documents still have no chunks.
  • default_transforms handles mixed-length corpora:
  • The long-document pipeline no longer crashes on documents under 500 tokens.
  • The medium pipeline is used whenever any document is over 100 tokens.
  • An empty list raises ValueError.
  • If every document is 100 tokens or shorter, the error points to generate_with_chunks().
  • CustomNodeFilter scores chunks you supplied yourself using their own summary. It used to skip them with a warning.
  • apply_transforms stops at the first failure. It cancels the step's remaining tasks, doesn't run later steps, and raises the error. It used to run every task in the step first. There is no rollback, so retry on a fresh KnowledgeGraph.

Testset generation: cost and metadata (#90, #91)

  • ⚠ silent: testset.total_tokens() / total_cost() from the generate_with_* methods now include building the knowledge graph, usually most of the cost, so the totals are higher. The transforms also get the run's RunConfig timeouts and retries.
  • generate_with_llamaindex_docs without run_config= keeps your LLM's own settings. It used to apply defaults.
  • New field SingleTurnSample.reference_contexts_metadata: the source document's metadata for each reference context, kept as JSON-safe values. It is a new column in to_pandas() output. If you write CSV from datasets where only some samples have the field, to_csv() can fail on the mixed columns.

Few-shot examples (#92, #93)

  • ⚠ silent: the similarity threshold applies. FewShotPydanticPrompt.threshold_for_examples, which DemonstrationConfig.threshold sets during metric.train(...), never reached the example store, so its default of 0.7 always applied. It does now.
  • ⚠ silent: DynamicFewShotPrompt shows no examples when none reach similarity_threshold. It used to fall back to the most recent examples. See Metric score changes.
  • Asking for 0 examples or history entries returns none. This applies to top_k=0, max_similar_examples=0 and DSPy inspect_history(0). [-0:] used to return them all. A negative count raises ValueError.
  • metric.train(...) with annotations whose output fails validation no longer shifts the remaining examples. Before, each later input was paired with the previous annotation's output.
  • Custom example stores: the abstract ExampleStore.get_examples and SimpleExampleStore.get_examples now take threshold. A store without that parameter still works, but the threshold is ignored and a warning says so once. Add threshold: float = 0.7 to your store to use it.

Prompt output handling (#94)

This applies to PydanticPrompt and ImageTextPrompt: the prompt classes behind testset generation, the private legacy metric modules, and metrics with customised prompts.

  • Error type: parse failures raise RagasOutputParserException. This covers a ValidationError or JSONDecodeError, and instructor's retries-exhausted errors with an llm_factory() LLM. Some of these used to escape as their raw types. The original is kept as __cause__, and the message includes it. If you caught pydantic.ValidationError around a prompt call, catch RagasOutputParserException instead. Provider errors such as timeouts, rate limits and connection errors still pass through unchanged.
  • Every failure is reported to callbacks with on_chain_error, so traces no longer leave a failed prompt run open.
  • retries_left is the number of fix attempts. When the LLM's output doesn't parse, ragas asks the LLM to fix it up to retries_left times (default 3). Each fixed output is parsed like the original, including JSON inside code fences or surrounding text. Before, there was one fix attempt, and its result failed if it came back fenced.
  • Output that is already valid JSON is parsed as is. JSON with a fenced block inside one of its string values used to be cut down to that inner block.
  • ⚠ silent: llm_factory() LLMs run prompts concurrently. The prompt called the blocking generate() for an LLM on a sync client, so concurrent prompts, such as every testset transform, ran one at a time. It now awaits agenerate(), which runs sync clients in a thread. Expect faster testset generation and more parallel requests to your provider. Set RunConfig(max_workers=...) if you need to limit them.
  • ⚠ silent: n and temperature apply with llm_factory() LLMs.
  • generate_multiple(n=3) makes three calls at a sampling temperature, 0.3 unless you pass one. It used to make one call and warn.
  • A temperature= passed to a prompt is now used. It used to be ignored.
  • Testset personas vary between runs again. Persona generation has asked for temperature 1.0 since it was added, so that personas come out varied. That worked with the legacy LangChain-style LLM wrappers, which honour the temperature. The code path for llm_factory() LLMs, added later, dropped it, so those users got nearly the same personas every time without anyone choosing that. The original design applies again: a new generator or script run gets a different set of personas. To keep them fixed, see Repeatable personas across runs.
  • See Metric score changes.

@metric function arguments (#95)

This applies to metrics created with @discrete_metric, @numeric_metric, @ranking_metric or create_metric_decorator().

  • ⚠ silent: arguments arrive as the objects you passed. The decorator validates inputs with pydantic and then called your function with model_dump(), which turned pydantic-model arguments, such as a SingleTurnSample or your own BaseModel, into plain dicts. Your function now receives the objects themselves. Coercion still applies, so "2.5" becomes 2.5 for a float parameter. If your function worked around this by indexing a dict (sample["user_input"]), switch to attribute access (sample.user_input).
  • ⚠ silent: concurrent calls no longer share inputs. The validated inputs were stored on the metric object between validation and the call. Two calls on one metric from different event loops, such as aevaluate() on your loop alongside score(), could run with each other's inputs and return the wrong score. Each call now keeps its own inputs.
  • If the inputs can't be validated, for example because of a type hint pydantic can't resolve, the function is called with your arguments and a warning is issued once. Before, it either crashed score() or called the function with no arguments, and the error was returned as MetricResult(value=None).

CSV files: every row's columns, JSON cells, typed loads (#100)

This applies to LocalCSVBackend ("local/csv") and EvaluationDataset.to_csv().

  • Rows may differ. The header is the union of every row's keys, in first-seen order, and a missing value is an empty cell. Before, any row with a key the first row lacked raised ValueError: dict contains fields not in fieldnames. Because @experiment saves its results at the end, that lost the whole run's results.
  • ⚠ silent: lists and dicts are JSON text. Cells used to hold Python repr, such as ['a', 'b']. They now hold JSON, such as ["a", "b"]. If you parse these cells yourself with ast.literal_eval, use json.loads.
  • Typed loads restore types. Dataset.load(..., data_model=Row), reload() and validate_with(Row) decode each cell using the field's type before validation, so lists, dicts, nested models, numbers, booleans, dates and empty optional fields load back. Before, any list, dict or Optional[...] = None field failed validation.
  • str fields are never changed. CSV can't tell "" from None, so an Optional[str] saved as None loads as "". Without a data_model, CSV loads still return strings. Use "local/jsonl" to keep types without a model.
  • Dataset.save() writes model rows with model_dump(mode="json"), so values such as UUIDs, enums and paths are saved as strings, and the model parses them back. Before, they could fail to save to JSONL.

Backends load what was stored (#101, #102)

Backends no longer guess types on load. Types come from the data_model you load with.

  • ⚠ silent: JSONL keeps strings as strings. LocalJSONLBackend ("local/jsonl") turned any string that looked like an ISO date, such as "2024-01-02", into a date or datetime on load. That changed your text, and a model's str field then rejected it. Now:
    • Loading with a data_model: date / datetime fields parse their ISO strings and str fields keep their text, so typed datasets round-trip exactly.
    • Loading without one: dates you saved come back as ISO strings, such as "2024-01-15T10:30:00". Use datetime.fromisoformat(...), or load with a model.
  • A malformed JSONL line raises ValueError naming the line and file. It used to be skipped with a print(), so rows disappeared silently. Blank lines are still skipped.
  • ⚠ silent: Google Drive cells stay text. GDriveBackend turned numeric-looking cells into numbers on load, so "007" became 7. Cells are now strings unless a data_model types them, as with CSV. Also:
    • None is saved as an empty cell instead of the text "None".
    • Lists and dicts are saved as JSON text.
    • Loads read every column. Only columns A to Z were read before, so wider datasets silently lost columns.
  • Dataset files are UTF-8. to_csv(), to_jsonl(), from_jsonl(), MetricAnnotation.from_json() and SingleMetricAnnotation.from_json() read and write UTF-8 and close their files. They used the platform encoding, which breaks non-ASCII text on Windows. from_jsonl() ignores blank lines.

Message converters: AG-UI snapshots and OpenAI-format histories (#96, #98)

  • ⚠ silent: AG-UI MessagesSnapshotEvent tool calls keep their name and arguments. convert_messages_snapshot() read name / args from the wrong place: AG-UI nests them under function, with arguments as a JSON string. So every snapshot tool call came out as unknown_tool with {}, and tool-call metrics such as ToolCallAccuracy scored those calls as wrong. Scores for snapshot-based samples change. Arguments that aren't valid JSON are kept as {"raw_args": "..."}, as in the streaming path.
  • AG-UI message content: None is "". It used to be the text "None". A list of content parts keeps its text parts, joined by newlines; image, audio, video, document and binary parts are skipped. It used to be the pydantic repr of the whole list.
  • ragas.integrations.swarm.convert_to_ragas_messages() takes any OpenAI-format chat history, not only Swarm's:
    • An assistant message without tool_calls (or with None) no longer raises KeyError.
    • Arguments that aren't a JSON object are kept as raw_args instead of raising.
    • List-of-parts content keeps its text parts.
    • system and developer messages are skipped instead of raising ValueError.
    • An unknown or missing role still raises.

Langfuse helper removed (#97)

ragas.integrations.tracing.langfuse is removed, and importing it raises ModuleNotFoundError. It never worked in ragas-ng: sync_trace() returned a placeholder trace with no observations, and LangfuseTrace.filter() always returned []. Langfuse's own SDK and OpenTelemetry cover both of its jobs without vendor-specific code in ragas, so it is removed rather than ported to the Langfuse SDK.

Before Now
from ragas.integrations.tracing.langfuse import observe (or from ragas.integrations.tracing import observe) from langfuse import observe
await sync_trace(trace_id) / LangfuseTrace(...) from langfuse import get_client; get_client().api.trace.get(trace_id)
trace.filter("span_name") [o for o in trace.observations if o.name == "span_name"] on the Langfuse SDK's trace
add_query_param(url, name, value) urllib.parse: urlencode / urlunparse
pip install "ragas[tracing]" installed langfuse it installs only mlflow; pip install langfuse if you use it

To see ragas runs in Langfuse, export ragas' OpenTelemetry spans to it. The Langfuse SDK's default export filter drops spans from unknown instrumentations, ragas among them, so allow the ragas scope with should_export_span. The Langfuse guide shows both set-ups and how to score your app's traces with ragas.

The MLflow helper, ragas.integrations.tracing.mlflow, is unchanged.

Adapted testset prompts write in the target language (#106)

adapt_prompts(language, ...) recorded the target language on each prompt but never put it in the rendered prompt. An adapted prompt carried only a translated instruction (with adapt_instruction=True) and translated examples, so the generated queries followed the language of the documents and personas. A French corpus adapted to "spanish" produced a mix of French, English and Spanish queries.

  • The single-hop and multi-hop query/answer generation prompts now tell the model to write their output in the adapted language. Testset queries and reference answers come out in that language, whatever the corpus's language is.
  • This is opt-in per prompt, through the states_output_language class attribute on PydanticPrompt. Other prompts don't get the line: persona matching, entity extraction and theme combination must copy values from their input exactly, and translating those values (for example, a persona named "curious student" coming back as "Estudiante curioso") drops every scenario.
  • Metric prompts are unchanged. Adapting a metric still translates its instruction and examples. It doesn't promise the language of the metric's output (reasons, extracted statements), and scores are unaffected.
  • Prompts in English, and prompts never adapted, render exactly as before.

Experiment backends by name (#165)

@experiment(backend=...) and arun(..., backend=...) accepted a backend name such as "local/jsonl", but created it with no constructor arguments, so it raised TypeError for the missing root_dir. Only "inmemory" worked by name. Both now take backend_kwargs:

@experiment(Result, backend="local/jsonl", backend_kwargs={"root_dir": "./data"})
async def run(row): ...


await run.arun(dataset, backend="local/csv", backend_kwargs={"root_dir": "./data"})
  • backend_kwargs given to arun replaces the decorator's.
  • backend_kwargs with a backend instance, or with no backend, raises ValueError. Before, there was no such parameter, so this breaks nothing.
  • backend_kwargs is keyword-only and reserved, like name and backend. An experiment function with a parameter of that name can't receive it through arun.

Experiments: concurrency, threads, order and failures (#117)

ExperimentWrapper.arun (await my_experiment.arun(dataset)) changed in four ways:

  • At most 16 rows run at once by default. Before, every row started at once, so a large dataset sent all its LLM calls together and hit provider rate limits. Set @experiment(max_workers=...) or arun(..., max_workers=...). -1 restores the old unlimited behavior. Runs with many fast rows may take longer than before.
  • Sync experiment functions run in worker threads, up to max_workers at once. Before, they ran one after another on the event loop, blocking it. A function that isn't thread-safe needs max_workers=1. Calling the decorated function directly (await my_experiment(row)) still runs it inline.
  • ⚠ silent: results are in dataset order. Before, they were stored in the order rows finished, so row i of the results was usually not row i of the dataset.
  • Failed rows are logged. A row whose function raises is still left out of the results. It's now a warning on the ragas.experiment logger, with the row index and error, instead of a print(), and a final warning gives "N of M rows failed and were not saved".

max_workers is a keyword-only parameter of arun, like name, backend and backend_kwargs, so an experiment function can't receive a parameter of that name through arun.

Retired model IDs replaced (#176)

Docs, examples, ragas quickstart templates and docstrings named models that their providers have shut down, or will shut down soon: - gpt-3.5-turbo, gpt-4, gpt-4.1-nano and gemini-2.0-flash, which fail today; - gpt-5-mini and gpt-5-nano (2026-12-11), and claude-sonnet-4-5 (2026-11-30); - gpt-4o and gpt-4o-mini (2027-01-14).

They now use each provider's listed replacement:

Was Now
gpt-4o, gpt-4o-mini, gpt-4.1, gpt-4.1-mini gpt-6-luna
gpt-3.5-turbo, gpt-5-mini gpt-5.6-terra
gpt-4.1-nano, gpt-5-nano gpt-5.6-luna
gpt-4, gpt-4-turbo, gpt-5, o1/o3 gpt-5.6-sol
claude-sonnet-4-5, Claude 3.x Sonnet claude-sonnet-5-5
gemini-2.0-flash, gemini-1.5-flash gemini-3.6-flash
gemini-1.5-pro, gemini-2.5-pro gemini-3.8-flash
text-embedding-004 (Gemini) gemini-embedding-2
  • The default judge model changed: see Score changes.
  • The new OpenAI models are all reasoning models. llm_factory() already maps parameters for them. Code that calls the OpenAI client directly can't send temperature (other than 1) or max_tokens (use max_completion_tokens). On Chat Completions, gpt-6-luna supports tool calls only with reasoning_effort="none". The workflow_eval and agent_evals templates were updated accordingly.
  • Docs output: pages that show LLM output from earlier runs say so at the top. That output was produced with the old models.
  • Your own code: if it names any of the old IDs, switch it as well. See your provider's deprecations page.

Claude: sampling parameters (#177)

  • llm_factory(..., provider="anthropic", client=Anthropic()) no longer sends ragas' default temperature / top_p. Every call with anthropic SDK 2.6 used to fail with TypeError: ... unexpected keyword argument 'temperature'.
  • A temperature or top_p you pass (llm_factory(..., temperature=0.2)) is still sent. On current Claude models it fails, and the provider's error says why: only temperature=1 is accepted on older API versions, and none on API 2026-10-21.
  • A per-call temperature ragas chooses for sampling (strictness, legacy prompts with n > 1) is not sent to Claude. Through LiteLLM it is sent with drop_params=True, so LiteLLM drops it where a model rejects it.
  • The planned move of every call onto LiteLLM's parameter handling is #182.

Claude: JSON mode by default (#184)

Without a mode, llm_factory() now asks Claude for structured output as JSON: - Mode.ANTHROPIC_JSON with an Anthropic client (provider="anthropic", client=Anthropic()); - Mode.JSON for Claude through LiteLLM ("anthropic/claude-...", including Bedrock's anthropic.claude-... IDs).

Instructor's defaults (ANTHROPIC_TOOLS / TOOLS) force a tool call. claude-sonnet-5-5 rejects that with 400 tool_choice: type "tool" and "any" are not supported for this model, so every structured call failed on both paths.

  • Verified live with claude-sonnet-5-5 and claude-haiku-5-5 on both paths: generation, AspectCritic (with and without strictness), and AnswerAccuracy.
  • A mode you pass is kept, so mode=instructor.Mode.ANTHROPIC_TOOLS restores the old behavior for models that accept it.
  • A LiteLLM Router or module passed as client keeps instructor's default, because the model isn't known when it's wrapped. Pass mode= there.

NVIDIA collections metrics raise on LLM errors (#185)

  • AnswerAccuracy, ResponseGroundedness and ContextRelevance used to catch every exception from the judge call and turn it into nan. A misconfigured judge (wrong model, bad key, a provider rejecting a parameter) scored nan and the error was lost.
  • Provider and transport errors now propagate, as in other collections metrics. With evaluate(raise_exceptions=False) the row is still missing, and the error is logged.
  • Only an invalid rating (outside the metric scale after max_retries attempts) still gives nan.
  • DataCompyScore is unchanged: its except covers CSV parsing, not an LLM call.

Model names in examples and hints (#181)

  • Example templates name each model once, near the top of the module that uses it:
  • RAGAS_EXAMPLE_APP_MODEL sets the model the app or agent under test calls;
  • RAGAS_EXAMPLE_JUDGE_MODEL sets the model that judges.

Unset, each template uses the same default as before. - Error messages and deprecation hints that suggest a model use ragas.llms.base.DEFAULT_LLM_MODEL. - Metric docstrings show llm_factory("<model>", client=client) instead of a specific model. - Model retirements are tracked in scripts/model_registry.toml, and a unit test fails when a retired model is named in code, docs or examples. See scripts/README.md. - text2sql/analyze_errors.py used gpt-5 (shutting down 2026-12-11) and now uses gpt-5.6-sol. A source comment and two doc lines still named gpt-5 or the shut-down o1; they were updated.

One path for every LLM: LiteLLM (#182)

llm_factory() returns an InstructorLLM, and every call goes through LiteLLM. ragas no longer wraps each provider's own SDK.

Breaking: native SDK clients of providers other than OpenAI are rejected with ValueError. That covers Anthropic(), genai.Client() and GenerativeModel, Mistral(), Groq(), Cohere(), boto3 Bedrock and xAI, and instructor-patched clients. The error names the model string to use:

Before Now
llm_factory("claude-sonnet-5-5", provider="anthropic", client=Anthropic()) llm_factory("anthropic/claude-sonnet-5-5")
llm_factory("gemini-3.6-flash", provider="google", client=genai.Client(...)) llm_factory("gemini/gemini-3.6-flash")
a google-genai Vertex client llm_factory("vertex_ai/gemini-3.6-flash", vertex_project=..., vertex_location=...)
Mistral / Groq / Cohere clients llm_factory("mistral/<m>") / "groq/<m>" / "cohere_chat/<m>"
a boto3 Bedrock client llm_factory("bedrock/<model id>", aws_region_name=...)
mode=instructor.Mode.ANTHROPIC_JSON / ANTHROPIC_TOOLS (and other native modes) Mode.JSON / Mode.TOOLS; native modes raise

The provider's key comes from its environment variable (ANTHROPIC_API_KEY, GOOGLE_API_KEY or GEMINI_API_KEY, ...), or pass api_key=. To keep your own instructor client, construct InstructorLLM(client=..., model=..., provider=...) directly. Its per-provider parameter rules are unchanged.

Unchanged, but routed differently: - OpenAI / AsyncOpenAI and AzureOpenAI / AsyncAzureOpenAI clients still work, through LiteLLM, as openai/<model> or azure/<deployment>. - LiteLLM gets a copy of the client, because it writes max_retries and organization onto the object it receives. Your client is left untouched. - evaluate() still builds its default embeddings from your client. - A litellm.Router as client works as before.

Deprecated: - llm_factory(adapter=...) is ignored, with a DeprecationWarning. - ragas.llms.adapters (get_adapter, auto_detect_adapter, InstructorAdapter, LiteLLMAdapter) still imports, and both adapters call llm_factory(). - Constructing LiteLLMStructuredLLM warns. llm_factory() no longer returns it.

All three are removed in the next release.

Parameters (one set of rules for every llm_factory() LLM): - ragas' defaults temperature=0.01 and top_p=0.1 are sent with LiteLLM's drop_params, so LiteLLM drops them for models that reject them. - Values you set are sent as given. - max_tokens is sent only if you set it. The default of 1024 used to truncate long and reasoning outputs on the client path. - The OpenAI reasoning-model and Claude rules are unchanged (see LLMs (llm_factory)).

Token usage: responses are OpenAI-shaped for every provider, so use get_token_usage_for_openai.

Embeddings: the default embeddings for a Gemini LLM are now ragas' default (OpenAI's text-embedding-ada-002 through LiteLLM). They used to be Google's, when you passed a google-genai client. Pass GoogleEmbeddings(...) explicitly for a Gemini-only setup. GoogleEmbeddings still takes a google-genai client.

Tracing: metrics, LLM calls and experiments (#196)

Collections metrics, llm_factory() LLMs and @experiment take no callbacks, so they used to emit no OpenTelemetry spans and add nothing to the run tree. Now:

  • Metrics: every metric's score() / ascore() is an EVALUATOR span. That covers ragas.metrics.collections metrics and DiscreteMetric, NumericMetric, RankingMetric and decorator metrics. A metric called on its own is a root span, or a child of your application's active span. Under evaluate() it is the metric span evaluate() opens, not a second one.
  • LLM calls: every call to an llm_factory() LLM is an LLM span named after its response model. It records the prompt, the validated response, llm.model_name, the messages, the invocation parameters and token counts. It nests under the metric, legacy prompt or experiment row that made it.
  • Prompts reach your tracing service, truncated by default. LLM spans carry the judge's prompt, which holds your evaluation data. Their messages, like every span's input.value / output.value, are cut to 4,096 characters. Set RAGAS_TRACE_FULL_CONTENT=true to send them in full. See Observability.
  • Experiments: arun() is an experiment <name> span, with one row N span per row. A failed row is an error span. Metrics scored in the experiment function nest under their row.
  • result.traces has each LLM call's input and output again. Since metrics imported from ragas.metrics started handing off to collections, result.traces[i][metric] had been empty for llm_factory() LLMs. It now maps each call to {"input": {"prompt": ...}, "output": <response model>}, keyed by the response model's name, e.g. AspectCriticOutput.
  • A legacy prompt run without callbacks nests under the run in progress instead of starting a new trace.
  • Span payloads render pydantic models as JSON fields instead of repr().

Without an OpenTelemetry SDK, spans are still no-ops. The LLM call is unchanged unless a usage collector or a recording span needs the provider's raw response for token counts.

FaithfulnesswithHHEM loads a pinned model commit (#195)

The HHEM model is loaded with trust_remote_code=True, which runs the model repo's modeling_hhem_v2.py. Both FaithfulnesswithHHEM metrics (collections and legacy) used to load it from the repo's main, so a new push there changed the code ragas ran with no ragas change. They now load commit de9a5e6 ("Support transformers 5", 2026-10-06), which was reviewed. The collections metric takes revision= to load another. Scores are unchanged: on the model card's examples, this commit gives the card's published scores under both transformers 4.57 and 5.17.

The reported 0.0 for "The first superbowl was held on Jan 15, 1967" is HHEM working as designed. The example's context names only the "First AFL–NFL World Championship Game", and HHEM, a strict entailment model, does not credit what the context leaves to outside knowledge. The docs example now uses a context that states it.

Metric reasons: the verb agrees with the count (#200)

The first line of MetricResult.reason for Faithfulness, FaithfulnesswithHHEM, FactualCorrectness, NoiseSensitivity and ContextRecall always said "are" after the count, as in "1 of 1 statement in the response are entailed". The verb is now "is" when one item is counted or there is only one ("1 of 2 statements … is supported", "0 of 1 statement … is"), and "are" otherwise. Scores are unchanged. Code that matches on the exact reason text may need updating.

evaluate(): default embeddings follow a LiteLLM LLM's provider (#190)

When a metric needs embeddings and none are passed, evaluate() used to give every client-less llm_factory() LLM OpenAI's text-embedding-ada-002, so llm_factory("gemini/...") with AnswerRelevancy or AnswerCorrectness needed OPENAI_API_KEY. The embedding model now comes from the LLM's prefix:

  • gemini/... gets gemini/gemini-embedding-2, using the LLM's api_key if it was given one.
  • vertex_ai/... gets vertex_ai/gemini-embedding-2, with the LLM's vertex_project, vertex_location and vertex_credentials.
  • Bare and openai/ names keep text-embedding-ada-002.
  • Other providers (ollama/, anthropic/, bedrock/, ...) also keep it, but now with a UserWarning that says to pass embeddings=.

⚠ Scores change for Gemini and Vertex users of embedding metrics who relied on the default, since a different embedding model gives different similarities. Pass embeddings= to keep the old model. LLMs built with an OpenAI or Azure client are unchanged: they still take embeddings from that client.

Experiments: raise_exceptions stops on the first failed row (#170)

@experiment(raise_exceptions=True) or arun(..., raise_exceptions=True) makes arun stop at the first row whose function raises, like evaluate(raise_exceptions=True). The rows still running are cancelled, rows not yet started never run, nothing is saved, and that row's exception is raised from arun, with a note naming the row index. The default, False, is unchanged: the row is logged and left out, and the run goes on.

Add save_partial=True (decorator or arun) to save the rows that finished before the stop instead; the error is still raised, and its note says how many rows were saved. Without raise_exceptions it has no effect.

raise_exceptions and save_partial are now arun's own keyword arguments, like name, backend, backend_kwargs and max_workers. An experiment function that took a keyword argument of either name through arun no longer receives it.

Experiments: a sync run() (#172)

@experiment functions now have run(dataset, ...), the synchronous form of await arun(dataset, ...). It takes the same arguments and returns the same Experiment, so a script no longer needs asyncio.run(fn.arun(...)). Like evaluate() and metric.score(), it runs on ragas' shared background event loop (#74, #80), so it also works inside a running loop such as Jupyter. Ctrl-C cancels the rows still running.

evaluate() / Executor: stopping stops the jobs (#207)

  • raise_exceptions=True no longer leaves jobs running. The first error used to be raised while the other jobs ran on in the background: for evaluate(), LLM calls still being made, and paid for, after the error. Now the jobs still running are cancelled and awaited before the error propagates, and no job starts after the failing one (with max_workers=1, the rest never run).
  • Executor.cancel() takes effect within about 50 ms. It used to wait for the next job to finish, then cancel only what hadn't finished yet. Jobs still running are cancelled; results() returns what had finished.
  • The cancel path no longer leaves a coroutine un-awaited (RuntimeWarning: coroutine ... was never awaited).

ragas.async_utils.as_completed is unchanged apart from that warning; ragas itself now uses concurrent_results.

Requests and tokens per minute: RateLimit (#171)

New: llm_factory(..., rate_limit=RateLimit(rpm=..., tpm=...)) (from ragas.rate_limit) paces every request the LLM sends, waiting until the limit has room instead of hitting a 429. max_workers only limited jobs at once. Pass the same RateLimit to every LLM that shares an API key; LiteLLMEmbeddings(rate_limit=...) takes one too. Without rate_limit nothing changes. It adds one dependency, pyrate-limiter (pure Python, no dependencies of its own). See Requests and tokens per minute.

EvaluationResult.check(): thresholds for CI (#169)

New: result.check(min_mean=..., max_mean=..., allow_missing=0) raises ThresholdError (an AssertionError, from ragas.exceptions) unless each metric's mean is within its bounds and no row is unscored. The message is a per-metric table of mean, bounds, count and missing rows. A bound for a metric the result doesn't have raises ValueError. See Adding to your CI pipeline.

Copying LLMs, embeddings and metrics (#211)

copy.deepcopy() of an llm_factory() LLM raised RecursionError, and of LiteLLMEmbeddings, of a metric holding either, or of a string-distance metric (NonLLMStringSimilarity, NonLLMContextRecall, ...) raised TypeError: cannot pickle 'module' object. They now copy. A copy has its own settings (model_args, system_prompt, run config, ...) and shares the provider clients, the cache and the RateLimit: a copied LLM goes to the same provider, under the same rate limit.

A copy (deep or shallow) of an LLM or embeddings with a cache used to keep the original's cached generate / embed_text, so its calls ran on the original object, with the original's settings. They now run on the copy.

distance_measure_map on the string-distance metrics and OverlapScoreBuilder is now a read-only property rather than an instance attribute.

Prompt optimizers rank losses by their direction (#216)

Loss now declares its direction with a higher_is_better class attribute: False for MSELoss (an error), True for BinaryMetricLoss (accuracy or F1). Loss.fitness(predicted, actual) returns a value where higher is always better, and both optimizers rank by it.

  • ⚠ silent: GeneticOptimizer kept the prompt with the highest MSE, and passed (actual, predicted) to the loss. It now keeps the best prompt and passes (predicted, actual). Cross-over paired every candidate with the first one, including the first with itself; each candidate now crosses with its nearest other candidate. Optimized prompts will differ.
  • ⚠ silent: the DSPy metric negated every loss, so under BinaryMetricLoss DSPyOptimizer minimised accuracy. It now maximises it. Under MSELoss it still returns the negated error.
  • GeneticOptimizer.evaluate_fitness() now returns fitness values (higher is better), not raw losses.
  • MSELoss(reduction="sum"), BinaryMetricLoss(metric="f1_score") and InstructionConfig(loss=MSELoss()) raised TypeError and now work. An unknown option raises ValueError.
  • A custom Loss that is higher-is-better must set higher_is_better = True; the default is lower-is-better.
  • Each InstructionConfig gets its own copy of optimizer_config, so editing one no longer changes the others.

Collections metric constructors: unknown arguments raise (#216)

  • Any ragas.metrics.collections metric now raises TypeError for a keyword argument it doesn't take. They used to be dropped, so AspectCritic(..., strictnes=3) silently built a metric with strictness=1. Fix the spelling or remove the argument.
  • ContextPrecision(llm, name="x") and ContextUtilization(llm, name="x") raised TypeError: multiple values for keyword argument 'name'. They now accept name; the defaults are unchanged.
  • A custom BaseMetric subclass that doesn't define ascore can no longer be instantiated (TypeError). It used to score 0.0 from a placeholder.
  • AnswerCorrectness(embeddings=...) now checks that the embeddings are a modern BaseRagasEmbedding, as every other collections metric does; anything else raises ValueError. embeddings=None with weights=[1.0, 0.0] still works.

Testset generation: scenarios, filters and personas (#216)

  • SingleHopScenario is now defined once. The class exported from ragas.testset.synthesizers.single_hop was a different class from the one the synthesizer checks for, so passing a scenario you built yourself raised TypeError: scenario type should be SingleHopScenario.
  • ⚠ silent: JaccardSimilarityBuilder now honours filter_nodes in a transform pipeline. It compared every node in the graph, so with filter_nodes set you now get fewer relationships, limited to the nodes you selected. SummaryCosineSimilarityBuilder now links only document nodes in a pipeline, as documented; it linked every node with a summary embedding, chunks included. Neither is in the default transforms.
  • OverlapScoreBuilder().name is now "OverlapScoreBuilder". It was an empty string.
  • ⚠ silent: NodeFilter.transform() now removes every node its filter flags, as the pipeline does. It removed only the first.
  • ⚠ silent: generate_personas_from_kg() now groups similar summaries by true cosine similarity, comparing each summary with its own node's embedding. It used an unnormalised dot product, and when some nodes had no summary it could compare a summary with another node's embedding. You may get different personas from the same graph. A node with a summary but no summary_embedding now raises a clear ValueError.

One deprecation warning, Vertex clients used, LiteLLM batch sizes per provider (#216)

  • evaluate() now emits one DeprecationWarning, pointing at your call. It added a second one from aevaluate() that pointed inside ragas. aevaluate() called directly still warns once.
  • GoogleEmbeddings(use_vertex=True, client=...) now uses the client you pass. A TextEmbeddingModel is called directly, and a google-genai genai.Client(vertexai=True, ...) goes through models.embed_content. Any other client loads model once and reuses it; TextEmbeddingModel.from_pretrained used to run on every call, ignoring the client.
  • ⚠ silent: LiteLLMEmbeddings without batch_size= now picks its batch size from the model's provider: 100 for OpenAI, 96 for Cohere, 5 for Vertex AI, 32 for Hugging Face, and 10 otherwise. Every model got 10. OpenAI and Cohere users see fewer, larger embedding requests. Pass batch_size= to keep the old behaviour.

The LLM takes chat messages, and the multimodal metrics use them (#214)

generate() and agenerate() on llm_factory() LLMs now accept a list of OpenAI-format chat messages as well as a prompt string; message content can be a list of text and image_url parts. A messages list takes the same path as a string: retries, cache, rate limit, tracing, token usage, provider parameter mapping, and sync or async clients. The LLM's system_prompt is prepended unless the list already has a system message. String prompts are sent and cached exactly as before, so existing caches still hit.

MultiModalFaithfulness and MultiModalRelevance (collections) called the provider client directly, so they skipped retries, the cache, tracing, usage recording and system_prompt, and failed with a sync client such as OpenAI(). They now go through llm.agenerate(), so all of these apply and a sync client works. Scores are unchanged, except that a configured system_prompt is now sent and can change the verdict.

One prompt class (#214)

ragas.prompt.metrics.BasePrompt, the prompt class collections metrics use, is now ragas' one structured-prompt implementation, and ragas.prompt.PydanticPrompt is a subclass that only adds support for legacy BaseRagasLLM wrappers. Rendered prompts are byte-for-byte unchanged for both, so cache keys and scores don't move.

  • BasePrompt gains what PydanticPrompt had: generate() / generate_multiple() through an llm_factory() LLM (with n concurrent samples, process_input / process_output hooks and a traced prompt run), name, content-based == and hash(), and save() / load().
  • ⚠ silent: PydanticPrompt.adapt() with an llm_factory() LLM now translates with the same prompt collections metrics use, so translations of adapted testset prompts can differ. With a legacy LLM it still uses TranslateStatements.
  • BasePrompt.adapt(..., adapt_instruction=True) now translates the instruction even when the prompt has no example strings. It returned without translating it.
  • save() logs the saved path at INFO instead of printing it.

evaluate() runs modern metrics natively (#214)

evaluate() and aevaluate() no longer wrap ragas.metrics.collections metrics as legacy metrics. They call the metric's ascore() with the sample's fields by name. Scores, reasons, traces and token usage are the same as before.

Metrics that take the LLM per call now work in evaluate(): NumericMetric (and other prompt-template metrics) and @numeric_metric-decorated functions. They get the llm passed to evaluate(), or the default LLM if none was passed; every row used to score NaN. Passing a legacy BaseRagasLLM to such a metric now raises TypeError before the run starts, and a decorated function whose required parameters are not sample fields raises ValueError up front instead of scoring NaN on every row.

⚠ silent: evaluate(metrics=None) now runs the collections AnswerRelevancy, ContextPrecision, Faithfulness and ContextRecall, built with the run's LLM and embeddings, instead of the ragas.metrics hand-off classes. Scores should not change, since the hand-off already scored through collections. With a legacy LLM or legacy embeddings you still get the old defaults.

Prompt optimization moves to collections metrics (#214)

metric.train(), GeneticOptimizer and DSPyOptimizer now tune the metrics in ragas.metrics.collections. The legacy ragas.metrics classes are being removed (#215), and their train() is removed with this change. InstructionConfig and DemonstrationConfig now take only llm_factory() LLMs and embedding_factory() embeddings, and raise a ValidationError for legacy wrappers.

  • Collections metrics gain get_prompts() and set_prompts(**prompts). Prompts are keyed by attribute name (prompt for AspectCritic; statement_generator_prompt and nli_statement_prompt for Faithfulness). get_prompts() returns copies, so to change a prompt you edit the copy and pass it to set_prompts(). set_prompts() copies what it is given, rejects unknown names with ValueError, and rejects a non-BasePrompt, or a prompt with different input or output models, with TypeError.
  • Annotation files key each sample's prompts by those names. Legacy names such as single_turn_aspect_critic_prompt raise a ValueError that lists the metric's names. The sample file for the Vertex AI alignment guide uses the new names.
  • metric.train(path, demonstration_config, instruction_config, *, dataset=None, run_config=None, batch_size=None, raise_exceptions=True) also accepts the annotations as an object through dataset=. It no longer takes callbacks or with_debugging_logs. A config with enabled=False is now skipped; it used to run anyway.
  • ⚠ silent: train() no longer modifies the metric's prompt objects in place. Optimized prompts replace them through set_prompts(), and candidates are scored on a copy of the metric. In-place edits used to reach every metric sharing a prompt object.
  • Optimizer.optimize() now returns optimized copies of the prompts ({name: BasePrompt}) for set_prompts(), rather than {name: instruction}. DSPyOptimizer's copies also carry the demonstrations MIPROv2 chose, which it used to drop.
  • Each collections metric has an output_type, which picks the default loss: binary for AspectCritic, AgentGoalAccuracyWithReference/WithoutReference and SQLSemanticEquivalence; discrete for the rubric metrics and SimpleCriteriaScore; continuous otherwise.
  • Few-shot demonstrations (DemonstrationConfig) make each annotated prompt a copy of itself that, per input, renders the top_k annotated examples at least threshold similar to it. technique="random" now works: a fixed random sample of top_k examples, with no embedding calls. It was ignored.
  • DSPyOptimizer now drives DSPy through a dspy.LM built from the LLM's LiteLLM model name and credentials, rather than wrapping a legacy LLM. Pass DSPyOptimizer(lm=dspy.LM(...)) for an LLM over a litellm.Router or an Azure client, which raise ValueError. DSPy's own cache and retries apply to its calls, and llm_factory(rate_limit=...) doesn't.
  • DSPyOptimizer failed with every current DSPy: MIPROv2 takes its metric in the constructor and rejects num_candidates with an auto level. It now passes them that way, runs num_trials (from optimizer_config, default num_candidates) when auto=None, and doesn't stop to ask for confirmation. DSPy signatures now carry the prompt fields' types, and only prompts with annotations are optimized.
  • ⚠ silent: DSPy's metric compared an output field named after the metric, which no prompt has, so every prediction scored 0. It now compares that field if present, else the prompt's first numeric output field, such as verdict.
  • GeneticOptimizer's feedback mutation reads each prompt's input and output from the traced LLM calls of the metric. A custom LLM that doesn't trace its calls gets no feedback, with a warning, and keeps the candidate's instruction.

Testset generation takes modern LLMs, embeddings and prompts only (#214)

Testset generation (TestsetGenerator, the synthesizers, the transforms, default_transforms() and generate_personas_from_kg()) now takes only an llm_factory() LLM and a modern embedding model (BaseRagasEmbedding, such as embedding_factory() output). A legacy BaseRagasLLM or BaseRagasEmbeddings wrapper raises TypeError when you build the object or call the function, with a message that names what to use instead.

  • TestsetGenerator.from_llama_index() and generate_with_llamaindex_docs() are removed. Build the generator with llm_factory() and embedding_factory(), and pass LlamaIndex documents to generate_with_docs() with document_adapter=lambda d: Document(page_content=d.text, metadata=d.metadata). See LlamaIndex.
  • New: generate_with_docs() and generate_with_chunks() take document_adapter=, a function that turns each of your documents into one with page_content and metadata, so documents from any framework work without ragas depending on it. generate_with_docs() also accepts plain strings now, as generate_with_chunks() did. generate_with_langchain_docs() stays as a deprecated alias of generate_with_docs().
  • ⚠ silent: generate_with_docs() now skips documents with no text, as generate_with_chunks() and the removed LlamaIndex method did. Blank documents used to become empty graph nodes and counted toward the document-length shares that pick the default transforms, so a corpus with blank documents can now get a different pipeline.
  • The testset prompts (extractors, filters, synthesizers, personas) are now ragas.prompt.metrics.BasePrompt subclasses, not PydanticPrompt. Their rendered text is byte-for-byte unchanged, so cache keys don't move. Their generate() no longer takes stop= or retries_left=. get_prompts() and set_prompts() on testset classes find and take any BasePrompt, including a PydanticPrompt.
  • EmbeddingExtractor always calls aembed_text(). With an embedding model that had no is_async attribute, such as LiteLLMEmbeddings, it ran the sync embed_text() on a worker thread and warned (UserWarning); it now makes the async call and doesn't warn. The embeddings are the same.

The dspy extra installs DSPy 3 with optuna

pip install "ragas[dspy]" now installs dspy[optuna]>=3 instead of dspy-ai>=2.4.0. DSPyOptimizer runs MIPROv2, which on DSPy 3 needs optuna; the old extra left it out, so the optimizer failed on import of a fresh install. DSPy 2.x is no longer supported: DSPyOptimizer already needed DSPy 3's MIPROv2 constructor (#214).

Dead code removed: ragas evals, as_completed, find_indirect_clusters (#215)

These had no callers in ragas, and the CLI command could not run.

  • The ragas evals CLI command is removed. It called a run_async method that experiments don't have and a Project class that no longer exists, so every run failed. Run an experiment from Python instead: Dataset.load(...), then my_experiment.run(dataset) or await my_experiment.arun(dataset). ragas quickstart and ragas hello-world are unchanged, except that the hello_world/evals.py script that ragas hello-world writes is now a working @experiment you run with python hello_world/evals.py. It used to be meant for ragas evals and could not run either.
  • ragas.async_utils.as_completed() and ragas.async_utils.process_futures() are removed. Use concurrent_results(), which ragas has used since #207 and which cancels what is still running when you stop iterating, or asyncio.as_completed().
  • KnowledgeGraph.find_indirect_clusters() is removed. Testset generation uses find_n_indirect_clusters(), which stays. With it goes ragas' last use of scikit-network (its Leiden clustering) and of networkx.
  • The module ragas.backends.utils is removed. Its MemorableNames was a copy of ragas.utils.MemorableNames; import it from there.

The legacy layer is removed (#215)

The legacy metrics, LLM and embedding wrappers and prompt classes are deleted, not deprecated. Everything they did has a modern replacement, which #214 moved testset generation, the optimizers and evaluate() onto.

Removed Use instead
The legacy metrics: from ragas.metrics import Faithfulness, faithfulness, RubricsScore, …, and the private modules ragas.metrics._* from ragas.metrics.collections import Faithfulness; Faithfulness(llm=llm). Collections metrics take their models at construction and score with await metric.ascore(user_input=..., response=...).
Renamed metrics LLMContextRecall → ContextRecall, ResponseRelevancy → AnswerRelevancy, RubricsScore → DomainSpecificRubrics, AnswerSimilarity → SemanticSimilarity, ChrfScore → CHRFScore, InstanceRubrics → InstanceSpecificRubrics, LLMContextPrecisionWith[out]Reference → ContextPrecisionWith[out]Reference, LLMSQLEquivalence → SQLSemanticEquivalence, SummarizationScore → SummaryScore, TopicAdherenceScore → TopicAdherence
ragas.metrics.quoted_spans ragas.metrics.collections.QuotedSpansAlignment
Metric, MetricWithLLM, MetricWithEmbeddings, SingleTurnMetric, MultiTurnMetric, ModeMetric, ensembler (ragas.metrics.base) Subclass ragas.metrics.collections.BaseMetric, or use DiscreteMetric / NumericMetric / RankingMetric
BaseRagasLLM, LlamaIndexLLMWrapper, HaystackLLMWrapper, OCIGenAIWrapper / oci_genai_factory, LiteLLMStructuredLLM, the ragas.llms.adapters package, ragas.llms.output (LLMResult, Generation) llm_factory(), which reaches providers through LiteLLM model prefixes (e.g. llm_factory("anthropic/..."))
BaseRagasEmbeddings, HuggingfaceEmbeddings, LlamaIndexEmbeddingsWrapper, HaystackEmbeddingsWrapper BaseRagasEmbedding and its providers: embedding_factory(), HuggingFaceEmbeddings, LiteLLMEmbeddings, OpenAIEmbeddings, GoogleEmbeddings
ragas.run_config.add_retry / add_async_retry, which retried every exception, a 400 included llm_factory() LLMs retry transient errors per RunConfig themselves
PydanticPrompt, StringPrompt, the legacy ragas.prompt.base.BasePrompt, FewShotPydanticPrompt, ExampleStore / InMemoryExampleStore, ImageTextPrompt / ImageTextPromptValue, RagasOutputParser, FixOutputFormat, TranslateStatements, ragas.prompt.value, ragas.prompt.utils.extract_json ragas.prompt.BasePrompt, which is now ragas.prompt.metrics.BasePrompt (subclass it the way you subclassed PydanticPrompt); DynamicFewShotPrompt for few-shot examples; messages-based llm.agenerate for images
ragas.validation.validate_required_columns / validate_supported_metrics, ragas.utils.get_required_columns_v1, get_metric_language, DeprecationHelper Nothing: they only served the legacy metrics
The ai-frameworks (Haystack) and oci extras llm_factory() over LiteLLM
  • from ragas.metrics import <old name> raises ImportError with the replacement import, e.g. 'RubricsScore' was removed from 'ragas.metrics' along with the legacy metrics. Use: from ragas.metrics.collections import DomainSpecificRubrics. It is a hint, not a shim: nothing is re-exported. DiscreteMetric, NumericMetric, RankingMetric, the *_metric decorators, MetricResult, BaseMetric, LLMMetric, MetricType and MetricOutputType stay in ragas.metrics.
  • evaluate() and aevaluate() take only SimpleBaseMetric metrics (collections metrics and the DiscreteMetric family), an llm_factory() LLM and a BaseRagasEmbedding; anything else raises TypeError before the run starts. evaluate(metrics=None) always runs the collections defaults.
  • ragas.integrations.llama_index.evaluate() no longer wraps LlamaIndex LLMs and embeddings. Its llm and embeddings are now ragas models (llm_factory(), embedding_factory()), and its metrics are collections metrics.
  • PromptMixin.get_prompts() / set_prompts() (testset synthesizers and transforms) find and take any ragas.prompt.BasePrompt; the testset-only override is folded into it.
  • ragas.metrics._collections_bridge is gone; the sample-to-ascore() mapping evaluate() uses is ragas.metrics.sample_inputs.
  • DistanceMeasure for OverlapScoreBuilder comes from ragas.metrics.collections; the values are unchanged.

Multimodal metrics restrict where context images are loaded from (#229)

MultiModalFaithfulness and MultiModalRelevance load the images named in retrieved_contexts. They used to open any local image path and fetch any http(s) URL, following redirects to any host, internal ones included. Upstream's checks for this (vibrantlabsai/ragas#1991) were only in the legacy ImageTextPrompt. They now apply to both metrics, configured through the new ragas.metrics.collections.ImageSourcePolicy, passed as image_policy=:

  • Local image paths are no longer read by default; such a context raises ImageLoadError (#231). To read local images, pass image_policy=ImageSourcePolicy(local_dir="images/"). Paths are resolved against that directory, and a path that resolves outside it is refused, symlinks included.
  • A URL whose host resolves to a non-public address (private, loopback, link-local, shared, reserved or multicast, IPv4 or IPv6) is not fetched, and neither is a redirect to one; it raises ImageLoadError (#231). For an internal image server, pass ImageSourcePolicy(allow_private_urls=True).
  • URLs with credentials, a backslash in the host part, or a host that urllib and requests parse differently are refused. At most 5 redirects are followed.
  • The size limit (10 MB) is now ImageSourcePolicy(max_bytes=...).
  • Data URIs are unchanged.
  • The image helpers process_image_to_base64() and is_image_path_or_url() moved from the multi_modal_faithfulness.util and multi_modal_relevance.util modules to ragas.metrics.collections.image_sources. process_image_to_base64() takes the policy as its second argument.

Multimodal metrics raise when an image context can't be loaded (#231)

MultiModalFaithfulness and MultiModalRelevance used to send an image reference they could not load to the judge as text, such as Context 1: path/to/tesla.jpg, and still return a score. That covered missing files, failed downloads, non-image content, and everything #229 refuses. The judge never saw the image, so the score looked valid but was not.

  • An image reference that is refused or fails to load now raises ragas.exceptions.ImageLoadError, a ValueError, before the LLM is called. The message gives the context's position, the reason, and the setting that would allow it. Under evaluate() this follows raise_exceptions: it is raised, or the score is NaN and the error is logged.
  • ImageSourcePolicy(on_blocked="warn") restores the old fallback: a warning is logged and the reference is sent as text.
  • An image reference is an image data URI, an http(s) URL, or a path ending in an image extension (.jpg, .jpeg, .png, .gif, .webp, .bmp). Any other context is text, as before.
  • ⚠ silent: under local_dir, a file without an image extension is no longer opened. It is sent as text, even if it holds an image.
  • Other URL schemes, such as ftp://, raise instead of being treated as file paths.

Custom backends implement _load, _save and _list (#217)

BaseBackend used to have six abstract methods (load_dataset, load_experiment, save_dataset, save_experiment, list_datasets, list_experiments), and every backend wrote the same six one-line wrappers. Now the six are concrete, and a backend implements three abstract methods that take the store, kind ("datasets" or "experiments"):

  • _load(kind, name), _save(kind, name, data, data_model) and _list(kind). A custom backend that only defines the six public methods can no longer be instantiated (TypeError: Can't instantiate abstract class); rename them to these three.
  • The new ragas.backends.FileBackend(root_dir) is the base for one-file-per-name backends: set suffix and implement _read(path) and _write(path, data). LocalCSVBackend and LocalJSONLBackend are built on it.
  • The private helpers _get_data_dir(), _get_file_path() (now FileBackend._path(kind, name)) and LocalJSONLBackend._serialize_datetime() are gone, as are GDriveBackend's _load_data_from_spreadsheet(), _save_data_to_spreadsheet() and _list_data_names().
  • The registry names, on-disk formats and error messages are unchanged. See Storage backends.

run_async_tasks() returns results in order and runs on Executor (#217)

ragas.async_utils.run_async_tasks() had its own copy of Executor's batching, progress-bar and cancellation loop. It is now a thin wrapper over Executor, and @experiment's arun() runs its rows through concurrent_results() instead of its own semaphore-and-as_completed loop.

  • run_async_tasks() returns results in the order of tasks; they used to come back in completion order. Errors are unchanged: with fail_fast=True the first failure cancels the rest and is raised; without it every task runs and the first error to happen is raised at the end.
  • Its unused keyword argument cancel_check is removed. To stop a run early, use Executor and call executor.cancel().
  • arun()'s max_workers, raise_exceptions, save_partial, ordering and cancellation behave as before.

EvaluationDataset: one rule for sample types, from_dict takes columns (#217)

Which sample class a row becomes was decided in three places. It is now ragas.dataset_schema.sample_type_for(row): a row whose user_input is a list (of messages) is a MultiTurnSample, any other row a SingleTurnSample. SAMPLE_TYPES lists the two classes.

  • EvaluationDataset.from_list() decides per row. It used to look only at the first row's user_input and build every row as that type. A list mixing single- and multi-turn rows now raises ValueError: Sample at index N is of type ...; it used to raise a pydantic validation error on the first row that didn't fit.
  • EvaluationDataset.from_dict() takes columns, {"user_input": [...], "response": [...]}, as its Dict annotation said, and calls from_list(). It used to iterate its argument as a list of rows, so only a list worked and a real dict raised TypeError. Pass a list of rows to from_list(); from_dict() raises TypeError for one.
  • EvaluationDataset.backend and .name, and from_list()'s backend= and name= arguments, are removed. Nothing read them. For a named dataset saved to a backend, use ragas.Dataset.
  • ragas.validation.get_supported_metric_type() is removed. evaluate() reports an unsupported sample class as The metric '<name>' does not support the sample type <class>, instead of Unsupported sample type <class>.

Names moved out of ragas.utils (#217)

ragas.utils is split up. Nothing is re-exported from the old location:

Was Now
ragas.utils.MemorableNames, memorable_names, create_nano_id ragas.names
ragas.utils.async_to_sync ragas.async_utils.async_to_sync
ragas.utils.convert_v1_to_v2_dataset, REQUIRED_COLS_v1 ragas.validation
ragas.utils.convert_row_v1_to_v2, convert_v2_to_v1_dataset Removed: nothing used them

Dataset.train_test_split() no longer shuffles the dataset or the global random (#217)

With random_state set, train_test_split() called random.seed(random_state) and shuffled the dataset's own rows in place. Now it shuffles a copy with random.Random(random_state).

  • For a given seed and dataset, the same rows land in each split as before.
  • The dataset keeps its row order, and the global random state is not reseeded. Code that relied on train_test_split() seeding random for later calls must seed it itself.
  • Calling it twice with the same seed now gives the same split. It used to shuffle the already-shuffled rows again, so the second call gave a different split.
  • With random_state=None the rows are not shuffled, as before: the last test_size of them are the test split.

ragas.llms.base is split into modules (#217)

llm_factory(), InstructorLLM, InstructorBaseRagasLLM, InstructorModelArgs, DEFAULT_LLM_MODEL and default_llm import from ragas.llms and ragas.llms.base as before. Parameter mapping, cache keys and scores are unchanged.

  • is_reasoning_model moved to ragas.llms.provider_rules and is exported from ragas.llms: use from ragas.llms import is_reasoning_model. from ragas.llms.base import is_reasoning_model raises ImportError.
  • The per-provider parameter rules are ragas.llms.provider_rules.PROVIDER_RULES, a list of rule objects where the first match wins. The InstructorLLM._map_google_params, _map_anthropic_params, _map_openai_params and _map_litellm_defaults methods are gone; _map_provider_params still applies them. Claude's sampling warning is logged by ragas.llms.provider_rules.
  • InstructorLLM(..., is_async=...) says whether client is async. llm_factory() sets it. Without it, InstructorLLM checks whether client.chat.completions.create is a coroutine function, which holds for instructor's AsyncInstructor. It no longer looks inside instructor's closures or at client.client. If you construct InstructorLLM around your own client and it is detected wrongly, pass is_async=.
  • With provider="google", InstructorLLM calls client.chat.completions.create, like every other provider, instead of client.create. On an instructor client these are the same method.

DiscreteMetric, NumericMetric and RankingMetric: one scoring path and range checks (#217)

These metrics, and SimpleLLMMetric (LLMMetric) subclasses, were refactored. Saving, loading and alignment moved to their own modules; the public methods stay.

  • ⚠ silent: score() now runs ascore(), so it calls the LLM's agenerate(), never generate(). llm_factory() LLMs support both, with sync or async clients, so most code is unaffected. A custom LLM object needs agenerate(). result.traces["input"] is now the formatted prompt from score() too; it used to hold the keyword arguments.
  • ascore() and score() check the LLM's value against allowed_values and raise ValueError when it is outside them. That means a NumericMetric value outside its range, or a RankingMetric list of the wrong length. Before, the value was returned as is. Under evaluate() and @experiment this follows raise_exceptions, like any other metric error: it is raised, or the row has no score and the error is logged. DiscreteMetric values were already restricted by its response model. See Score changes.
  • A metric with no prompt raises ValueError instead of a bare Exception.
  • load() checks the saved metric type in the base class, so LLMMetric.load() and a custom subclass's load() reject another class's file too. The DiscreteMetric/NumericMetric/RankingMetric overrides are unchanged. Files saved by earlier versions load as before.
  • A NumericMetric with allowed_values=range(...) can now be saved. It used to raise TypeError. The range is written as {"range": [start, stop, step]}, which earlier versions can't load.
  • align() and align_and_validate() take max_similar_examples and similarity_threshold as named arguments. Other keyword arguments raise TypeError; they were ignored before. String values are no longer converted to numbers. The aligner itself is ragas.metrics.alignment.Aligner.
  • validate_alignment() pairs each label with its prediction before dropping rows where either is missing. Before, missing labels and missing predictions were dropped from separate lists, so after a gap the labels were compared with other rows' predictions. correlation and agreement_rate change for test sets with missing labels or predictions. A test set with no complete row raises ValueError instead of ZeroDivisionError. Its mapping argument defaults to None.
  • Persistence lives in ragas.metrics.serialization. The private SimpleLLMMetric helpers _serialize_prompt, _deserialize_prompt, _get_metric_config, _serialize_response_model_info, _read_metric_type and _validate_metric_type are gone.
  • Prompt, DynamicFewShotPrompt and metric save()/load() share one gzip-aware JSON helper (ragas.prompt.utils.read_json_file/write_json_file). A file that is not valid UTF-8 now raises ValueError("Cannot load ..."), like other unreadable files, instead of UnicodeDecodeError.

AG-UI: one tool-argument parser and a separate SSE client (#217)

  • Streamed (ToolCallStart/Args/End) and chunked (ToolCallChunk) tool calls now parse their arguments the same way as MessagesSnapshot tool calls. Arguments that are valid JSON but not an object, such as [1, 2] or 5, are kept as {"raw_args": "<the text>"} with a warning. They used to fail ToolCall validation and stop the conversion. Invalid JSON is kept as raw_args, as before.
  • With metadata=True, a ToolMessage now gets the current step_name, like text messages.
  • The SSE client moved to ragas.integrations.sse.post_sse_json, and call_ag_ui_endpoint() uses it. It also accepts data: lines with no space after the colon, as the SSE spec allows. Without httpx, the ImportError now comes from the SSE client.

Testset generation and optimizers: one code path per job (#217)

TestsetGenerator's three entry points now share one graph builder and one check of their inputs.

  • generate_with_docs() and generate_with_chunks() take num_personas and batch_size, as generate() does. They used to always use 3 personas and no batching.
  • An LLM and an embedding model are needed only to build the default transforms. With your own transforms, generate_with_docs() and generate_with_chunks() no longer raise ValueError when the generator has no embedding model.
  • generate_with_docs(transforms=[]) applies no transforms, as generate_with_chunks(transforms=[]) already did. It used to replace an empty list with the default transforms.
  • BaseSynthesizer.is_applicable(kg) says whether a synthesizer can work on a knowledge graph. It returns True unless overridden. default_query_distribution(kg=...) calls it, where it used to call each synthesizer's get_node_clusters(). The three built-in synthesizers give the same answers as before. The shared theme extraction, persona matching, combination sampling and sample generation moved to BaseSynthesizer helpers. The generated testsets are unchanged.
  • The seven LLM extractors (SummaryExtractor, TitleExtractor, TopicDescriptionExtractor, KeyphrasesExtractor, HeadlinesExtractor, NERExtractor, ThemesExtractor) are built on two shared bases. Their fields, prompts and outputs are unchanged. LLMBasedExtractor no longer has the merge_if_possible field, which nothing read. Passing it raises TypeError.
  • CosineSimilarityBuilder uses the base RelationshipBuilder execution plan, as the other relationship builders do. A node without an embedding, or embeddings of different lengths, now raise ValueError when the transform runs, not when apply_transforms() builds the plan. The relationships built are unchanged.
  • KnowledgeGraph.find_n_indirect_clusters() and KnowledgeGraph.find_two_nodes_single_rel() moved to ragas.testset.graph_queries as functions that take the graph first: find_n_indirect_clusters(kg, n, ...) and find_two_nodes_single_rel(kg, ...). The methods are removed. The results are unchanged.
  • Optimizer.optimize(), GeneticOptimizer.optimize() and DSPyOptimizer.optimize() are now optimize(dataset, loss, config=None, *, run_config=None, batch_size=None, callbacks=None, raise_exceptions=True). config is optional, and the run options are keyword-only. The with_debugging_logs argument, which no optimizer read, is removed. metric.train() is unchanged.
  • GeneticOptimizer checks its metric and llm once, when optimize() starts. Its steps no longer check them again.

Shared metric helpers (#217)

The collections metrics now share helpers instead of carrying copies. Metric class names, constructor arguments and ascore() signatures are unchanged. Score effects are in Metric score changes.

Custom metrics declare their components. ragas.metrics.collections.BaseMetric validates the components a metric lists in required_components (and, if not None, those in optional_components), and takes them as keyword arguments:

class MyMetric(BaseMetric):
    required_components = ("llm",)

    def __init__(self, llm, name="my_metric", **kwargs):
        super().__init__(llm=llm, name=name, **kwargs)
  • It used to validate any llm / embeddings attribute that happened to be set before super().__init__(). A subclass that still sets self.llm and doesn't declare it is no longer validated. Declare it.
  • Passing llm= or embeddings= to a metric that doesn't declare it raises TypeError, like any other unknown keyword argument.

One wording for a missing input. Every collections metric raises ValueError("<field> is missing. Please add <field> to the test sample.") for a missing or empty required input. Before, some said "<field> cannot be empty" (ContextPrecision, ContextRecall, AnswerRelevancy), "Please provide ..." (the multimodal metrics), "reference must be provided for AgentGoalAccuracyWithReference" or "rubrics must be provided for instance-specific evaluation". Code matching the old messages must match the new one. Type checks (e.g. "user_input must be a list of messages") and whitespace checks are unchanged.

The NVIDIA dual-judge metrics run both judges at once. AnswerAccuracy, ResponseGroundedness and ContextRelevance send their two judge requests concurrently, so a sample has two LLM calls in flight. When the provider raises, both calls have been made. A retry after an off-scale rating uses temperature 0.3 and its own sample_index (see score changes).

FaithfulnesswithHHEM inherits ascore(). Faithfulness.ascore() checks inputs, extracts statements, then calls _verify_statements(statements, context_str). HHEM overrides only that hook. A subclass of either that verifies statements differently should override the hook too.

Moved or removed names:

Removed Use instead
StatementGeneratorInput / Output / Prompt in collections.answer_correctness.util the same names in ragas.metrics.collections.faithfulness.util
NLIStatementInput / Output / Prompt, StatementFaithfulnessAnswer in collections.factual_correctness.util the same names in ragas.metrics.collections.faithfulness.util
the module collections.noise_sensitivity.util (StatementFaithfulnessInput / Output / Prompt aliases) NLIStatementInput / Output / Prompt in ragas.metrics.collections.faithfulness.util
collections.tool_call_f1.util.calculate_f1_score ragas.metrics.utils.fbeta_score
ragas.testset.transforms.relationship_builders.cosine.cosine_similarity_matrix ragas.similarity.cosine_matrix

The private methods _calculate_average_precision (ContextPrecision), _format_conversation (AgentGoalAccuracy, TopicAdherence), _compare_outcomes (AgentGoalAccuracy) and _get_judge_rating / _average_scores (the dual-judge metrics) are gone. ContextPrecision no longer logs "Invalid response format ...": the warning could not fire.

MLflowTrace.get_url() checks MLFLOW_HOST and escapes trace IDs (#243)

  • MLFLOW_HOST must be an http(s) URL with a host, with no query string or fragment; anything else raises ValueError. A port and a path prefix (https://example.com/mlflow) still work, and the URL for a valid host is unchanged.
  • The experiment ID and trace ID are percent-encoded, so an ID containing /, ?, & or # can no longer add path segments or query parameters to the URL.

Published to PyPI as ragas-ng (#242)

  • The distributions are renamed: ragas → ragas-ng, and ragas-examples → ragas-ng-examples. The import names are unchanged: import ragas and import ragas_examples. Extras keep their names: pip install "ragas-ng[all]".
  • Versions use calendar versioning, YYYY.M.PATCH (for example 2026.11.0), so they can't be confused with upstream's 0.x releases. A ragas-ng version says nothing about which upstream release it matches. Pin it like any other version: ragas-ng>=2026.11.
  • pip install ragas still installs upstream ragas. Both distributions provide the ragas import package, so don't install both in one environment.
  • The quickstart templates' pyproject.toml files depend on ragas-ng[all] (they had ragas[all]>=0.3.0, which resolved to upstream).
  • Install hints in error messages and the docs say ragas-ng[...]. The ragas[examples] hint, an extra that never existed here, is now the ragas-ng-examples package.
  • ragas.experimental no longer lazily imports upstream's separate ragas_experimental package. It raises AttributeError, like any other missing attribute.

Decorator metrics raise instead of returning value=None (#239)

Metrics built with @discrete_metric, @numeric_metric and @ranking_metric now handle errors the way DiscreteMetric / NumericMetric / RankingMetric do.

  • An invalid value raises ValueError. That covers a value not in the allowed values, a number outside the range, or a ranking list of the wrong length. The message is the same one those classes use, e.g. Metric m returned 1.5 but expected value in range (0.0, 1.0). Before, score() / ascore() returned MetricResult(value=None, reason=<that message>).
  • Errors from your function propagate. Before, any exception, including LLM, network and coding errors, came back as MetricResult(value=None, reason="Error executing metric <name>: ...").
  • ⚠ silent: result.value is None no longer detects a failure. Code that checked for it, or that read result.reason for the error, now gets the exception instead. If a failure should count as a score, catch it inside the function and return that score; for example, return "incorrect" when the model's output can't be parsed.
  • Under evaluate() and @experiment these errors follow raise_exceptions, like any other metric error: raised, or the row gets no score (evaluate()) or is left out (@experiment) and the error is logged. Before, raise_exceptions=True could never surface them.
  • The private _validate_result_value() method is gone; use validate_result_value(), which returns the error message or None.

Leftover aliases removed; DSPy run options; seeded annotation sampling (#240)

Removed aliases:

Removed Use instead
ragas.cost.CostCallbackHandler ragas.cost.TokenUsageCollector (the same class)
TestsetGenerator.generate_with_langchain_docs() TestsetGenerator.generate_with_docs() (the same arguments)
SingleMetricAnnotation.train_test_split() none: it only raised NotImplementedError

DSPyOptimizer uses train()'s run options. It used to accept them and ignore them.

  • run_config: its max_retries and timeout become the dspy.LM's num_retries and timeout, unless the llm_factory() arguments set them. train() always passes a RunConfig, so by default DSPy's calls now retry up to 10 times (DSPy's default was 3), with a 180-second timeout. A DSPy LM passed as DSPyOptimizer(lm=...) keeps its own settings.
  • raise_exceptions sets MIPROv2's error limit when DSPyOptimizer(max_errors=...) is unset. With True, train()'s default, DSPy's own limit applies (dspy.settings.max_errors, 10), as before. False now never stops on errors. An explicit max_errors still wins.
  • batch_size and callbacks raise NotImplementedError instead of being ignored. MIPROv2 sizes its own minibatches and traces its own calls.

SingleMetricAnnotation.sample(), batch() and stratified_batches() take seed.

  • They draw from a private random.Random(seed) and never use or change the global random state.
  • ⚠ silent: random.seed(...) no longer makes them reproducible; pass seed= instead. Without a seed, each call draws fresh.
  • ⚠ silent: GeneticOptimizer passes seeds derived from run_config.seed, which train() defaults to 42. The annotations it samples are now the same on every run with the same seed. Before, they followed the global random state.

metrics_experiment, summarize and check (#253)

New, as the first step of replacing evaluate() (#241). Nothing is removed yet.

  • ragas.metrics_experiment(metrics, ...) scores metrics over a Dataset as an experiment. It maps the fields the same way evaluate() does, and runs the same up-front checks. Each row gets a <metric> and a <metric>_reason column. See Experiments.
  • ragas.summarize(exp) and ragas.check(exp, min_mean=..., max_mean=..., allow_missing=...) give the same numbers and the same ThresholdError as EvaluationResult.summary() / check(), for any experiment with metric columns.
  • Differences from evaluate():
  • there are no default metrics and no default OpenAI LLM or embeddings;
  • max_workers counts rows, and a row's metrics run concurrently;
  • token usage goes to a TokenUsageCollector you pass as usage=;
  • traces come from OpenTelemetry spans.

Annotation types moved to ragas.optimizers.annotations; the genetic optimizer scores without evaluate() (#254)

Step 2 of replacing evaluate() (#241).

The annotation types moved. There is no alias at the old path.

Removed Use instead
from ragas.dataset_schema import PromptAnnotation, SampleAnnotation, MetricAnnotation, SingleMetricAnnotation from ragas.optimizers.annotations import ... (the same classes)
SingleMetricAnnotation.to_evaluation_dataset() [s.metric_input for s in annotations], the rows it held

GeneticOptimizer scores candidates directly. Before, each candidate went through evaluate().

  • evaluate_candidate() / aevaluate_candidate(candidate=, eval_dataset=, ...) are now score_candidate() / ascore_candidate(candidate=, rows=, ...). rows are metric_input dicts, and the result is a RowScores with scores (one per row, NaN for a failed row) and traces (each row's traced prompt calls), not an EvaluationResult. The run_id argument is gone: the scoring run nests under callbacks.
  • Scores, traces, fitness ranking and feedback mutation are unchanged.
  • A metric that takes its llm or embeddings per call (such as DiscreteMetric) raises ValueError. Before, evaluate() filled in a default OpenAI model. Optimize a metric that holds its own models.

Testset and the integrations return a Dataset (#255)

Step 3 of replacing evaluate() (#241). Score the results with metrics_experiment.

Removed Use instead
Testset.to_evaluation_dataset() Testset.to_dataset(name, backend="inmemory", **backend_kwargs): a Dataset of copies of the samples, with data_model set to SingleTurnSample or MultiTurnSample. Call save() to write it. A testset that mixes the two raises ValueError.
Testset.to_hf_dataset() / Testset.from_hf_dataset() to_list() / from_list(), or to_jsonl() / from_jsonl()
ragas.integrations.llama_index.evaluate(query_engine, dataset, metrics, ...) evaluate_query_engine(query_engine, dataset, metrics, ...) (aevaluate_query_engine in async code)
  • Save a testset you'll score as JSONL, not CSV. CSV loads a missing text value, such as a sample with no reference, as "". metrics_experiment scores "" as a value instead of reporting the column missing. See What each backend keeps.
  • Testset is no longer a RagasDataset. It keeps to_list, from_list, to_pandas, to_csv, to_jsonl, from_jsonl, from_annotated, total_tokens and total_cost, and iterates and has a length as before. from_pandas, from_dict, features() and get_sample_type() are gone.
  • griptape and r2r: transform_to_ragas_dataset() returns a Dataset of SingleTurnSample rows, not an EvaluationDataset. It's in memory by default; the new keyword arguments name, backend and the backend's own arguments choose where it lives.
  • evaluate_query_engine() takes a Dataset of single-turn rows (samples or dicts) and leaves it unchanged. It runs the engine, then scores the answered rows with metrics_experiment and returns the Experiment: use ragas.summarize() / ragas.check() on it, not EvaluationResult methods.
  • token_usage_parser is replaced by usage=TokenUsageCollector(...).
  • callbacks and in_ci are gone; traces come from OpenTelemetry spans.
  • name, backend and backend_kwargs name the experiment and choose where it's saved (in memory by default).
  • There's no default LLM or embeddings for metrics that take them per call.

evaluate_query_engine(): a failed query scores missing (#264)

⚠ silent. When the query engine failed on a row, the row was still scored with response=None. A metric that accepts a missing response, such as ExactMatch, gave it a real score (0.0), which lowered the mean while summarize() reported nothing missing. The old ragas.integrations.llama_index.evaluate() did the same.

  • Now a row whose query failed scores NaN for every metric, and each <metric>_reason is query engine failed: <ExceptionType>: <message>.
  • summarize() counts the row as missing, and check() fails on it unless allow_missing permits it.
  • If every query fails, the run still completes, with every row missing. Before, the up-front column check raised.
  • With raise_exceptions=True, the first failed query still stops the run.

evaluate(), aevaluate() and EvaluationResult removed (#257)

Step 5 of replacing evaluate() (#241). See From evaluate() to experiments for a before/after of every argument and method.

Removed Use instead
ragas.evaluate() / ragas.aevaluate(), ragas.evaluation ragas.metrics_experiment(metrics).run(dataset) / await ....arun(dataset)
ragas.dataset_schema.EvaluationResult the returned Experiment, with ragas.summarize() and ragas.check()
ragas.validation (remap_column_names, convert_v1_to_v2_dataset, ...) metrics_experiment(..., column_map={...}); rename v1 columns (question, answer, contexts, ground_truth) yourself
ragas.exceptions.ExceptionInRunner nothing: only evaluate() raised it
ragas.embeddings.base.default_embeddings_for_litellm_llm() pass embeddings= to the metrics that need them, e.g. embedding_factory("litellm", model="gemini/gemini-embedding-2")
  • No default metrics, LLM or embeddings. evaluate(metrics=None) ran four default metrics with a default OpenAI LLM and embeddings matched to your LLM. Build the metrics with the models you want.
  • default_llm() stays: testset generation still uses it.
  • ThresholdError stays; ragas.check() raises it.

EvaluationDataset removed; datasets is no longer a dependency (#258)

The last step of replacing evaluate() (#241). See From evaluate() to experiments for each constructor and converter.

Removed Use instead
ragas.EvaluationDataset, ragas.dataset_schema.EvaluationDataset Dataset("name", "inmemory", data_model=SingleTurnSample, data=samples)
ragas.dataset_schema.RagasDataset nothing: Testset (#255) and EvaluationDataset were its only subclasses
EvaluationDataset.from_hf_dataset(hf) / to_hf_dataset() Dataset.from_pandas(hf.to_pandas(), ...) / datasets.Dataset.from_pandas(dataset.to_pandas())

ragas.dataset_schema now holds the sample types (BaseSample, SingleTurnSample, MultiTurnSample) and sample_from_dict / sample_type_for.

Dependencies:

  • Hugging Face datasets is no longer installed with ragas, and neither is pyarrow, which only datasets needed. To load data from the Hub, pip install datasets.
  • pandas is now a direct dependency. It used to arrive only through datasets, and to_pandas() and ragas hello-world need it.
  • huggingface_hub is no longer guaranteed. HuggingFaceEmbeddings(use_api=True) needs pip install huggingface-hub, and says so if it's missing. The local sentence-transformers path brings its own copy.

Quickstart projects install on Python 3.13+; [all] skips r2r there (#270)

  • ragas-ng[all] no longer installs r2r on Python 3.13 or newer. r2r's releases support only Python <3.13 and pin tiktoken<0.9, which has no wheel for 3.14. Because uv ignores upper bounds on requires-python, ragas-ng[all] failed to install there. The r2r integration (ragas.integrations.r2r) doesn't import r2r, so it still works. If you need the r2r client itself, use Python 3.12 or older.
  • The rag_eval and improve_rag quickstart templates depend on what they import instead of ragas-ng[all]. improve_rag no longer lists LangChain packages, which it never imported, and now lists openai-agents, which its agentic mode needs. A project generated before this change can replace ragas-ng[all] in its pyproject.toml with the new list.
  • The ragas-ng-examples package ships each template's pyproject.toml. In 2026.11.0 it didn't, so ragas quickstart with ragas-ng-examples installed created a project with nothing for uv sync to install. Without ragas-ng-examples, the CLI downloads templates from GitHub and wasn't affected.