Scoring a dataset
metrics_experiment() scores metrics over a Dataset as an experiment; summarize() and check() read the per-metric means and gate CI on them. They replace evaluate(): see From evaluate() to experiments.
from ragas import check, metrics_experiment, summarize
run = metrics_experiment([Faithfulness(llm=llm)])
exp = run.run(dataset, name="nightly") # await run.arun(...) in async code
summarize(exp) # {'faithfulness': {'mean': 0.86, 'count': 48, 'missing': 2}}
check(exp, min_mean={"faithfulness": 0.8})
metrics_experiment()
An experiment that scores metrics on every row of a dataset.
Run it like any experiment: await run.arun(dataset, name=...), or
run.run(...) from sync code. Each saved row is the dataset row plus,
per metric, <name> (the score; NaN if the metric raised) and
<name>_reason (its explanation, or None). Use :func:summarize and
:func:check on the result.
Rows may be SingleTurnSample / MultiTurnSample objects (a
Dataset with data_model=SingleTurnSample), dicts or other pydantic
models; dicts become samples (multi-turn when user_input is a list).
All rows must be the same sample type. Before any row runs, a required
input that no row has, or a metric that can't score the sample type,
raises ValueError.
Only None counts as missing: a field set to "" is passed to the
metric and scored. The CSV and Google Sheets backends load a missing text
value as "", so a row without a reference that went through CSV is
scored against an empty one instead of raising. Keep datasets to be
scored in local/jsonl or inmemory, which keep None.
Args:
metrics: Metric objects (ragas.metrics.collections, or any
SimpleBaseMetric). Their names name the columns.
llm: For metrics whose ascore() takes llm per call (e.g.
DiscreteMetric). Collections metrics take theirs at
construction. There is no default LLM.
embeddings: The same, for embeddings.
column_map: {sample_field: dataset_column}, for a dataset whose
columns are named differently, e.g. {"user_input": "question"}.
run_config: Timeouts and retries for the metrics' LLM calls; its
timeout also bounds each metric call. Default: each LLM's own.
usage: A TokenUsageCollector that records the run's token usage
(usage.total_tokens() / total_cost(...)).
backend, backend_kwargs, name_prefix, max_workers, raise_exceptions,
save_partial: As for @experiment. max_workers counts rows;
a row's metrics run concurrently. With raise_exceptions, the
first metric that raises stops the run; otherwise it scores NaN
and the error is logged.
Raises: TypeError: metrics that aren't metric objects, or an llm/embeddings ragas can't call. ValueError: duplicate metric names, a metric no sample type can drive, or a per-call model that wasn't passed.
Source code in src/ragas/scoring.py
MetricsExperiment
Bases: ExperimentWrapper
An experiment that scores metrics on each row: see :func:metrics_experiment.
Source code in src/ragas/scoring.py
arun
async
arun(
dataset: Any,
name: Optional[str] = None,
backend: Optional[Union[BaseBackend, str]] = None,
*args: Any,
backend_kwargs: Optional[Dict[str, Any]] = None,
max_workers: Optional[int] = None,
raise_exceptions: Optional[bool] = None,
save_partial: Optional[bool] = None,
**kwargs: Any,
) -> Experiment
Score every row of dataset; see :meth:ExperimentWrapper.arun.
A metric that raises on a row scores NaN there, with the error logged,
and the row is kept. With raise_exceptions the run stops on it.
Source code in src/ragas/scoring.py
run
run(
dataset: Dataset,
name: Optional[str] = None,
backend: Optional[Union[BaseBackend, str]] = None,
*args,
backend_kwargs: Optional[Dict[str, Any]] = None,
max_workers: Optional[int] = None,
raise_exceptions: Optional[bool] = None,
save_partial: Optional[bool] = None,
**kwargs,
) -> Experiment
Run the experiment from synchronous code: arun, without await.
Takes the same arguments as :meth:arun and returns the same
Experiment. Like metric.score(), it runs on
ragas' shared background event loop (ragas.async_utils.run_sync),
so it works in a plain script and inside a running loop (Jupyter)
alike. Ctrl-C cancels the rows still running. In async code, prefer
await arun(...), which doesn't block the event loop.
Source code in src/ragas/experiment.py
summarize()
Per-metric mean with the number of rows it was computed over.
mean excludes rows without a score (None or NaN): rows whose metric
raised and rows where the score is undefined, e.g. no claims to judge.
count is how many rows the mean covers and missing how many it
doesn't::
>>> summarize(exp)
{'faithfulness': {'mean': 0.86, 'count': 48, 'missing': 2}}
Args:
experiment: An Experiment (or any rows) with metric columns,
including one reloaded from a backend.
metrics: Metric names or objects to summarize. Default: every column
metrics_experiment wrote (a <name> column with a
<name>_reason column beside it).
Source code in src/ragas/scoring.py
check()
Raise ThresholdError unless every metric meets its thresholds.
The one-call CI check (#169)::
check(exp, min_mean={"answer_relevancy": 0.9, "faithfulness": 0.3})
# a range: both bounds
check(exp, min_mean={"faithfulness": 0.3}, max_mean={"faithfulness": 0.5})
Args:
experiment: An Experiment with metric columns (see :func:summarize).
min_mean: Lowest acceptable mean, per metric ({name: bound}) or
one number for every metric.
max_mean: Highest acceptable mean, the same way.
allow_missing: How many rows per metric may be without a score
(errored, or undefined). The default, 0, fails on any: a missing
row is left out of the mean, so the mean alone would hide it.
None doesn't check.
metrics: The metrics to check, as for :func:summarize.
A metric named in a bound but not in the results is an error, so a typo can't pass silently. A metric with no scored row fails any bound.
Raises:
ThresholdError: an AssertionError whose message is a table of
every checked metric's mean, bounds, count and missing rows.