Skip to content

Scoring a dataset

metrics_experiment() scores metrics over a Dataset as an experiment; summarize() and check() read the per-metric means and gate CI on them. They replace evaluate(): see From evaluate() to experiments.

from ragas import check, metrics_experiment, summarize

run = metrics_experiment([Faithfulness(llm=llm)])
exp = run.run(dataset, name="nightly")  # await run.arun(...) in async code
summarize(exp)  # {'faithfulness': {'mean': 0.86, 'count': 48, 'missing': 2}}
check(exp, min_mean={"faithfulness": 0.8})

metrics_experiment()

An experiment that scores metrics on every row of a dataset.

Run it like any experiment: await run.arun(dataset, name=...), or run.run(...) from sync code. Each saved row is the dataset row plus, per metric, <name> (the score; NaN if the metric raised) and <name>_reason (its explanation, or None). Use :func:summarize and :func:check on the result.

Rows may be SingleTurnSample / MultiTurnSample objects (a Dataset with data_model=SingleTurnSample), dicts or other pydantic models; dicts become samples (multi-turn when user_input is a list). All rows must be the same sample type. Before any row runs, a required input that no row has, or a metric that can't score the sample type, raises ValueError.

Only None counts as missing: a field set to "" is passed to the metric and scored. The CSV and Google Sheets backends load a missing text value as "", so a row without a reference that went through CSV is scored against an empty one instead of raising. Keep datasets to be scored in local/jsonl or inmemory, which keep None.

Args: metrics: Metric objects (ragas.metrics.collections, or any SimpleBaseMetric). Their names name the columns. llm: For metrics whose ascore() takes llm per call (e.g. DiscreteMetric). Collections metrics take theirs at construction. There is no default LLM. embeddings: The same, for embeddings. column_map: {sample_field: dataset_column}, for a dataset whose columns are named differently, e.g. {"user_input": "question"}. run_config: Timeouts and retries for the metrics' LLM calls; its timeout also bounds each metric call. Default: each LLM's own. usage: A TokenUsageCollector that records the run's token usage (usage.total_tokens() / total_cost(...)). backend, backend_kwargs, name_prefix, max_workers, raise_exceptions, save_partial: As for @experiment. max_workers counts rows; a row's metrics run concurrently. With raise_exceptions, the first metric that raises stops the run; otherwise it scores NaN and the error is logged.

Raises: TypeError: metrics that aren't metric objects, or an llm/embeddings ragas can't call. ValueError: duplicate metric names, a metric no sample type can drive, or a per-call model that wasn't passed.

Source code in src/ragas/scoring.py
def metrics_experiment(
    metrics: t.Sequence[SimpleBaseMetric],
    *,
    llm: t.Optional["InstructorBaseRagasLLM"] = None,
    embeddings: t.Optional["BaseRagasEmbedding"] = None,
    column_map: t.Optional[t.Mapping[str, str]] = None,
    run_config: t.Optional[RunConfig] = None,
    usage: t.Optional["TokenUsageCollector"] = None,
    backend: t.Optional[t.Union[BaseBackend, str]] = None,
    backend_kwargs: t.Optional[t.Dict[str, t.Any]] = None,
    name_prefix: str = "",
    max_workers: int = DEFAULT_MAX_WORKERS,
    raise_exceptions: bool = False,
    save_partial: bool = False,
) -> MetricsExperiment:
    """An experiment that scores ``metrics`` on every row of a dataset.

    Run it like any experiment: ``await run.arun(dataset, name=...)``, or
    ``run.run(...)`` from sync code. Each saved row is the dataset row plus,
    per metric, ``<name>`` (the score; NaN if the metric raised) and
    ``<name>_reason`` (its explanation, or None). Use :func:`summarize` and
    :func:`check` on the result.

    Rows may be ``SingleTurnSample`` / ``MultiTurnSample`` objects (a
    ``Dataset`` with ``data_model=SingleTurnSample``), dicts or other pydantic
    models; dicts become samples (multi-turn when ``user_input`` is a list).
    All rows must be the same sample type. Before any row runs, a required
    input that no row has, or a metric that can't score the sample type,
    raises ``ValueError``.

    Only ``None`` counts as missing: a field set to ``""`` is passed to the
    metric and scored. The CSV and Google Sheets backends load a missing text
    value as ``""``, so a row without a ``reference`` that went through CSV is
    scored against an empty one instead of raising. Keep datasets to be
    scored in ``local/jsonl`` or ``inmemory``, which keep ``None``.

    Args:
        metrics: Metric objects (``ragas.metrics.collections``, or any
            ``SimpleBaseMetric``). Their names name the columns.
        llm: For metrics whose ``ascore()`` takes ``llm`` per call (e.g.
            ``DiscreteMetric``). Collections metrics take theirs at
            construction. There is no default LLM.
        embeddings: The same, for ``embeddings``.
        column_map: ``{sample_field: dataset_column}``, for a dataset whose
            columns are named differently, e.g. ``{"user_input": "question"}``.
        run_config: Timeouts and retries for the metrics' LLM calls; its
            ``timeout`` also bounds each metric call. Default: each LLM's own.
        usage: A ``TokenUsageCollector`` that records the run's token usage
            (``usage.total_tokens()`` / ``total_cost(...)``).
        backend, backend_kwargs, name_prefix, max_workers, raise_exceptions,
            save_partial: As for ``@experiment``. ``max_workers`` counts rows;
            a row's metrics run concurrently. With ``raise_exceptions``, the
            first metric that raises stops the run; otherwise it scores NaN
            and the error is logged.

    Raises:
        TypeError: metrics that aren't metric objects, or an llm/embeddings
            ragas can't call.
        ValueError: duplicate metric names, a metric no sample type can
            drive, or a per-call model that wasn't passed.
    """
    return MetricsExperiment(
        metrics,
        llm=llm,
        embeddings=embeddings,
        column_map=column_map,
        run_config=run_config,
        usage=usage,
        backend=backend,
        backend_kwargs=backend_kwargs,
        name_prefix=name_prefix,
        max_workers=max_workers,
        raise_exceptions=raise_exceptions,
        save_partial=save_partial,
    )

MetricsExperiment

Bases: ExperimentWrapper

An experiment that scores metrics on each row: see :func:metrics_experiment.

Source code in src/ragas/scoring.py
def __init__(
    self,
    metrics: t.Sequence[SimpleBaseMetric],
    *,
    llm: t.Optional["InstructorBaseRagasLLM"] = None,
    embeddings: t.Optional["BaseRagasEmbedding"] = None,
    column_map: t.Optional[t.Mapping[str, str]] = None,
    run_config: t.Optional[RunConfig] = None,
    usage: t.Optional["TokenUsageCollector"] = None,
    backend: t.Optional[t.Union[BaseBackend, str]] = None,
    backend_kwargs: t.Optional[t.Dict[str, t.Any]] = None,
    name_prefix: str = "",
    max_workers: int = DEFAULT_MAX_WORKERS,
    raise_exceptions: bool = False,
    save_partial: bool = False,
):
    if isinstance(metrics, (str, bytes)) or not isinstance(
        metrics, collections.abc.Sequence
    ):
        raise TypeError(
            "metrics must be a list of metrics, e.g. [Faithfulness(llm=llm)]"
        )
    if not metrics:
        raise ValueError("metrics_experiment() needs at least one metric.")
    if any(not isinstance(m, SimpleBaseMetric) for m in metrics):
        raise TypeError(
            "All metrics must be initialised metric objects from "
            "ragas.metrics.collections (or other SimpleBaseMetric "
            "subclasses), e.g. [BleuScore(), Faithfulness(llm=llm)]"
        )
    names = [m.name for m in metrics]
    duplicates = sorted({n for n in names if names.count(n) > 1})
    if duplicates:
        raise ValueError(
            f"Metric names must be unique, as each names a column; "
            f"duplicated: {duplicates}"
        )
    check_models(llm, embeddings)

    self.scorers = [_Scorer(m, input_spec(m)) for m in metrics]
    models = {"llm": llm, "embeddings": embeddings}
    for scorer in self.scorers:
        # A collections metric takes its models at construction; one that
        # takes them per call (a prompt-template metric's ``llm``) gets
        # these. There are no default models.
        for component in sorted(scorer.spec.components):
            if models[component] is None:
                raise ValueError(
                    f"Metric '{scorer.name}' takes `{component}` when it "
                    f"scores; pass metrics_experiment(..., {component}=...)."
                )
            scorer.components[component] = models[component]

    self.column_map = dict(column_map or {})
    self.run_config = run_config
    self.usage = usage
    super().__init__(
        func=self._score_row,
        default_backend=backend,
        name_prefix=name_prefix,
        default_backend_kwargs=backend_kwargs,
        max_workers=max_workers,
        raise_exceptions=raise_exceptions,
        save_partial=save_partial,
    )
    self.__name__ = "metrics_experiment"

metric_names property

metric_names: List[str]

arun async

arun(
    dataset: Any,
    name: Optional[str] = None,
    backend: Optional[Union[BaseBackend, str]] = None,
    *args: Any,
    backend_kwargs: Optional[Dict[str, Any]] = None,
    max_workers: Optional[int] = None,
    raise_exceptions: Optional[bool] = None,
    save_partial: Optional[bool] = None,
    **kwargs: Any,
) -> Experiment

Score every row of dataset; see :meth:ExperimentWrapper.arun.

A metric that raises on a row scores NaN there, with the error logged, and the row is kept. With raise_exceptions the run stops on it.

Source code in src/ragas/scoring.py
async def arun(  # type: ignore[override]
    self,
    dataset: t.Any,
    name: t.Optional[str] = None,
    backend: t.Optional[t.Union[BaseBackend, str]] = None,
    *args: t.Any,
    backend_kwargs: t.Optional[t.Dict[str, t.Any]] = None,
    max_workers: t.Optional[int] = None,
    raise_exceptions: t.Optional[bool] = None,
    save_partial: t.Optional[bool] = None,
    **kwargs: t.Any,
) -> Experiment:
    """Score every row of ``dataset``; see :meth:`ExperimentWrapper.arun`.

    A metric that raises on a row scores NaN there, with the error logged,
    and the row is kept. With ``raise_exceptions`` the run stops on it.
    """
    if args or kwargs:
        raise TypeError(
            "metrics_experiment rows take no extra arguments; got "
            f"{len(args)} positional and {sorted(kwargs)} keyword."
        )
    self._check(dataset)
    fail_fast = (
        self.raise_exceptions if raise_exceptions is None else raise_exceptions
    )
    token = _fail_fast.set(fail_fast)
    try:
        with run_scope(run_config=self.run_config, usage_collector=self.usage):
            result = await super().arun(
                dataset,
                name,
                backend,
                backend_kwargs=backend_kwargs,
                max_workers=max_workers,
                raise_exceptions=raise_exceptions,
                save_partial=save_partial,
            )
    finally:
        _fail_fast.reset(token)
    _warn_on_missing_scores(summarize(result, self.metric_names), result.name)
    return result

run

run(
    dataset: Dataset,
    name: Optional[str] = None,
    backend: Optional[Union[BaseBackend, str]] = None,
    *args,
    backend_kwargs: Optional[Dict[str, Any]] = None,
    max_workers: Optional[int] = None,
    raise_exceptions: Optional[bool] = None,
    save_partial: Optional[bool] = None,
    **kwargs,
) -> Experiment

Run the experiment from synchronous code: arun, without await.

Takes the same arguments as :meth:arun and returns the same Experiment. Like metric.score(), it runs on ragas' shared background event loop (ragas.async_utils.run_sync), so it works in a plain script and inside a running loop (Jupyter) alike. Ctrl-C cancels the rows still running. In async code, prefer await arun(...), which doesn't block the event loop.

Source code in src/ragas/experiment.py
def run(
    self,
    dataset: Dataset,
    name: t.Optional[str] = None,
    backend: t.Optional[t.Union[BaseBackend, str]] = None,
    *args,
    backend_kwargs: t.Optional[t.Dict[str, t.Any]] = None,
    max_workers: t.Optional[int] = None,
    raise_exceptions: t.Optional[bool] = None,
    save_partial: t.Optional[bool] = None,
    **kwargs,
) -> "Experiment":
    """Run the experiment from synchronous code: ``arun``, without ``await``.

    Takes the same arguments as :meth:`arun` and returns the same
    ``Experiment``. Like ``metric.score()``, it runs on
    ragas' shared background event loop (``ragas.async_utils.run_sync``),
    so it works in a plain script and inside a running loop (Jupyter)
    alike. Ctrl-C cancels the rows still running. In async code, prefer
    ``await arun(...)``, which doesn't block the event loop.
    """
    return run_sync(
        self.arun(
            dataset,
            name,
            backend,
            *args,
            backend_kwargs=backend_kwargs,
            max_workers=max_workers,
            raise_exceptions=raise_exceptions,
            save_partial=save_partial,
            **kwargs,
        )
    )

summarize()

Per-metric mean with the number of rows it was computed over.

mean excludes rows without a score (None or NaN): rows whose metric raised and rows where the score is undefined, e.g. no claims to judge. count is how many rows the mean covers and missing how many it doesn't::

>>> summarize(exp)
{'faithfulness': {'mean': 0.86, 'count': 48, 'missing': 2}}

Args: experiment: An Experiment (or any rows) with metric columns, including one reloaded from a backend. metrics: Metric names or objects to summarize. Default: every column metrics_experiment wrote (a <name> column with a <name>_reason column beside it).

Source code in src/ragas/scoring.py
def summarize(
    experiment: t.Iterable[t.Any],
    metrics: t.Optional[t.Sequence[MetricRef]] = None,
) -> Summary:
    """Per-metric mean with the number of rows it was computed over.

    ``mean`` excludes rows without a score (None or NaN): rows whose metric
    raised and rows where the score is undefined, e.g. no claims to judge.
    ``count`` is how many rows the mean covers and ``missing`` how many it
    doesn't::

        >>> summarize(exp)
        {'faithfulness': {'mean': 0.86, 'count': 48, 'missing': 2}}

    Args:
        experiment: An ``Experiment`` (or any rows) with metric columns,
            including one reloaded from a backend.
        metrics: Metric names or objects to summarize. Default: every column
            ``metrics_experiment`` wrote (a ``<name>`` column with a
            ``<name>_reason`` column beside it).
    """
    rows = [_row_dict(row) for row in experiment]
    names = _metric_names(metrics, rows)
    return summarize_columns(
        {name: [_score_value(row.get(name)) for row in rows] for name in names}
    )

check()

Raise ThresholdError unless every metric meets its thresholds.

The one-call CI check (#169)::

check(exp, min_mean={"answer_relevancy": 0.9, "faithfulness": 0.3})
# a range: both bounds
check(exp, min_mean={"faithfulness": 0.3}, max_mean={"faithfulness": 0.5})

Args: experiment: An Experiment with metric columns (see :func:summarize). min_mean: Lowest acceptable mean, per metric ({name: bound}) or one number for every metric. max_mean: Highest acceptable mean, the same way. allow_missing: How many rows per metric may be without a score (errored, or undefined). The default, 0, fails on any: a missing row is left out of the mean, so the mean alone would hide it. None doesn't check. metrics: The metrics to check, as for :func:summarize.

A metric named in a bound but not in the results is an error, so a typo can't pass silently. A metric with no scored row fails any bound.

Raises: ThresholdError: an AssertionError whose message is a table of every checked metric's mean, bounds, count and missing rows.

Source code in src/ragas/scoring.py
def check(
    experiment: t.Iterable[t.Any],
    min_mean: t.Union[float, t.Mapping[str, float], None] = None,
    max_mean: t.Union[float, t.Mapping[str, float], None] = None,
    allow_missing: t.Optional[int] = 0,
    metrics: t.Optional[t.Sequence[MetricRef]] = None,
) -> None:
    """Raise ``ThresholdError`` unless every metric meets its thresholds.

    The one-call CI check (#169)::

        check(exp, min_mean={"answer_relevancy": 0.9, "faithfulness": 0.3})
        # a range: both bounds
        check(exp, min_mean={"faithfulness": 0.3}, max_mean={"faithfulness": 0.5})

    Args:
        experiment: An ``Experiment`` with metric columns (see :func:`summarize`).
        min_mean: Lowest acceptable mean, per metric (``{name: bound}``) or
            one number for every metric.
        max_mean: Highest acceptable mean, the same way.
        allow_missing: How many rows per metric may be without a score
            (errored, or undefined). The default, 0, fails on any: a missing
            row is left out of the mean, so the mean alone would hide it.
            ``None`` doesn't check.
        metrics: The metrics to check, as for :func:`summarize`.

    A metric named in a bound but not in the results is an error, so a typo
    can't pass silently. A metric with no scored row fails any bound.

    Raises:
        ThresholdError: an ``AssertionError`` whose message is a table of
            every checked metric's mean, bounds, count and missing rows.
    """
    check_summary(summarize(experiment, metrics), min_mean, max_mean, allow_missing)