Skip to content

Adding to your CI pipeline with Pytest

You can add Ragas evaluations to your continuous integration pipeline to track the quality of your RAG pipeline over time. Treat them as part of your end-to-end test suite, and run them before major changes and releases.

A Ragas CI test is an ordinary Pytest test that scores a dataset with metrics_experiment and checks the result with check(): a lower or upper bound on each metric's mean, and no unscored rows.

Note

This example uses a dataset that already holds outputs from a reference RAG pipeline. When testing your own system, generate the response and retrieved_contexts columns with the pipeline you want to test. See Evaluation Dataset for how to build a Dataset from your own data.

import pytest
from datasets import load_dataset  # pip install datasets
from openai import AsyncOpenAI

from ragas import Dataset, check, metrics_experiment
from ragas.dataset_schema import SingleTurnSample
from ragas.embeddings import embedding_factory
from ragas.llms import llm_factory
from ragas.metrics.collections import (
    AnswerRelevancy,
    ContextPrecision,
    ContextRecall,
    Faithfulness,
)


def test_amnesty_e2e():
    # columns: user_input, response, retrieved_contexts, reference
    df = load_dataset("vibrantlabsai/amnesty_qa", "english_v3")["eval"].to_pandas()
    df = df.astype(object).where(df.notna(), None)  # NaN -> None
    dataset = Dataset.from_pandas(
        df, "amnesty_qa", "inmemory", data_model=SingleTurnSample
    )

    client = AsyncOpenAI()
    llm = llm_factory("gpt-6-luna", client=client)  # pin the judge model
    embeddings = embedding_factory(
        "openai", model="text-embedding-3-small", client=client
    )

    run = metrics_experiment(
        [
            AnswerRelevancy(llm=llm, embeddings=embeddings),
            Faithfulness(llm=llm),
            ContextRecall(llm=llm),
            ContextPrecision(llm=llm),
        ]
    )
    exp = run.run(dataset, name="amnesty_ci")
    check(
        exp,
        min_mean={
            "answer_relevancy": 0.9,
            "context_recall": 0.95,
            "context_precision": 0.95,
            "faithfulness": 0.3,
        },
        max_mean={"faithfulness": 0.5},  # with min_mean: a range, 0.4 +/- 0.1
    )

check() also fails if any row of any metric has no score. A metric that errors or can't judge a row (a NaN score) is left out of the mean, so the mean alone would hide it. Pass allow_missing=n to allow up to n per metric, or allow_missing=None not to check. A metric name that isn't in the experiment raises ValueError, so a typo can't pass silently. min_mean=0.8 (one number) applies to every metric.

When a check fails, check() raises ThresholdError, an AssertionError, so pytest shows it as a failed assertion with a table:

1 of 4 metrics failed their thresholds:
metric             mean    required    count  missing  result
answer_relevancy   0.9312  >= 0.9      50     0        ok
context_recall     0.9600  >= 0.95     50     0        ok
context_precision  0.9533  >= 0.95     50     0        ok
faithfulness       0.5800  0.3 .. 0.5  48     2        mean above 0.5; 2 rows missing (allowed 0)

For other checks, summarize(exp) gives each metric's mean, count and missing, and [row["faithfulness"] for row in exp] the per-row scores.

Keeping CI scores stable

LLM-judged scores vary a little from run to run. To keep CI from flapping:

  • Pin the judge model and its version, as above. A model upgrade is a deliberate change, and you re-baseline your thresholds when you make it.
  • Keep the defaults deterministic. llm_factory() defaults to a low temperature. Metrics that sample on purpose (strictness > 1) average or vote over several samples, which smooths the result at the cost of more calls.
  • Assert with tolerances, not exact values: a lower bound, or a min_mean/max_mean range around a baseline you measured.
  • Fail on missing rows, as check() does by default. A provider outage or parse failure otherwise shows up only as a smaller count.

Using Pytest Markers for Ragas E2E tests

These end-to-end tests are slow and call an LLM. Pytest markers let you tag them so that they run only when needed.

To add a ragas_ci marker, add the following to your conftest.py:

def pytest_configure(config):
    """Configure pytest."""
    config.addinivalue_line(
        "markers", "ragas_ci: Set of tests that will be run as part of Ragas CI"
    )

Now mark the Ragas tests with it:

import pytest


@pytest.mark.ragas_ci
def test_amnesty_e2e(): ...  # as above

Then run only them with pytest -m ragas_ci, or skip them in fast runs with pytest -m "not ragas_ci".