Skip to content

Optimizers API Reference

Ragas provides optimizers to improve metric prompts through automated optimization. This page documents the available optimizer classes and their configuration.

Overview

Optimizers use annotated datasets with ground truth scores to refine the prompts of ragas.metrics.collections metrics, improving accuracy through:

  • Instruction optimization: Finding better prompt wording
  • Demonstration optimization: Selecting effective few-shot examples
  • Search strategies: Exploring the prompt space efficiently

A collections metric lists its prompts with get_prompts() (copies, keyed by attribute name) and takes replacements with set_prompts(**prompts). Optimizer.optimize() returns optimized copies of those prompts and never changes the metric; metric.train() runs an optimizer and applies the result.

Training a metric

from ragas.config import DemonstrationConfig, InstructionConfig
from ragas.embeddings import embedding_factory
from ragas.llms import llm_factory
from ragas.metrics.collections import AspectCritic

llm = llm_factory("gpt-6-luna")
metric = AspectCritic(
    name="helpfulness", definition="Is the response helpful?", llm=llm
)

metric.train(
    "annotated_data.json",  # or dataset=SingleMetricAnnotation(...)
    instruction_config=InstructionConfig(llm=llm),  # GeneticOptimizer by default
    demonstration_config=DemonstrationConfig(
        embedding=embedding_factory("openai", model="text-embedding-3-small")
    ),
)

train(path=None, demonstration_config=None, instruction_config=None, *, dataset=None, run_config=None, batch_size=None, raise_exceptions=True) optimizes the instructions first, then adds demonstrations; a config that is None or has enabled=False is skipped. The same function is ragas.optimizers.train_metric(metric, ...); the two steps on their own are optimize_instructions() and optimize_demonstrations(), which return prompts for set_prompts without applying them.

The default loss follows the metric's output_type: BinaryMetricLoss (accuracy) for a binary verdict such as AspectCritic's, MSELoss otherwise. Set InstructionConfig(loss=...) to choose another.

Core Classes

Optimizer dataclass

Optimizer(
    metric: Optional[BaseMetric] = None,
    llm: Optional[InstructorBaseRagasLLM] = None,
)

Bases: ABC

Abstract base class for all optimizers.

An optimizer tunes the prompts of a collections metric (one that lists them with get_prompts()) against human annotations, using llm (an llm_factory() LLM) for its own calls. The metric is not modified: optimize returns new prompts, which metric.set_prompts(**prompts) applies. metric.train() does both.

optimize abstractmethod

optimize(
    dataset: SingleMetricAnnotation,
    loss: Loss,
    config: Optional[Mapping[str, Any]] = None,
    *,
    run_config: Optional[RunConfig] = None,
    batch_size: Optional[int] = None,
    callbacks: Optional[Callbacks] = None,
    raise_exceptions: bool = True,
) -> Dict[str, BasePrompt]

Optimizes the prompts for the given metric.

Parameters:

Name Type Description Default
dataset SingleMetricAnnotation

The annotated samples to optimize against.

required
loss Loss

The loss used to score candidate prompts.

required
config Mapping[str, Any]

The optimizer's settings (optimizer_config), by default none.

None
run_config Optional[RunConfig]

Execution limits for the underlying runs, by default None.

None
batch_size Optional[int]

The number of samples per batch, by default None.

None
callbacks Optional[Callbacks]

The callbacks to use during optimization, by default None.

None
raise_exceptions bool

Whether to raise instead of swallowing errors, by default True.

True

Returns:

Type Description
Dict[str, BasePrompt]

The optimized prompts, keyed by the metric's get_prompts() names: copies of the metric's prompts with the optimized instruction (and, for DSPyOptimizer, examples). Pass them to metric.set_prompts(**prompts).

Source code in src/ragas/optimizers/base.py
@abstractmethod
def optimize(
    self,
    dataset: SingleMetricAnnotation,
    loss: Loss,
    config: t.Optional[t.Mapping[str, t.Any]] = None,
    *,
    run_config: t.Optional[RunConfig] = None,
    batch_size: t.Optional[int] = None,
    callbacks: t.Optional[Callbacks] = None,
    raise_exceptions: bool = True,
) -> t.Dict[str, BasePrompt]:
    """
    Optimizes the prompts for the given metric.

    Parameters
    ----------
    dataset : SingleMetricAnnotation
        The annotated samples to optimize against.
    loss : Loss
        The loss used to score candidate prompts.
    config : Mapping[str, Any], optional
        The optimizer's settings (``optimizer_config``), by default none.
    run_config : Optional[RunConfig], optional
        Execution limits for the underlying runs, by default None.
    batch_size : Optional[int], optional
        The number of samples per batch, by default None.
    callbacks : Optional[Callbacks], optional
        The callbacks to use during optimization, by default None.
    raise_exceptions : bool, optional
        Whether to raise instead of swallowing errors, by default True.

    Returns
    -------
    Dict[str, BasePrompt]
        The optimized prompts, keyed by the metric's ``get_prompts()``
        names: copies of the metric's prompts with the optimized
        instruction (and, for ``DSPyOptimizer``, examples). Pass them to
        ``metric.set_prompts(**prompts)``.
    """
    raise NotImplementedError("The method `optimize` must be implemented.")

GeneticOptimizer dataclass

GeneticOptimizer(
    metric: Optional[BaseMetric] = None,
    llm: Optional[InstructorBaseRagasLLM] = None,
)

Bases: Optimizer

A genetic algorithm optimizer that balances exploration and exploitation.

Tunes the instructions of a collections metric's prompts. Each candidate is scored by running a copy of the metric over the annotations, so the metric being optimized is never modified. config (optimizer_config) takes population_size (default 3), num_demonstrations (examples per reverse-engineered instruction, default 3) and sample_size (samples per feedback mutation, default 12).

candidate_metric

candidate_metric(candidate: Dict[str, str]) -> BaseMetric

A copy of the metric with candidate's instructions.

The metric under optimization is never changed, so candidates scored concurrently can't overwrite each other's instructions.

Source code in src/ragas/optimizers/genetic.py
def candidate_metric(self, candidate: t.Dict[str, str]) -> BaseMetric:
    """A copy of the metric with ``candidate``'s instructions.

    The metric under optimization is never changed, so candidates scored
    concurrently can't overwrite each other's instructions.
    """
    scored = copy.copy(self._metric)
    scored.set_prompts(**self._with_instructions(candidate))
    return scored

ascore_candidate async

ascore_candidate(
    *,
    candidate: Dict[str, str],
    rows: Sequence[Mapping[str, Any]],
    run_config: Optional[RunConfig] = None,
    batch_size: Optional[int] = None,
    callbacks: Optional[Callbacks] = None,
    raise_exceptions: bool = True,
    parent_pbar: Optional[tqdm] = None,
) -> RowScores

Score candidate (a copy of the metric) on each of rows.

The scoring run nests under callbacks, and the result carries each row's traced prompt calls for feedback mutation.

Source code in src/ragas/optimizers/genetic.py
async def ascore_candidate(
    self,
    *,
    candidate: t.Dict[str, str],
    rows: t.Sequence[t.Mapping[str, t.Any]],
    run_config: t.Optional[RunConfig] = None,
    batch_size: t.Optional[int] = None,
    callbacks: t.Optional[Callbacks] = None,
    raise_exceptions: bool = True,
    parent_pbar: t.Optional[tqdm] = None,
) -> RowScores:
    """Score ``candidate`` (a copy of the metric) on each of ``rows``.

    The scoring run nests under ``callbacks``, and the result carries each
    row's traced prompt calls for feedback mutation.
    """
    return await ascore_rows(
        self.candidate_metric(candidate),
        rows,
        run_config=run_config,
        batch_size=batch_size,
        callbacks=callbacks,
        raise_exceptions=raise_exceptions,
        pbar=parent_pbar,
    )

score_candidate

score_candidate(
    *,
    candidate: Dict[str, str],
    rows: Sequence[Mapping[str, Any]],
    run_config: Optional[RunConfig] = None,
    batch_size: Optional[int] = None,
    callbacks: Optional[Callbacks] = None,
    raise_exceptions: bool = True,
    parent_pbar: Optional[tqdm] = None,
) -> RowScores

Sync ascore_candidate.

Source code in src/ragas/optimizers/genetic.py
def score_candidate(
    self,
    *,
    candidate: t.Dict[str, str],
    rows: t.Sequence[t.Mapping[str, t.Any]],
    run_config: t.Optional[RunConfig] = None,
    batch_size: t.Optional[int] = None,
    callbacks: t.Optional[Callbacks] = None,
    raise_exceptions: bool = True,
    parent_pbar: t.Optional[tqdm] = None,
) -> RowScores:
    """Sync ``ascore_candidate``."""
    return run_sync(
        self.ascore_candidate(
            candidate=candidate,
            rows=rows,
            run_config=run_config,
            batch_size=batch_size,
            callbacks=callbacks,
            raise_exceptions=raise_exceptions,
            parent_pbar=parent_pbar,
        )
    )

DSPyOptimizer dataclass

DSPyOptimizer(
    metric: Optional[BaseMetric] = None,
    llm: Optional[InstructorBaseRagasLLM] = None,
    num_candidates: int = 10,
    max_bootstrapped_demos: int = 5,
    max_labeled_demos: int = 5,
    init_temperature: float = 1.0,
    auto: Optional[
        Literal["light", "medium", "heavy"]
    ] = "light",
    num_threads: Optional[int] = None,
    max_errors: Optional[int] = None,
    seed: int = 9,
    verbose: bool = False,
    track_stats: bool = True,
    log_dir: Optional[str] = None,
    metric_threshold: Optional[float] = None,
    cache: Optional[CacheInterface] = None,
    lm: Optional[Any] = None,
)

Bases: Optimizer

Advanced prompt optimizer using DSPy's MIPROv2.

MIPROv2 performs sophisticated prompt optimization by combining: - Instruction optimization (prompt engineering) - Demonstration optimization (few-shot examples) - Combined search over both spaces

Requires: pip install 'ragas-ng[dspy]' (DSPy 3 with optuna)

Each of the metric's prompts is compiled on its own annotations. The result is a copy of the prompt with MIPROv2's instruction and its chosen demonstrations as examples. DSPy calls the model llm calls, through its own LiteLLM dspy.LM (see dspy_adapter.dspy_lm_from_llm), or lm when given.

Parameters:

Name Type Description Default
lm Any

A DSPy LM to optimize with, instead of one built from llm. Use it for an LLM over a litellm.Router or Azure client.

None
num_candidates int

Number of prompt variants to try during optimization.

10
max_bootstrapped_demos int

Maximum number of auto-generated examples to use.

5
max_labeled_demos int

Maximum number of human-annotated examples to use.

5
init_temperature float

Exploration temperature for optimization.

1.0
auto str

Automatic configuration level: 'light', 'medium', or 'heavy'. Controls the depth of optimization search.

'light'
num_threads int

Number of parallel threads for optimization.

None
max_errors int

Maximum errors tolerated during optimization before stopping. When unset, optimize()'s raise_exceptions decides: DSPy's own limit (dspy.settings.max_errors, 10 by default) when True, or never stop on errors when False.

None
seed int

Random seed for reproducibility.

9
verbose bool

Enable verbose logging during optimization.

False
track_stats bool

Track and report optimization statistics.

True
log_dir str

Directory for saving optimization logs and progress.

None
metric_threshold float

Minimum acceptable metric value to achieve.

None
cache CacheInterface

Cache backend for storing optimization results.

None

optimize

optimize(
    dataset: SingleMetricAnnotation,
    loss: Loss,
    config: Optional[Mapping[str, Any]] = None,
    *,
    run_config: Optional[RunConfig] = None,
    batch_size: Optional[int] = None,
    callbacks: Optional[Callbacks] = None,
    raise_exceptions: bool = True,
) -> Dict[str, BasePrompt]

Optimize metric prompts using DSPy MIPROv2.

Steps, for each prompt the dataset annotates:

  1. Convert the prompt to a DSPy Signature
  2. Create DSPy Module with signature
  3. Convert the prompt's annotations to DSPy Examples
  4. Run MIPROv2 optimization
  5. Extract the optimized instruction and demonstrations
  6. Return a copy of the prompt with them

Parameters:

Name Type Description Default
dataset SingleMetricAnnotation

Annotated dataset with ground truth scores.

required
loss Loss

Loss function to optimize.

required
config Mapping[str, Any]

Additional settings: num_trials (without auto).

None
run_config RunConfig

max_retries and timeout for the dspy.LM built from llm (its num_retries and timeout). A DSPy LM passed as lm keeps its own settings.

None
batch_size None

DSPy sizes its own minibatches; any other value raises NotImplementedError.

None
callbacks None

DSPy traces its own calls; ragas callbacks raise NotImplementedError.

None
raise_exceptions bool

Without max_errors, True (the default) keeps DSPy's own error limit (dspy.settings.max_errors, 10 by default), so a few bad replies don't end a long, paid run; False never stops on errors. DSPy logs each error; at the limit it raises.

True

Returns:

Type Description
Dict[str, BasePrompt]

For each annotated prompt, a copy of the metric's prompt with the optimized instruction and demonstrations as its examples.

Source code in src/ragas/optimizers/dspy_optimizer.py
def optimize(
    self,
    dataset: SingleMetricAnnotation,
    loss: Loss,
    config: t.Optional[t.Mapping[str, t.Any]] = None,
    *,
    run_config: t.Optional[RunConfig] = None,
    batch_size: t.Optional[int] = None,
    callbacks: t.Optional[Callbacks] = None,
    raise_exceptions: bool = True,
) -> t.Dict[str, BasePrompt]:
    """
    Optimize metric prompts using DSPy MIPROv2.

    Steps, for each prompt the dataset annotates:

    1. Convert the prompt to a DSPy Signature
    2. Create DSPy Module with signature
    3. Convert the prompt's annotations to DSPy Examples
    4. Run MIPROv2 optimization
    5. Extract the optimized instruction and demonstrations
    6. Return a copy of the prompt with them

    Parameters
    ----------
    dataset : SingleMetricAnnotation
        Annotated dataset with ground truth scores.
    loss : Loss
        Loss function to optimize.
    config : Mapping[str, Any], optional
        Additional settings: ``num_trials`` (without ``auto``).
    run_config : RunConfig, optional
        ``max_retries`` and ``timeout`` for the ``dspy.LM`` built from
        ``llm`` (its ``num_retries`` and ``timeout``). A DSPy LM passed as
        ``lm`` keeps its own settings.
    batch_size : None
        DSPy sizes its own minibatches; any other value raises
        ``NotImplementedError``.
    callbacks : None
        DSPy traces its own calls; ragas callbacks raise
        ``NotImplementedError``.
    raise_exceptions : bool
        Without ``max_errors``, True (the default) keeps DSPy's own error
        limit (``dspy.settings.max_errors``, 10 by default), so a few bad
        replies don't end a long, paid run; False never stops on errors.
        DSPy logs each error; at the limit it raises.

    Returns
    -------
    Dict[str, BasePrompt]
        For each annotated prompt, a copy of the metric's prompt with the
        optimized instruction and demonstrations as its examples.
    """
    if self.metric is None:
        raise ValueError("No metric provided for optimization.")

    if self.llm is None and self.lm is None:
        raise ValueError("No llm provided for optimization.")

    if self._dspy is None:
        raise RuntimeError("DSPy module not loaded.")

    if batch_size is not None:
        raise NotImplementedError(
            "DSPyOptimizer can't honor batch_size: MIPROv2 sizes its own "
            "minibatches. Leave it unset."
        )
    if callbacks:
        raise NotImplementedError(
            "DSPyOptimizer can't honor ragas callbacks: DSPy runs and "
            "traces its own calls. Leave callbacks unset."
        )
    # None leaves DSPy's own limit (dspy.settings.max_errors) in place.
    max_errors = self.max_errors
    if max_errors is None and not raise_exceptions:
        max_errors = sys.maxsize

    metric = self.metric
    config = dict(config or {})
    if self.cache is not None:
        cache_key = self._generate_cache_key(
            dataset, loss, config, max_errors=max_errors
        )
        if self.cache.has_key(cache_key):
            logger.info(f"Cache hit for DSPy optimization of metric: {metric.name}")
            return self._from_cacheable(self.cache.get(cache_key))

    logger.info(f"Starting DSPy optimization for metric: {metric.name}")

    from ragas.optimizers.dspy_adapter import (
        create_dspy_metric,
        dspy_demos_to_examples,
        prompt_to_dspy_signature,
        ragas_dataset_to_dspy_examples,
        setup_dspy_llm,
    )

    setup_dspy_llm(
        self._dspy, self.lm if self.lm is not None else self.llm, run_config
    )

    prompts = metric.get_prompts()  # copies
    optimized_prompts: t.Dict[str, BasePrompt] = {}

    for prompt_name, prompt in prompts.items():
        examples = ragas_dataset_to_dspy_examples(dataset, prompt_name)
        if not examples:
            logger.info(f"No annotations for prompt {prompt_name}; skipped.")
            continue
        logger.info(f"Optimizing prompt: {prompt_name}")

        signature = prompt_to_dspy_signature(prompt)
        module = self._dspy.Predict(signature)

        metric_fn = create_dspy_metric(loss, _score_field(prompt, dataset.name))

        # MIPROv2 takes the metric here, not in compile(). With an auto
        # level it sizes the search itself and rejects num_candidates;
        # without one it needs num_trials.
        teleprompter = self._dspy.MIPROv2(
            metric=metric_fn,
            num_candidates=None if self.auto else self.num_candidates,
            max_bootstrapped_demos=self.max_bootstrapped_demos,
            max_labeled_demos=self.max_labeled_demos,
            init_temperature=self.init_temperature,
            auto=self.auto,
            num_threads=self.num_threads,
            max_errors=max_errors,
            seed=self.seed,
            verbose=self.verbose,
            track_stats=self.track_stats,
            log_dir=self.log_dir,
            metric_threshold=self.metric_threshold,
        )

        compile_kwargs: t.Dict[str, t.Any] = {"requires_permission_to_run": False}
        if not self.auto:
            compile_kwargs["num_trials"] = config.get(
                "num_trials", self.num_candidates
            )
        optimized = teleprompter.compile(
            module,
            trainset=examples,
            **compile_kwargs,
        )

        optimized_instruction = self._extract_instruction(optimized)
        if optimized_instruction:
            prompt.instruction = optimized_instruction
        demos = getattr(optimized, "demos", None)
        if isinstance(demos, (list, tuple)) and demos:
            prompt.examples = dspy_demos_to_examples(prompt, demos)
        optimized_prompts[prompt_name] = prompt

        logger.info(
            f"Optimized prompt for {prompt_name}: {prompt.instruction[:100]}..."
        )

    if self.cache is not None:
        cache_key = self._generate_cache_key(
            dataset, loss, config, max_errors=max_errors
        )
        self.cache.set(cache_key, self._to_cacheable(optimized_prompts))
        logger.info("Cached optimization results")

    return optimized_prompts

train_metric

train_metric(
    metric: "BaseMetric",
    path: Optional[Union[str, PathLike]] = None,
    *,
    dataset: Optional[
        Union[MetricAnnotation, SingleMetricAnnotation]
    ] = None,
    demonstration_config: Optional[
        "DemonstrationConfig"
    ] = None,
    instruction_config: Optional[
        "InstructionConfig"
    ] = None,
    run_config: Optional[RunConfig] = None,
    batch_size: Optional[int] = None,
    raise_exceptions: bool = True,
) -> None

Tune metric's prompts to human annotations (metric.train()).

The instructions are optimized first (instruction_config), then demonstrations added (demonstration_config); a config that is None or has enabled=False is skipped. Each step's prompts replace the metric's through set_prompts, so the prompt objects it held before, and any shared with other metrics, are never modified.

Args: metric: A collections metric (ragas.metrics.collections). path: A JSON annotation file in MetricAnnotation format. dataset: The annotations as an object, instead of path. A MetricAnnotation is narrowed to metric.name. demonstration_config: Few-shot demonstration settings. instruction_config: Instruction optimization settings. run_config: Execution limits for the scoring runs. batch_size: Jobs per batch for the scoring runs. raise_exceptions: Raise on a failed job instead of scoring NaN.

Raises: ValueError: without exactly one of path and dataset, for a path that isn't .json, or for annotations naming a prompt the metric doesn't have.

Source code in src/ragas/optimizers/training.py
def train_metric(
    metric: "BaseMetric",
    path: t.Optional[t.Union[str, os.PathLike]] = None,
    *,
    dataset: t.Optional[t.Union[MetricAnnotation, SingleMetricAnnotation]] = None,
    demonstration_config: t.Optional["DemonstrationConfig"] = None,
    instruction_config: t.Optional["InstructionConfig"] = None,
    run_config: t.Optional[RunConfig] = None,
    batch_size: t.Optional[int] = None,
    raise_exceptions: bool = True,
) -> None:
    """Tune ``metric``'s prompts to human annotations (``metric.train()``).

    The instructions are optimized first (``instruction_config``), then
    demonstrations added (``demonstration_config``); a config that is None or
    has ``enabled=False`` is skipped. Each step's prompts replace the metric's
    through ``set_prompts``, so the prompt objects it held before, and any
    shared with other metrics, are never modified.

    Args:
        metric: A collections metric (``ragas.metrics.collections``).
        path: A JSON annotation file in ``MetricAnnotation`` format.
        dataset: The annotations as an object, instead of ``path``. A
            ``MetricAnnotation`` is narrowed to ``metric.name``.
        demonstration_config: Few-shot demonstration settings.
        instruction_config: Instruction optimization settings.
        run_config: Execution limits for the scoring runs.
        batch_size: Jobs per batch for the scoring runs.
        raise_exceptions: Raise on a failed job instead of scoring NaN.

    Raises:
        ValueError: without exactly one of ``path`` and ``dataset``, for a
            path that isn't ``.json``, or for annotations naming a prompt the
            metric doesn't have.
    """
    annotations = _load_annotations(metric, path, dataset)
    _check_prompt_names(metric, annotations)

    if instruction_config is not None and instruction_config.enabled:
        optimized = optimize_instructions(
            metric,
            annotations,
            instruction_config,
            run_config=run_config or RunConfig(),
            batch_size=batch_size,
            raise_exceptions=raise_exceptions,
        )
        metric.set_prompts(**optimized)

    if demonstration_config is not None and demonstration_config.enabled:
        metric.set_prompts(
            **optimize_demonstrations(metric, annotations, demonstration_config)
        )

optimize_instructions

optimize_instructions(
    metric: "BaseMetric",
    dataset: SingleMetricAnnotation,
    config: "InstructionConfig",
    run_config: Optional[RunConfig] = None,
    batch_size: Optional[int] = None,
    raise_exceptions: bool = True,
) -> Dict[str, "BasePrompt"]

Optimized copies of metric's annotated prompts.

Runs config.optimizer on a copy of it, so the optimizer in the config is left as it was. metric is not changed; apply the result with metric.set_prompts(**prompts).

Source code in src/ragas/optimizers/training.py
def optimize_instructions(
    metric: "BaseMetric",
    dataset: SingleMetricAnnotation,
    config: "InstructionConfig",
    run_config: t.Optional[RunConfig] = None,
    batch_size: t.Optional[int] = None,
    raise_exceptions: bool = True,
) -> t.Dict[str, "BasePrompt"]:
    """Optimized copies of ``metric``'s annotated prompts.

    Runs ``config.optimizer`` on a copy of it, so the optimizer in the config
    is left as it was. ``metric`` is not changed; apply the result with
    ``metric.set_prompts(**prompts)``.
    """
    optimizer = copy.copy(config.optimizer)
    optimizer.metric = metric
    if optimizer.llm is None:
        optimizer.llm = config.llm
    loss = config.loss if config.loss is not None else default_loss(metric)
    return optimizer.optimize(
        dataset,
        loss,
        dict(config.optimizer_config or {}),
        run_config=run_config,
        batch_size=batch_size,
        raise_exceptions=raise_exceptions,
    )

optimize_demonstrations

optimize_demonstrations(
    metric: "BaseMetric",
    dataset: "SingleMetricAnnotation",
    config: "DemonstrationConfig",
) -> Dict[str, BasePrompt]

Prompts with the accepted annotations of dataset as demonstrations.

Returns copies of the annotated prompts, keyed by get_prompts() name, for metric.set_prompts(**prompts); metric itself is not changed.

Raises: ValueError: for an annotated prompt name the metric doesn't have.

Source code in src/ragas/optimizers/demonstrations.py
def optimize_demonstrations(
    metric: "BaseMetric",
    dataset: "SingleMetricAnnotation",
    config: "DemonstrationConfig",
) -> t.Dict[str, BasePrompt]:
    """Prompts with the accepted annotations of ``dataset`` as demonstrations.

    Returns copies of the annotated prompts, keyed by ``get_prompts()`` name,
    for ``metric.set_prompts(**prompts)``; ``metric`` itself is not changed.

    Raises:
        ValueError: for an annotated prompt name the metric doesn't have.
    """
    check_example_count(config.top_k)
    prompts = metric.get_prompts()  # copies
    result: t.Dict[str, BasePrompt] = {}
    for prompt_name, prompt_annotations in dataset.get_prompt_annotations().items():
        if prompt_name not in prompts:
            raise ValueError(
                f"Prompt '{prompt_name}' not found in metric '{metric.name}'. "
                f"Its prompts are: {', '.join(sorted(prompts)) or 'none'}."
            )
        prompt = prompts[prompt_name]
        examples = annotation_examples(prompt, prompt_annotations, prompt_name)
        if config.technique == "random":
            pool = list(prompt.examples) + examples
            prompt.examples = random.sample(pool, min(config.top_k, len(pool)))
            result[prompt_name] = prompt
        else:
            result[prompt_name] = with_similar_examples(
                prompt,
                examples,
                embeddings=config.embedding,
                top_k=config.top_k,
                threshold=config.threshold,
            )
    return result

GeneticOptimizer

Simple evolutionary optimizer for prompt instructions. It takes no constructor parameters besides metric and llm, which train() sets; its options go in InstructionConfig.optimizer_config.

Options

Option Type Default Description
population_size int 3 Candidate instructions, including the metric's own
num_demonstrations int 3 Annotations per reverse-engineered instruction
sample_size int 12 Annotations per feedback mutation

Usage

from ragas.config import InstructionConfig
from ragas.optimizers import GeneticOptimizer

config = InstructionConfig(
    llm=llm,
    optimizer=GeneticOptimizer(),
    optimizer_config={"population_size": 4},
)
metric.train(dataset=dataset, instruction_config=config)

How it Works

  1. Reverse-engineers candidate instructions from batches of annotations, and adds the metric's own
  2. Scores each candidate on a copy of the metric and asks the LLM for feedback on the samples it got wrong, then rewrites it
  3. Crosses each candidate with the one whose mistakes differ least
  4. Keeps the candidate with the best loss

It needs at least 10 annotations. Pros: Simple, works with limited data, scores the whole metric. Cons: Instruction-only.

DSPyOptimizer

Advanced optimizer using DSPy's MIPROv2 algorithm.

Parameters

Parameter Type Default Description
auto str "light" MIPROv2's search size: "light", "medium", "heavy" or None
num_candidates int 10 Number of prompt variants to try, with auto=None
max_bootstrapped_demos int 5 Max auto-generated examples
max_labeled_demos int 5 Max human-annotated examples
init_temperature float 1.0 Exploration temperature (0.0-2.0)
max_errors int None Failed calls MIPROv2 tolerates; when None, train(raise_exceptions=...) decides (see below)
lm DSPy LM None The LM DSPy uses, instead of one built from the LLM

Usage

from ragas.config import InstructionConfig
from ragas.optimizers import DSPyOptimizer

optimizer = DSPyOptimizer(
    num_candidates=10,
    max_bootstrapped_demos=5,
    max_labeled_demos=5,
)

config = InstructionConfig(llm=llm, optimizer=optimizer)
metric.train(dataset=dataset, instruction_config=config)

How it Works

  1. Generates candidate prompt instructions
  2. Bootstraps few-shot demonstrations from data
  3. Selects best human-annotated examples
  4. Evaluates all combinations on dataset
  5. Returns, per annotated prompt, a copy with the best instruction and its demonstrations as examples

DSPy calls the model the llm_factory() LLM calls, through a dspy.LM over the same LiteLLM model name and credentials. Pass lm= for an LLM over a litellm.Router or Azure client.

train()'s options apply as follows:

  • run_config: its max_retries and timeout become the dspy.LM's num_retries and timeout, unless your llm_factory() arguments set them. A DSPy LM passed as lm= keeps its own settings.
  • raise_exceptions: with max_errors=None, True (the default) keeps DSPy's own error limit (dspy.settings.max_errors, 10 by default), and False never stops on errors. DSPy logs each error; when it stops, its error is a generic "Execution cancelled" and the cause is in the log.
  • batch_size: not supported. MIPROv2 sizes its own minibatches, and any value raises NotImplementedError.

Learn more about DSPy concepts: - Signatures - DSPy's approach to defining input/output specifications - Optimizers - Algorithms for improving prompts and LM weights - Modules - Building blocks for LLM programs

Pros: Better results, combines instructions + demos Cons: Requires DSPy installation, more LLM calls

Installation

DSPy is an optional dependency:

# Using uv (recommended)
uv add "ragas-ng[dspy]"

# Using pip
pip install "ragas-ng[dspy]"

Cost Estimation

Approximate LLM calls per optimization:

Total calls ≈ num_candidates × 30 + max_bootstrapped_demos × 7

Examples:

  • Default config (10, 5, 5): ~335 calls
  • Budget config (5, 2, 3): ~164 calls
  • Aggressive config (20, 10, 10): ~670 calls

Optimizer Base Class

Bases: ABC

Abstract base class for all optimizers.

An optimizer tunes the prompts of a collections metric (one that lists them with get_prompts()) against human annotations, using llm (an llm_factory() LLM) for its own calls. The metric is not modified: optimize returns new prompts, which metric.set_prompts(**prompts) applies. metric.train() does both.

optimize abstractmethod

optimize(
    dataset: SingleMetricAnnotation,
    loss: Loss,
    config: Optional[Mapping[str, Any]] = None,
    *,
    run_config: Optional[RunConfig] = None,
    batch_size: Optional[int] = None,
    callbacks: Optional[Callbacks] = None,
    raise_exceptions: bool = True,
) -> Dict[str, BasePrompt]

Optimizes the prompts for the given metric.

Parameters:

Name Type Description Default
dataset SingleMetricAnnotation

The annotated samples to optimize against.

required
loss Loss

The loss used to score candidate prompts.

required
config Mapping[str, Any]

The optimizer's settings (optimizer_config), by default none.

None
run_config Optional[RunConfig]

Execution limits for the underlying runs, by default None.

None
batch_size Optional[int]

The number of samples per batch, by default None.

None
callbacks Optional[Callbacks]

The callbacks to use during optimization, by default None.

None
raise_exceptions bool

Whether to raise instead of swallowing errors, by default True.

True

Returns:

Type Description
Dict[str, BasePrompt]

The optimized prompts, keyed by the metric's get_prompts() names: copies of the metric's prompts with the optimized instruction (and, for DSPyOptimizer, examples). Pass them to metric.set_prompts(**prompts).

Configuration

Both optimizers are used with InstructionConfig; few-shot demonstrations use DemonstrationConfig:

from ragas.config import DemonstrationConfig, InstructionConfig

config = InstructionConfig(
    llm=llm,  # an llm_factory() LLM for the optimizer's own calls
    optimizer=optimizer_instance,  # Optimizer to use (GeneticOptimizer by default)
)
demonstrations = DemonstrationConfig(
    embedding=embeddings,  # an embedding_factory() model
    top_k=3,  # examples shown per input
    threshold=0.7,  # minimum cosine similarity
    technique="similarity",  # or "random": a fixed random sample
)

metric.train(
    dataset=dataset, instruction_config=config, demonstration_config=demonstrations
)

Dataset Format

Optimizers require annotated datasets with ground truth scores. The keys of prompts are the metric's get_prompts() names:

from ragas.optimizers.annotations import (
    PromptAnnotation,
    SampleAnnotation,
    SingleMetricAnnotation,
)

# Create annotated sample
prompt_annotation = PromptAnnotation(
    prompt_input={"user_input": "...", "response": "..."},
    prompt_output={"reason": "...", "verdict": 1},
    edited_output=None,  # Optional: corrected output
)

sample = SampleAnnotation(
    metric_input={"user_input": "...", "response": "..."},
    metric_output=1,  # Ground truth score
    prompts={"prompt": prompt_annotation},  # AspectCritic's prompt is "prompt"
    is_accepted=True,  # Include in optimization
)

# Create dataset
dataset = SingleMetricAnnotation(
    name="helpfulness",
    samples=[sample, ...],  # 20-50+ samples recommended
)

Annotated metric runs: the training data for prompt optimization.

MetricAnnotation (a JSON file of human-reviewed metric runs, keyed by metric name) and the SingleMetricAnnotation that metric.train() and the optimizers take. Each SampleAnnotation is one scored row: the metric's inputs and output, the review (is_accepted) and each prompt's input, output and edited output.

MetricAnnotation

Bases: BaseModel

from_json classmethod

from_json(
    path: str, metric_name: Optional[str]
) -> "MetricAnnotation"

Load annotations from a JSON file

Source code in src/ragas/optimizers/annotations.py
@classmethod
def from_json(cls, path: str, metric_name: t.Optional[str]) -> "MetricAnnotation":
    """Load annotations from a JSON file"""
    with open(path, encoding="utf-8") as f:  # closed, utf-8 (#102)
        dataset = json.load(f)
    return cls._process_dataset(dataset, metric_name)

SingleMetricAnnotation

Bases: BaseModel

sample

sample(
    n: int,
    stratify_key: Optional[str] = None,
    seed: Optional[int] = None,
) -> "SingleMetricAnnotation"

Create a subset of the dataset.

Parameters: n (int): The number of samples to include in the subset. stratify_key (str): The column to stratify the subset on. seed (int, optional): Seed for a private random.Random, so the same seed gives the same subset. The global random state is never used or changed.

Returns: SingleMetricAnnotation: A subset of the dataset with n samples.

Source code in src/ragas/optimizers/annotations.py
def sample(
    self,
    n: int,
    stratify_key: t.Optional[str] = None,
    seed: t.Optional[int] = None,
) -> "SingleMetricAnnotation":
    """
    Create a subset of the dataset.

    Parameters:
        n (int): The number of samples to include in the subset.
        stratify_key (str): The column to stratify the subset on.
        seed (int, optional): Seed for a private ``random.Random``, so the
            same seed gives the same subset. The global ``random`` state
            is never used or changed.

    Returns:
        SingleMetricAnnotation: A subset of the dataset with `n` samples.
    """
    if n > len(self.samples):
        raise ValueError(
            "Requested sample size exceeds the number of available samples."
        )

    rng = random.Random(seed)
    if stratify_key is None:
        # Simple random sampling
        sampled_indices = rng.sample(range(len(self.samples)), n)
        sampled_samples = [self.samples[i] for i in sampled_indices]
    else:
        # Stratified sampling
        class_groups = defaultdict(list)
        for idx, sample in enumerate(self.samples):
            key = sample[stratify_key]
            class_groups[key].append(idx)

        # Determine the proportion of samples to take from each class
        total_samples = sum(len(indices) for indices in class_groups.values())
        proportions = {
            cls: len(indices) / total_samples
            for cls, indices in class_groups.items()
        }

        sampled_indices = []
        for cls, indices in class_groups.items():
            cls_sample_count = int(np.round(proportions[cls] * n))
            cls_sample_count = min(
                cls_sample_count, len(indices)
            )  # Don't oversample
            sampled_indices.extend(rng.sample(indices, cls_sample_count))

        # Handle any rounding discrepancies to ensure exactly `n` samples
        while len(sampled_indices) < n:
            remaining_indices = set(range(len(self.samples))) - set(sampled_indices)
            if not remaining_indices:
                break
            sampled_indices.append(rng.choice(sorted(remaining_indices)))

        sampled_samples = [self.samples[i] for i in sampled_indices]

    return SingleMetricAnnotation(name=self.name, samples=sampled_samples)

batch

batch(
    batch_size: int,
    drop_last_batch: bool = False,
    seed: Optional[int] = None,
)

Split the shuffled samples into batches.

Parameters: batch_size (int): The number of samples in each batch. drop_last_batch (bool): Whether to drop the last batch if it is smaller than the specified batch size. seed (int, optional): Seed for a private random.Random, so the same seed gives the same batches. The global random state is never used or changed.

Source code in src/ragas/optimizers/annotations.py
def batch(
    self,
    batch_size: int,
    drop_last_batch: bool = False,
    seed: t.Optional[int] = None,
):
    """
    Split the shuffled samples into batches.

    Parameters:
        batch_size (int): The number of samples in each batch.
        drop_last_batch (bool): Whether to drop the last batch if it is smaller than the specified batch size.
        seed (int, optional): Seed for a private ``random.Random``, so the
            same seed gives the same batches. The global ``random`` state
            is never used or changed.
    """

    samples = self.samples[:]
    random.Random(seed).shuffle(samples)

    all_batches = [
        samples[i : i + batch_size]
        for i in range(0, len(samples), batch_size)
        if len(samples[i : i + batch_size]) == batch_size or not drop_last_batch
    ]

    return all_batches

stratified_batches

stratified_batches(
    batch_size: int,
    stratify_key: str,
    drop_last_batch: bool = False,
    replace: bool = False,
    seed: Optional[int] = None,
) -> List[List[SampleAnnotation]]

Create stratified batches based on a specified key, ensuring proportional representation.

Parameters: batch_size (int): Number of samples per batch. stratify_key (str): Key in metric_input used for stratification (e.g., class labels). drop_last_batch (bool): If True, drops the last batch if it has fewer samples than batch_size. replace (bool): If True, allows reusing samples from the same class to fill a batch if necessary. seed (int, optional): Seed for a private random.Random, so the same seed gives the same batches. The global random state is never used or changed.

Returns: List[List[SampleAnnotation]]: A list of stratified batches, each batch being a list of SampleAnnotation objects.

Source code in src/ragas/optimizers/annotations.py
def stratified_batches(
    self,
    batch_size: int,
    stratify_key: str,
    drop_last_batch: bool = False,
    replace: bool = False,
    seed: t.Optional[int] = None,
) -> t.List[t.List[SampleAnnotation]]:
    """
    Create stratified batches based on a specified key, ensuring proportional representation.

    Parameters:
        batch_size (int): Number of samples per batch.
        stratify_key (str): Key in `metric_input` used for stratification (e.g., class labels).
        drop_last_batch (bool): If True, drops the last batch if it has fewer samples than `batch_size`.
        replace (bool): If True, allows reusing samples from the same class to fill a batch if necessary.
        seed (int, optional): Seed for a private ``random.Random``, so the
            same seed gives the same batches. The global ``random`` state
            is never used or changed.

    Returns:
        List[List[SampleAnnotation]]: A list of stratified batches, each batch being a list of SampleAnnotation objects.
    """
    # Group samples based on the stratification key
    class_groups = defaultdict(list)
    for sample in self.samples:
        key = sample[stratify_key]
        class_groups[key].append(sample)

    rng = random.Random(seed)
    # Shuffle each class group for randomness
    for group in class_groups.values():
        rng.shuffle(group)

    # Determine the number of batches required
    total_samples = len(self.samples)
    num_batches = (
        np.ceil(total_samples / batch_size).astype(int)
        if drop_last_batch
        else np.floor(total_samples / batch_size).astype(int)
    )
    samples_per_class_per_batch = {
        cls: max(1, len(samples) // num_batches)
        for cls, samples in class_groups.items()
    }

    # Create stratified batches
    all_batches = []
    while len(all_batches) < num_batches:
        batch = []
        for cls, samples in list(class_groups.items()):
            # Determine the number of samples to take from this class
            count = min(
                samples_per_class_per_batch[cls],
                len(samples),
                batch_size - len(batch),
            )
            if count > 0:
                # Add samples from the current class
                batch.extend(samples[:count])
                class_groups[cls] = samples[count:]  # Remove used samples
            elif replace and len(batch) < batch_size:
                # Reuse samples if `replace` is True
                batch.extend(rng.choices(samples, k=batch_size - len(batch)))

        # Shuffle the batch to mix classes
        rng.shuffle(batch)
        if len(batch) == batch_size or not drop_last_batch:
            all_batches.append(batch)

    return all_batches

get_prompt_annotations

get_prompt_annotations() -> Dict[
    str, List[PromptAnnotation]
]

Get all the prompt annotations for each prompt as a list.

Source code in src/ragas/optimizers/annotations.py
def get_prompt_annotations(self) -> t.Dict[str, t.List[PromptAnnotation]]:
    """
    Get all the prompt annotations for each prompt as a list.
    """
    prompt_annotations = defaultdict(list)
    for sample in self.samples:
        if sample.is_accepted:
            for prompt_name, prompt_annotation in sample.prompts.items():
                prompt_annotations[prompt_name].append(prompt_annotation)
    return prompt_annotations

Loss Functions

Optimizers use loss functions to evaluate prompt quality:

from ragas.losses import BinaryMetricLoss, MSELoss

# Mean Squared Error (default for continuous and discrete scores)
loss = MSELoss()

# Accuracy or F1 (default for binary verdicts)
loss = BinaryMetricLoss(metric="f1_score")

# Use with config
config = InstructionConfig(llm=llm, optimizer=optimizer, loss=loss)

Comparison

Feature GeneticOptimizer DSPyOptimizer
Installation Built-in Requires ragas-ng[dspy]
Optimization Target Instructions only Instructions + Demos
Min Dataset Size 10+ samples 20+ samples
Typical LLM Calls 100-500 200-700
Accuracy Improvement +5-8% +8-12%
Best For Quick optimization Production metrics

See Also

Additional Resources

DSPy Documentation: - DSPy Official Documentation - Complete guide to DSPy - MIPROv2 API Reference - Detailed MIPROv2 documentation - DSPy Optimizers Overview - Guide to all DSPy optimizers - DSPy GitHub Repository - Source code and examples

Research Papers: - Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs - MIPROv2 paper