Optimizers API Reference
Ragas provides optimizers to improve metric prompts through automated optimization. This page documents the available optimizer classes and their configuration.
Overview
Optimizers use annotated datasets with ground truth scores to refine the prompts of ragas.metrics.collections metrics, improving accuracy through:
- Instruction optimization: Finding better prompt wording
- Demonstration optimization: Selecting effective few-shot examples
- Search strategies: Exploring the prompt space efficiently
A collections metric lists its prompts with get_prompts() (copies, keyed by attribute name) and takes replacements with set_prompts(**prompts). Optimizer.optimize() returns optimized copies of those prompts and never changes the metric; metric.train() runs an optimizer and applies the result.
Training a metric
from ragas.config import DemonstrationConfig, InstructionConfig
from ragas.embeddings import embedding_factory
from ragas.llms import llm_factory
from ragas.metrics.collections import AspectCritic
llm = llm_factory("gpt-6-luna")
metric = AspectCritic(
name="helpfulness", definition="Is the response helpful?", llm=llm
)
metric.train(
"annotated_data.json", # or dataset=SingleMetricAnnotation(...)
instruction_config=InstructionConfig(llm=llm), # GeneticOptimizer by default
demonstration_config=DemonstrationConfig(
embedding=embedding_factory("openai", model="text-embedding-3-small")
),
)
train(path=None, demonstration_config=None, instruction_config=None, *, dataset=None, run_config=None, batch_size=None, raise_exceptions=True) optimizes the instructions first, then adds demonstrations; a config that is None or has enabled=False is skipped. The same function is ragas.optimizers.train_metric(metric, ...); the two steps on their own are optimize_instructions() and optimize_demonstrations(), which return prompts for set_prompts without applying them.
The default loss follows the metric's output_type: BinaryMetricLoss (accuracy) for a binary verdict such as AspectCritic's, MSELoss otherwise. Set InstructionConfig(loss=...) to choose another.
Core Classes
Optimizer
dataclass
Optimizer(
metric: Optional[BaseMetric] = None,
llm: Optional[InstructorBaseRagasLLM] = None,
)
Bases: ABC
Abstract base class for all optimizers.
An optimizer tunes the prompts of a collections metric (one that
lists them with get_prompts()) against human annotations, using
llm (an llm_factory() LLM) for its own calls. The metric is not
modified: optimize returns new prompts, which
metric.set_prompts(**prompts) applies. metric.train() does both.
optimize
abstractmethod
optimize(
dataset: SingleMetricAnnotation,
loss: Loss,
config: Optional[Mapping[str, Any]] = None,
*,
run_config: Optional[RunConfig] = None,
batch_size: Optional[int] = None,
callbacks: Optional[Callbacks] = None,
raise_exceptions: bool = True,
) -> Dict[str, BasePrompt]
Optimizes the prompts for the given metric.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
SingleMetricAnnotation
|
The annotated samples to optimize against. |
required |
loss
|
Loss
|
The loss used to score candidate prompts. |
required |
config
|
Mapping[str, Any]
|
The optimizer's settings ( |
None
|
run_config
|
Optional[RunConfig]
|
Execution limits for the underlying runs, by default None. |
None
|
batch_size
|
Optional[int]
|
The number of samples per batch, by default None. |
None
|
callbacks
|
Optional[Callbacks]
|
The callbacks to use during optimization, by default None. |
None
|
raise_exceptions
|
bool
|
Whether to raise instead of swallowing errors, by default True. |
True
|
Returns:
| Type | Description |
|---|---|
Dict[str, BasePrompt]
|
The optimized prompts, keyed by the metric's |
Source code in src/ragas/optimizers/base.py
GeneticOptimizer
dataclass
GeneticOptimizer(
metric: Optional[BaseMetric] = None,
llm: Optional[InstructorBaseRagasLLM] = None,
)
Bases: Optimizer
A genetic algorithm optimizer that balances exploration and exploitation.
Tunes the instructions of a collections metric's prompts. Each candidate
is scored by running a copy of the metric over the annotations, so the metric
being optimized is never modified. config (optimizer_config)
takes population_size (default 3), num_demonstrations (examples
per reverse-engineered instruction, default 3) and sample_size
(samples per feedback mutation, default 12).
candidate_metric
candidate_metric(candidate: Dict[str, str]) -> BaseMetric
A copy of the metric with candidate's instructions.
The metric under optimization is never changed, so candidates scored concurrently can't overwrite each other's instructions.
Source code in src/ragas/optimizers/genetic.py
ascore_candidate
async
ascore_candidate(
*,
candidate: Dict[str, str],
rows: Sequence[Mapping[str, Any]],
run_config: Optional[RunConfig] = None,
batch_size: Optional[int] = None,
callbacks: Optional[Callbacks] = None,
raise_exceptions: bool = True,
parent_pbar: Optional[tqdm] = None,
) -> RowScores
Score candidate (a copy of the metric) on each of rows.
The scoring run nests under callbacks, and the result carries each
row's traced prompt calls for feedback mutation.
Source code in src/ragas/optimizers/genetic.py
score_candidate
score_candidate(
*,
candidate: Dict[str, str],
rows: Sequence[Mapping[str, Any]],
run_config: Optional[RunConfig] = None,
batch_size: Optional[int] = None,
callbacks: Optional[Callbacks] = None,
raise_exceptions: bool = True,
parent_pbar: Optional[tqdm] = None,
) -> RowScores
Sync ascore_candidate.
Source code in src/ragas/optimizers/genetic.py
DSPyOptimizer
dataclass
DSPyOptimizer(
metric: Optional[BaseMetric] = None,
llm: Optional[InstructorBaseRagasLLM] = None,
num_candidates: int = 10,
max_bootstrapped_demos: int = 5,
max_labeled_demos: int = 5,
init_temperature: float = 1.0,
auto: Optional[
Literal["light", "medium", "heavy"]
] = "light",
num_threads: Optional[int] = None,
max_errors: Optional[int] = None,
seed: int = 9,
verbose: bool = False,
track_stats: bool = True,
log_dir: Optional[str] = None,
metric_threshold: Optional[float] = None,
cache: Optional[CacheInterface] = None,
lm: Optional[Any] = None,
)
Bases: Optimizer
Advanced prompt optimizer using DSPy's MIPROv2.
MIPROv2 performs sophisticated prompt optimization by combining: - Instruction optimization (prompt engineering) - Demonstration optimization (few-shot examples) - Combined search over both spaces
Requires: pip install 'ragas-ng[dspy]' (DSPy 3 with optuna)
Each of the metric's prompts is compiled on its own annotations. The
result is a copy of the prompt with MIPROv2's instruction and its chosen
demonstrations as examples. DSPy calls the model llm calls, through
its own LiteLLM dspy.LM (see dspy_adapter.dspy_lm_from_llm), or
lm when given.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
lm
|
Any
|
A DSPy LM to optimize with, instead of one built from |
None
|
num_candidates
|
int
|
Number of prompt variants to try during optimization. |
10
|
max_bootstrapped_demos
|
int
|
Maximum number of auto-generated examples to use. |
5
|
max_labeled_demos
|
int
|
Maximum number of human-annotated examples to use. |
5
|
init_temperature
|
float
|
Exploration temperature for optimization. |
1.0
|
auto
|
str
|
Automatic configuration level: 'light', 'medium', or 'heavy'. Controls the depth of optimization search. |
'light'
|
num_threads
|
int
|
Number of parallel threads for optimization. |
None
|
max_errors
|
int
|
Maximum errors tolerated during optimization before stopping. When
unset, |
None
|
seed
|
int
|
Random seed for reproducibility. |
9
|
verbose
|
bool
|
Enable verbose logging during optimization. |
False
|
track_stats
|
bool
|
Track and report optimization statistics. |
True
|
log_dir
|
str
|
Directory for saving optimization logs and progress. |
None
|
metric_threshold
|
float
|
Minimum acceptable metric value to achieve. |
None
|
cache
|
CacheInterface
|
Cache backend for storing optimization results. |
None
|
optimize
optimize(
dataset: SingleMetricAnnotation,
loss: Loss,
config: Optional[Mapping[str, Any]] = None,
*,
run_config: Optional[RunConfig] = None,
batch_size: Optional[int] = None,
callbacks: Optional[Callbacks] = None,
raise_exceptions: bool = True,
) -> Dict[str, BasePrompt]
Optimize metric prompts using DSPy MIPROv2.
Steps, for each prompt the dataset annotates:
- Convert the prompt to a DSPy Signature
- Create DSPy Module with signature
- Convert the prompt's annotations to DSPy Examples
- Run MIPROv2 optimization
- Extract the optimized instruction and demonstrations
- Return a copy of the prompt with them
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
SingleMetricAnnotation
|
Annotated dataset with ground truth scores. |
required |
loss
|
Loss
|
Loss function to optimize. |
required |
config
|
Mapping[str, Any]
|
Additional settings: |
None
|
run_config
|
RunConfig
|
|
None
|
batch_size
|
None
|
DSPy sizes its own minibatches; any other value raises
|
None
|
callbacks
|
None
|
DSPy traces its own calls; ragas callbacks raise
|
None
|
raise_exceptions
|
bool
|
Without |
True
|
Returns:
| Type | Description |
|---|---|
Dict[str, BasePrompt]
|
For each annotated prompt, a copy of the metric's prompt with the optimized instruction and demonstrations as its examples. |
Source code in src/ragas/optimizers/dspy_optimizer.py
148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 | |
train_metric
train_metric(
metric: "BaseMetric",
path: Optional[Union[str, PathLike]] = None,
*,
dataset: Optional[
Union[MetricAnnotation, SingleMetricAnnotation]
] = None,
demonstration_config: Optional[
"DemonstrationConfig"
] = None,
instruction_config: Optional[
"InstructionConfig"
] = None,
run_config: Optional[RunConfig] = None,
batch_size: Optional[int] = None,
raise_exceptions: bool = True,
) -> None
Tune metric's prompts to human annotations (metric.train()).
The instructions are optimized first (instruction_config), then
demonstrations added (demonstration_config); a config that is None or
has enabled=False is skipped. Each step's prompts replace the metric's
through set_prompts, so the prompt objects it held before, and any
shared with other metrics, are never modified.
Args:
metric: A collections metric (ragas.metrics.collections).
path: A JSON annotation file in MetricAnnotation format.
dataset: The annotations as an object, instead of path. A
MetricAnnotation is narrowed to metric.name.
demonstration_config: Few-shot demonstration settings.
instruction_config: Instruction optimization settings.
run_config: Execution limits for the scoring runs.
batch_size: Jobs per batch for the scoring runs.
raise_exceptions: Raise on a failed job instead of scoring NaN.
Raises:
ValueError: without exactly one of path and dataset, for a
path that isn't .json, or for annotations naming a prompt the
metric doesn't have.
Source code in src/ragas/optimizers/training.py
optimize_instructions
optimize_instructions(
metric: "BaseMetric",
dataset: SingleMetricAnnotation,
config: "InstructionConfig",
run_config: Optional[RunConfig] = None,
batch_size: Optional[int] = None,
raise_exceptions: bool = True,
) -> Dict[str, "BasePrompt"]
Optimized copies of metric's annotated prompts.
Runs config.optimizer on a copy of it, so the optimizer in the config
is left as it was. metric is not changed; apply the result with
metric.set_prompts(**prompts).
Source code in src/ragas/optimizers/training.py
optimize_demonstrations
optimize_demonstrations(
metric: "BaseMetric",
dataset: "SingleMetricAnnotation",
config: "DemonstrationConfig",
) -> Dict[str, BasePrompt]
Prompts with the accepted annotations of dataset as demonstrations.
Returns copies of the annotated prompts, keyed by get_prompts() name,
for metric.set_prompts(**prompts); metric itself is not changed.
Raises: ValueError: for an annotated prompt name the metric doesn't have.
Source code in src/ragas/optimizers/demonstrations.py
GeneticOptimizer
Simple evolutionary optimizer for prompt instructions. It takes no constructor parameters besides metric and llm, which train() sets; its options go in InstructionConfig.optimizer_config.
Options
| Option | Type | Default | Description |
|---|---|---|---|
population_size |
int |
3 | Candidate instructions, including the metric's own |
num_demonstrations |
int |
3 | Annotations per reverse-engineered instruction |
sample_size |
int |
12 | Annotations per feedback mutation |
Usage
from ragas.config import InstructionConfig
from ragas.optimizers import GeneticOptimizer
config = InstructionConfig(
llm=llm,
optimizer=GeneticOptimizer(),
optimizer_config={"population_size": 4},
)
metric.train(dataset=dataset, instruction_config=config)
How it Works
- Reverse-engineers candidate instructions from batches of annotations, and adds the metric's own
- Scores each candidate on a copy of the metric and asks the LLM for feedback on the samples it got wrong, then rewrites it
- Crosses each candidate with the one whose mistakes differ least
- Keeps the candidate with the best loss
It needs at least 10 annotations. Pros: Simple, works with limited data, scores the whole metric. Cons: Instruction-only.
DSPyOptimizer
Advanced optimizer using DSPy's MIPROv2 algorithm.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
auto |
str |
"light" |
MIPROv2's search size: "light", "medium", "heavy" or None |
num_candidates |
int |
10 | Number of prompt variants to try, with auto=None |
max_bootstrapped_demos |
int |
5 | Max auto-generated examples |
max_labeled_demos |
int |
5 | Max human-annotated examples |
init_temperature |
float |
1.0 | Exploration temperature (0.0-2.0) |
max_errors |
int |
None |
Failed calls MIPROv2 tolerates; when None, train(raise_exceptions=...) decides (see below) |
lm |
DSPy LM | None |
The LM DSPy uses, instead of one built from the LLM |
Usage
from ragas.config import InstructionConfig
from ragas.optimizers import DSPyOptimizer
optimizer = DSPyOptimizer(
num_candidates=10,
max_bootstrapped_demos=5,
max_labeled_demos=5,
)
config = InstructionConfig(llm=llm, optimizer=optimizer)
metric.train(dataset=dataset, instruction_config=config)
How it Works
- Generates candidate prompt instructions
- Bootstraps few-shot demonstrations from data
- Selects best human-annotated examples
- Evaluates all combinations on dataset
- Returns, per annotated prompt, a copy with the best instruction and its demonstrations as examples
DSPy calls the model the llm_factory() LLM calls, through a dspy.LM over the same LiteLLM model name and credentials. Pass lm= for an LLM over a litellm.Router or Azure client.
train()'s options apply as follows:
run_config: itsmax_retriesandtimeoutbecome thedspy.LM'snum_retriesandtimeout, unless yourllm_factory()arguments set them. A DSPy LM passed aslm=keeps its own settings.raise_exceptions: withmax_errors=None,True(the default) keeps DSPy's own error limit (dspy.settings.max_errors, 10 by default), andFalsenever stops on errors. DSPy logs each error; when it stops, its error is a generic "Execution cancelled" and the cause is in the log.batch_size: not supported. MIPROv2 sizes its own minibatches, and any value raisesNotImplementedError.
Learn more about DSPy concepts: - Signatures - DSPy's approach to defining input/output specifications - Optimizers - Algorithms for improving prompts and LM weights - Modules - Building blocks for LLM programs
Pros: Better results, combines instructions + demos Cons: Requires DSPy installation, more LLM calls
Installation
DSPy is an optional dependency:
Cost Estimation
Approximate LLM calls per optimization:
Examples:
- Default config (10, 5, 5): ~335 calls
- Budget config (5, 2, 3): ~164 calls
- Aggressive config (20, 10, 10): ~670 calls
Optimizer Base Class
Bases: ABC
Abstract base class for all optimizers.
An optimizer tunes the prompts of a collections metric (one that
lists them with get_prompts()) against human annotations, using
llm (an llm_factory() LLM) for its own calls. The metric is not
modified: optimize returns new prompts, which
metric.set_prompts(**prompts) applies. metric.train() does both.
optimize
abstractmethod
optimize(
dataset: SingleMetricAnnotation,
loss: Loss,
config: Optional[Mapping[str, Any]] = None,
*,
run_config: Optional[RunConfig] = None,
batch_size: Optional[int] = None,
callbacks: Optional[Callbacks] = None,
raise_exceptions: bool = True,
) -> Dict[str, BasePrompt]
Optimizes the prompts for the given metric.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
SingleMetricAnnotation
|
The annotated samples to optimize against. |
required |
loss
|
Loss
|
The loss used to score candidate prompts. |
required |
config
|
Mapping[str, Any]
|
The optimizer's settings ( |
None
|
run_config
|
Optional[RunConfig]
|
Execution limits for the underlying runs, by default None. |
None
|
batch_size
|
Optional[int]
|
The number of samples per batch, by default None. |
None
|
callbacks
|
Optional[Callbacks]
|
The callbacks to use during optimization, by default None. |
None
|
raise_exceptions
|
bool
|
Whether to raise instead of swallowing errors, by default True. |
True
|
Returns:
| Type | Description |
|---|---|
Dict[str, BasePrompt]
|
The optimized prompts, keyed by the metric's |
Configuration
Both optimizers are used with InstructionConfig; few-shot demonstrations use DemonstrationConfig:
from ragas.config import DemonstrationConfig, InstructionConfig
config = InstructionConfig(
llm=llm, # an llm_factory() LLM for the optimizer's own calls
optimizer=optimizer_instance, # Optimizer to use (GeneticOptimizer by default)
)
demonstrations = DemonstrationConfig(
embedding=embeddings, # an embedding_factory() model
top_k=3, # examples shown per input
threshold=0.7, # minimum cosine similarity
technique="similarity", # or "random": a fixed random sample
)
metric.train(
dataset=dataset, instruction_config=config, demonstration_config=demonstrations
)
Dataset Format
Optimizers require annotated datasets with ground truth scores. The keys of prompts are the metric's get_prompts() names:
from ragas.optimizers.annotations import (
PromptAnnotation,
SampleAnnotation,
SingleMetricAnnotation,
)
# Create annotated sample
prompt_annotation = PromptAnnotation(
prompt_input={"user_input": "...", "response": "..."},
prompt_output={"reason": "...", "verdict": 1},
edited_output=None, # Optional: corrected output
)
sample = SampleAnnotation(
metric_input={"user_input": "...", "response": "..."},
metric_output=1, # Ground truth score
prompts={"prompt": prompt_annotation}, # AspectCritic's prompt is "prompt"
is_accepted=True, # Include in optimization
)
# Create dataset
dataset = SingleMetricAnnotation(
name="helpfulness",
samples=[sample, ...], # 20-50+ samples recommended
)
Annotated metric runs: the training data for prompt optimization.
MetricAnnotation (a JSON file of human-reviewed metric runs, keyed by
metric name) and the SingleMetricAnnotation that metric.train() and
the optimizers take. Each SampleAnnotation is one scored row: the metric's
inputs and output, the review (is_accepted) and each prompt's input,
output and edited output.
MetricAnnotation
Bases: BaseModel
from_json
classmethod
Load annotations from a JSON file
Source code in src/ragas/optimizers/annotations.py
SingleMetricAnnotation
Bases: BaseModel
sample
sample(
n: int,
stratify_key: Optional[str] = None,
seed: Optional[int] = None,
) -> "SingleMetricAnnotation"
Create a subset of the dataset.
Parameters:
n (int): The number of samples to include in the subset.
stratify_key (str): The column to stratify the subset on.
seed (int, optional): Seed for a private random.Random, so the
same seed gives the same subset. The global random state
is never used or changed.
Returns:
SingleMetricAnnotation: A subset of the dataset with n samples.
Source code in src/ragas/optimizers/annotations.py
batch
Split the shuffled samples into batches.
Parameters:
batch_size (int): The number of samples in each batch.
drop_last_batch (bool): Whether to drop the last batch if it is smaller than the specified batch size.
seed (int, optional): Seed for a private random.Random, so the
same seed gives the same batches. The global random state
is never used or changed.
Source code in src/ragas/optimizers/annotations.py
stratified_batches
stratified_batches(
batch_size: int,
stratify_key: str,
drop_last_batch: bool = False,
replace: bool = False,
seed: Optional[int] = None,
) -> List[List[SampleAnnotation]]
Create stratified batches based on a specified key, ensuring proportional representation.
Parameters:
batch_size (int): Number of samples per batch.
stratify_key (str): Key in metric_input used for stratification (e.g., class labels).
drop_last_batch (bool): If True, drops the last batch if it has fewer samples than batch_size.
replace (bool): If True, allows reusing samples from the same class to fill a batch if necessary.
seed (int, optional): Seed for a private random.Random, so the
same seed gives the same batches. The global random state
is never used or changed.
Returns: List[List[SampleAnnotation]]: A list of stratified batches, each batch being a list of SampleAnnotation objects.
Source code in src/ragas/optimizers/annotations.py
get_prompt_annotations
Get all the prompt annotations for each prompt as a list.
Source code in src/ragas/optimizers/annotations.py
Loss Functions
Optimizers use loss functions to evaluate prompt quality:
from ragas.losses import BinaryMetricLoss, MSELoss
# Mean Squared Error (default for continuous and discrete scores)
loss = MSELoss()
# Accuracy or F1 (default for binary verdicts)
loss = BinaryMetricLoss(metric="f1_score")
# Use with config
config = InstructionConfig(llm=llm, optimizer=optimizer, loss=loss)
Comparison
| Feature | GeneticOptimizer | DSPyOptimizer |
|---|---|---|
| Installation | Built-in | Requires ragas-ng[dspy] |
| Optimization Target | Instructions only | Instructions + Demos |
| Min Dataset Size | 10+ samples | 20+ samples |
| Typical LLM Calls | 100-500 | 200-700 |
| Accuracy Improvement | +5-8% | +8-12% |
| Best For | Quick optimization | Production metrics |
See Also
- DSPy Optimizer Guide - Detailed usage
- Metric Customization - Creating metrics
- Prompt API Reference - Understanding prompts
Additional Resources
DSPy Documentation: - DSPy Official Documentation - Complete guide to DSPy - MIPROv2 API Reference - Detailed MIPROv2 documentation - DSPy Optimizers Overview - Guide to all DSPy optimizers - DSPy GitHub Repository - Source code and examples
Research Papers: - Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs - MIPROv2 paper