Integrations
ragas.integrations.llama_index
aevaluate_query_engine
async
aevaluate_query_engine(
query_engine: Any,
dataset: DataTable,
metrics: Sequence["SimpleBaseMetric"],
*,
llm: Optional["InstructorBaseRagasLLM"] = None,
embeddings: Optional["BaseRagasEmbedding"] = None,
run_config: Optional[RunConfig] = None,
batch_size: Optional[int] = None,
usage: Optional["TokenUsageCollector"] = None,
raise_exceptions: bool = False,
column_map: Optional[Mapping[str, str]] = None,
show_progress: bool = True,
name: Optional[str] = None,
backend: Optional[Union["BaseBackend", str]] = None,
backend_kwargs: Optional[Dict[str, Any]] = None,
) -> "Experiment"
Run a LlamaIndex query engine over dataset and score the answers.
Each row's user_input goes to query_engine.aquery(); the answer
and its source nodes become the row's response and
retrieved_contexts. The filled-in rows are then scored with
:func:ragas.metrics_experiment, and its Experiment is returned:
use ragas.summarize() / ragas.check() on it.
Args:
query_engine: A LlamaIndex query engine (anything with aquery).
dataset: A Dataset of single-turn rows (SingleTurnSample
objects or dicts). It isn't modified.
metrics: ragas.metrics.collections metrics (or any
SimpleBaseMetric).
llm, embeddings: For metrics that take them per call; there is no
default. LlamaIndex LLMs aren't wrapped (#215): reach the same
provider with llm_factory().
run_config: Timeouts and retries for the queries and the metrics.
batch_size: How many queries run at once (default: all).
usage: A TokenUsageCollector for the metrics' token usage.
column_map: {sample_field: dataset_column}, for a dataset whose
columns are named differently, e.g. {"user_input": "question"}.
raise_exceptions: Stop at the first failed query or metric. Otherwise
a failed query leaves the row's response empty (None) and the row
scores NaN for every metric, with query engine failed: ... as
each reason, so summarize() counts it as missing and
check() fails on it unless allow_missing permits it. A
failed metric scores NaN on its own. Both are logged.
show_progress: Show the query progress bar.
name, backend, backend_kwargs: The experiment's name, and where to
save it (default: in memory).
Raises: NotImplementedError: multi-turn rows.
Source code in src/ragas/integrations/llama_index.py
69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 | |
evaluate_query_engine
evaluate_query_engine(
query_engine: Any,
dataset: DataTable,
metrics: Sequence["SimpleBaseMetric"],
**kwargs: Any,
) -> "Experiment"
Sync :func:aevaluate_query_engine; takes the same arguments.
Source code in src/ragas/integrations/llama_index.py
convert_to_ragas_messages
convert_to_ragas_messages(
events: List[Event],
) -> List[Message]
Convert a sequence of LlamIndex agent events into Ragas message objects.
This function processes a list of Event objects (e.g., AgentInput, AgentOutput,
and ToolCallResult) and converts them into a list of Message objects (HumanMessage,
AIMessage, and ToolMessage) that can be used for evaluation with the Ragas framework.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
events
|
List[Event]
|
A list of agent events that represent a conversation trace. These can include
user inputs ( |
required |
Returns:
| Type | Description |
|---|---|
List[Message]
|
A list of Ragas |
Source code in src/ragas/integrations/llama_index.py
220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 | |
ragas.integrations.helicone
ragas.integrations.amazon_bedrock
get_last_orchestration_value
Iterates through the traces to find the last occurrence of a specified key within the orchestrationTrace.
Returns: (index, value): Tuple where index is the last index at which the key was found, and value is the corresponding value, or (None, None) if not found.
Source code in src/ragas/integrations/amazon_bedrock.py
extract_messages_from_model_invocation
Extracts messages from the 'text' field of the modelInvocationInput. Ensures that each message's content is cast to a string.
Returns: List of messages as HumanMessage or AIMessage objects.
Source code in src/ragas/integrations/amazon_bedrock.py
convert_to_ragas_messages
Converts a list of trace dictionaries into a list of messages. It extracts messages from the last modelInvocationInput and appends the finalResponse from the observation (if it occurs after the model invocation).
Returns: List of HumanMessage and AIMessage objects.
Source code in src/ragas/integrations/amazon_bedrock.py
extract_kb_trace
Extracts groups of traces that follow the specific order: 1. An element with 'trace' -> 'orchestrationTrace' containing an 'invocationInput' with invocationType == "KNOWLEDGE_BASE" 2. Followed (later in the list or within the same trace) by an element with an 'observation' that contains 'knowledgeBaseLookupOutput' 3. Followed by an element with an 'observation' that contains 'finalResponse'
Returns a list of dictionaries each with keys: 'user_input', 'retrieved_contexts', and 'response'
This version supports multiple knowledge base invocation groups.
Source code in src/ragas/integrations/amazon_bedrock.py
ragas.integrations.swarm
convert_to_ragas_messages
convert_to_ragas_messages(
messages: List[Dict[str, Any]],
) -> List[Union[HumanMessage, AIMessage, ToolMessage]]
Convert Swarm messages to Ragas message format.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
messages
|
List[Union[Response, Dict]]
|
List of messages to convert, where each message can be either: - Response: A Swarm Response object containing messages - Dict: A dictionary containing a user message |
required |
Returns:
| Type | Description |
|---|---|
List[Union[HumanMessage, AIMessage, ToolMessage]]
|
List of converted Ragas format messages where: - HumanMessage: For user messages - AIMessage: For assistant messages with optional tool calls - ToolMessage: For tool response messages |
The messages are OpenAI chat-format dicts, so any OpenAI-format history
|
|
works, not only Swarm's. ``system`` and ``developer`` messages are
|
|
skipped. Content may be a string, ``None`` or a list of content parts
|
|
(text parts are kept).
|
|
Raises:
| Type | Description |
|---|---|
KeyError
|
If a message is missing the required 'role' key |
ValueError
|
If a message has a role other than user, assistant, tool, system or developer |