Schemas
TestsetSample
Bases: BaseSample
Represents a sample in a test set.
Attributes:
| Name | Type | Description |
|---|---|---|
eval_sample |
Union[SingleTurnSample, MultiTurnSample]
|
The evaluation sample, which can be either a single-turn or multi-turn sample. |
synthesizer_name |
str
|
The name of the synthesizer used to generate this sample. |
TestsetPacket
Bases: BaseModel
A packet of testset samples to be uploaded to the server.
Testset
dataclass
Testset(
samples: List[TestsetSample],
run_id: str = (lambda: str(uuid4()))(),
cost_cb: Optional[TokenUsageCollector] = None,
personas: Optional[List["Persona"]] = None,
)
Represents a test set containing multiple test samples.
Attributes:
| Name | Type | Description |
|---|---|---|
samples |
List[TestsetSample]
|
A list of TestsetSample objects representing the samples in the test set. |
personas |
Optional[List[Persona]]
|
The personas the samples were generated from, set by
|
to_dataset
to_dataset(
name: str,
backend: Union["BaseBackend", str] = "inmemory",
**backend_kwargs: Any,
) -> "Dataset"
The generated samples as a Dataset, to score or save.
Rows are copies of each sample's eval_sample, and data_model
is their type (SingleTurnSample or MultiTurnSample). The
synthesizer names are left out; to_list() keeps them. Nothing is
written until you call save()::
dataset = testset.to_dataset("testset", "local/jsonl", root_dir="data")
dataset.save()
exp = metrics_experiment([Faithfulness(llm=llm)]).run(dataset)
Prefer local/jsonl to local/csv for a testset you'll score.
CSV loads a missing text value (a sample with no reference) as
"", which metrics_experiment scores as an empty reference
instead of reporting the column missing.
Args:
name: The dataset's name.
backend: A backend name (default "inmemory") or instance.
**backend_kwargs: Constructor arguments for a backend given by
name, e.g. root_dir.
Raises: ValueError: the testset mixes single- and multi-turn samples.
Source code in src/ragas/testset/synthesizers/testset_schema.py
to_list
Converts the Testset to a list of dictionaries.
Source code in src/ragas/testset/synthesizers/testset_schema.py
from_list
classmethod
from_list(data: List[Dict]) -> Testset
Converts a list of dictionaries to a Testset.
Source code in src/ragas/testset/synthesizers/testset_schema.py
to_pandas
The samples (as to_list()) as a pandas DataFrame.
Source code in src/ragas/testset/synthesizers/testset_schema.py
to_csv
Write the samples (as to_list()) to a CSV file.
Source code in src/ragas/testset/synthesizers/testset_schema.py
to_jsonl
Write the samples (as to_list()) to a JSONL file.
Source code in src/ragas/testset/synthesizers/testset_schema.py
from_jsonl
classmethod
from_jsonl(path: Union[str, Path]) -> Testset
Read a testset written by to_jsonl().
Source code in src/ragas/testset/synthesizers/testset_schema.py
total_tokens
Compute the total tokens used in the evaluation.
Source code in src/ragas/testset/synthesizers/testset_schema.py
total_cost
total_cost(
cost_per_input_token: Optional[float] = None,
cost_per_output_token: Optional[float] = None,
) -> float
Compute the total cost of the evaluation.
Source code in src/ragas/testset/synthesizers/testset_schema.py
from_annotated
classmethod
from_annotated(path: str) -> Testset
Loads a testset from an annotated JSON file.
Source code in src/ragas/testset/synthesizers/testset_schema.py
QueryLength
Bases: str, Enum
Enumeration of query lengths. Available options are: LONG, MEDIUM, SHORT
QueryStyle
Bases: str, Enum
Enumeration of query styles. Available options are: MISSPELLED, PERFECT_GRAMMAR, POOR_GRAMMAR, WEB_SEARCH_LIKE
BaseScenario
Bases: BaseModel
Base class for representing a scenario for generating test samples.
Attributes:
| Name | Type | Description |
|---|---|---|
nodes |
List[Node]
|
List of nodes involved in the scenario. |
style |
QueryStyle
|
The style of the query. |
length |
QueryLength
|
The length of the query. |
persona |
Persona
|
A persona associated with the scenario. |
SingleHopSpecificQuerySynthesizer
dataclass
SingleHopSpecificQuerySynthesizer(
name: str = "single_hop_specific_query_synthesizer",
llm: InstructorBaseRagasLLM = _default_llm_factory(),
llm_context: Optional[str] = None,
generate_query_reference_prompt: BasePrompt = QueryAnswerGenerationPrompt(),
theme_persona_matching_prompt: BasePrompt = ThemesPersonasMatchingPrompt(),
property_name: str = "entities",
)
Bases: SingleHopQuerySynthesizer
MultiHopSpecificQuerySynthesizer
dataclass
MultiHopSpecificQuerySynthesizer(
name: str = "multi_hop_specific_query_synthesizer",
llm: InstructorBaseRagasLLM = _default_llm_factory(),
llm_context: Optional[str] = None,
generate_query_reference_prompt: BasePrompt = QueryAnswerGenerationPrompt(),
property_name: str = "entities",
relation_type: str = "entities_overlap",
relation_overlap_property: str = "overlapped_items",
theme_persona_matching_prompt: BasePrompt = ThemesPersonasMatchingPrompt(),
)
Bases: MultiHopQuerySynthesizer
Synthesize multi-hop queries based on a chunk cluster defined by entity overlap.
get_node_clusters
get_node_clusters(
knowledge_graph: KnowledgeGraph,
) -> List[Tuple]
Identify clusters of nodes based on the specified relationship condition.
Source code in src/ragas/testset/synthesizers/multi_hop/specific.py
ragas.testset.persona
Persona
Bases: BaseModel
A user archetype the generated queries are written from the point of view of.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
A short label for the persona, e.g. |
role_description |
str
|
What this persona knows and what they are trying to find out. This is what steers query generation, so it carries most of the signal. |
PersonaList
Bases: BaseModel
get
get(key: str) -> Optional[Persona]
Return the persona named key, or None if there is none.
Names match ignoring case and surrounding or repeated whitespace, because they come back from an LLM, which often changes either (#84). An exact match wins over a normalised one.
Source code in src/ragas/testset/persona.py
generate_personas_from_kg
generate_personas_from_kg(
kg: KnowledgeGraph,
llm: InstructorBaseRagasLLM,
persona_generation_prompt: PersonaGenerationPrompt = PersonaGenerationPrompt(),
num_personas: int = 3,
filter_fn: Callable[[Node], bool] = default_filter,
callbacks: Callbacks = [],
) -> List[Persona]
Generate personas from a knowledge graph based on cluster of similar document summaries.
parameters:
kg: KnowledgeGraph
The knowledge graph to generate personas from.
llm: InstructorBaseRagasLLM
The LLM to use for generating the persona, such as the output of
llm_factory().
persona_generation_prompt: PersonaGenerationPrompt
The prompt to use for generating the persona.
num_personas: int
The maximum number of personas to generate.
filter_fn: Callable[[Node], bool]
A function to filter nodes in the knowledge graph.
callbacks: Callbacks
The callbacks to use for the generation process.
returns: t.List[Persona] The list of generated personas.
Source code in src/ragas/testset/persona.py
99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 | |