Skip to content

Generation

DocumentAdapter module-attribute

DocumentAdapter = t.Callable[[t.Any], DocumentLike]

Converts one of your documents to a DocumentLike; see generate_with_docs.

TestsetGenerator dataclass

TestsetGenerator(
    llm: InstructorBaseRagasLLM,
    embedding_model: BaseRagasEmbedding,
    knowledge_graph: KnowledgeGraph = KnowledgeGraph(),
    persona_list: Optional[List[Persona]] = None,
    llm_context: Optional[str] = None,
)

Generates an evaluation dataset based on given scenarios and parameters.

Attributes:

Name Type Description
llm InstructorBaseRagasLLM

The language model to use for the generation process: the output of llm_factory().

embedding_model BaseRagasEmbedding

The embedding model used to build the knowledge graph, such as the output of embedding_factory().

knowledge_graph KnowledgeGraph, default empty

The knowledge graph to use for the generation process.

llm_context Optional[str], default None

Additional context to provide to the LLM when generating responses. This context will be used to guide how the LLM generates queries and answers.

generate_with_docs

generate_with_docs(
    documents: Sequence[Any],
    testset_size: int,
    transforms: Optional[Transforms] = None,
    transforms_llm: Optional[InstructorBaseRagasLLM] = None,
    transforms_embedding_model: Optional[
        BaseRagasEmbedding
    ] = None,
    query_distribution: Optional[QueryDistribution] = None,
    run_config: Optional[RunConfig] = None,
    callbacks: Optional[Callbacks] = None,
    token_usage_parser: Optional[TokenUsageParser] = None,
    with_debugging_logs=False,
    raise_exceptions: bool = True,
    return_executor: bool = False,
    document_adapter: Optional[DocumentAdapter] = None,
    num_personas: int = 3,
    batch_size: Optional[int] = None,
) -> Union[Testset, Executor]

Generates an evaluation dataset from whole documents.

transforms split, enrich and link the documents into a knowledge graph, then queries are generated from it.

Parameters:

Name Type Description Default
documents Sequence

The source documents: objects with page_content and metadata attributes (ragas.testset.document.Document, a LangChain Document), strings, or objects that document_adapter converts. Documents with no text are skipped.

required
testset_size int

The number of test samples to generate

required
transforms Optional[Transforms]

Custom transforms to apply to the documents, by default None

None
transforms_llm Optional[InstructorBaseRagasLLM]

LLM to use for transforms if different from instance LLM, by default None

None
transforms_embedding_model Optional[BaseRagasEmbedding]

Embedding model to use for transforms if different from instance model, by default None

None
query_distribution Optional[QueryDistribution]

Distribution of query types to generate, by default None

None
run_config Optional[RunConfig]

Configuration for the generation run, by default None

None
callbacks Optional[Callbacks]

Callbacks to use during generation, by default None

None
token_usage_parser Optional[TokenUsageParser]

Parse the LLMResult object and return a TokenUsage object. This is used to calculate the cost of the generation process.

None
with_debugging_logs bool

Whether to include debug logs, by default False

False
raise_exceptions bool

Whether to raise exceptions during generation, by default True

True
return_executor bool

If True, returns the Executor instance instead of running generation. The returned executor can be used to cancel execution by calling executor.cancel(). To get results, call executor.results(). Default is False.

False
document_adapter Optional[Callable[[Any], DocumentLike]]

Converts each document that is not a string to an object with page_content and metadata, for documents whose type uses other names. For LlamaIndex documents: lambda d: Document(page_content=d.text, metadata=d.metadata). By default the documents are used as they are.

None
num_personas int

The number of personas to generate or use, as in :meth:generate.

3
batch_size Optional[int]

How large generation batches should be, as in :meth:generate. By default no batching is done.

None

Returns:

Type Description
Testset or Executor

If return_executor is False, returns the generated evaluation dataset. If return_executor is True, returns the Executor instance for cancellable execution.

Raises:

Type Description
ValueError

If transforms is not given and no LLM or embedding model is provided, either on the generator or as arguments

TypeError

If an LLM is not an llm_factory() LLM, or an embedding model is not a BaseRagasEmbedding

Source code in src/ragas/testset/synthesizers/generate.py
def generate_with_docs(
    self,
    documents: t.Sequence[t.Any],
    testset_size: int,
    transforms: t.Optional[Transforms] = None,
    transforms_llm: t.Optional[InstructorBaseRagasLLM] = None,
    transforms_embedding_model: t.Optional[BaseRagasEmbedding] = None,
    query_distribution: t.Optional[QueryDistribution] = None,
    run_config: t.Optional[RunConfig] = None,
    callbacks: t.Optional[Callbacks] = None,
    token_usage_parser: t.Optional[TokenUsageParser] = None,
    with_debugging_logs=False,
    raise_exceptions: bool = True,
    return_executor: bool = False,
    document_adapter: t.Optional[DocumentAdapter] = None,
    num_personas: int = 3,
    batch_size: t.Optional[int] = None,
) -> t.Union[Testset, Executor]:
    """
    Generates an evaluation dataset from whole documents.

    ``transforms`` split, enrich and link the documents into a knowledge
    graph, then queries are generated from it.

    Parameters
    ----------
    documents : Sequence
        The source documents: objects with ``page_content`` and
        ``metadata`` attributes (``ragas.testset.document.Document``, a
        LangChain ``Document``), strings, or objects that
        ``document_adapter`` converts. Documents with no text are skipped.
    testset_size : int
        The number of test samples to generate
    transforms : Optional[Transforms], optional
        Custom transforms to apply to the documents, by default None
    transforms_llm : Optional[InstructorBaseRagasLLM], optional
        LLM to use for transforms if different from instance LLM, by default None
    transforms_embedding_model : Optional[BaseRagasEmbedding], optional
        Embedding model to use for transforms if different from instance model, by default None
    query_distribution : Optional[QueryDistribution], optional
        Distribution of query types to generate, by default None
    run_config : Optional[RunConfig], optional
        Configuration for the generation run, by default None
    callbacks : Optional[Callbacks], optional
        Callbacks to use during generation, by default None
    token_usage_parser : Optional[TokenUsageParser], optional
        Parse the LLMResult object and return a TokenUsage object. This is used to
        calculate the cost of the generation process.
    with_debugging_logs : bool, optional
        Whether to include debug logs, by default False
    raise_exceptions : bool, optional
        Whether to raise exceptions during generation, by default True
    return_executor : bool, optional
        If True, returns the Executor instance instead of running generation.
        The returned executor can be used to cancel execution by calling executor.cancel().
        To get results, call executor.results(). Default is False.
    document_adapter : Optional[Callable[[Any], DocumentLike]], optional
        Converts each document that is not a string to an object with
        ``page_content`` and ``metadata``, for documents whose type uses
        other names. For LlamaIndex documents:
        ``lambda d: Document(page_content=d.text, metadata=d.metadata)``.
        By default the documents are used as they are.
    num_personas : int, default 3
        The number of personas to generate or use, as in :meth:`generate`.
    batch_size : Optional[int], optional
        How large generation batches should be, as in :meth:`generate`.
        By default no batching is done.

    Returns
    -------
    Testset or Executor
        If return_executor is False, returns the generated evaluation dataset.
        If return_executor is True, returns the Executor instance for cancellable execution.

    Raises
    ------
    ValueError
        If ``transforms`` is not given and no LLM or embedding model is
        provided, either on the generator or as arguments
    TypeError
        If an LLM is not an ``llm_factory()`` LLM, or an embedding model
        is not a ``BaseRagasEmbedding``
    """

    kg, transforms = self._build_kg(
        documents,
        NodeType.DOCUMENT,
        document_adapter,
        transforms,
        transforms_llm,
        transforms_embedding_model,
    )
    return self._transform_and_generate(
        kg,
        transforms,
        testset_size=testset_size,
        query_distribution=query_distribution,
        num_personas=num_personas,
        run_config=run_config,
        batch_size=batch_size,
        callbacks=callbacks,
        token_usage_parser=token_usage_parser,
        with_debugging_logs=with_debugging_logs,
        raise_exceptions=raise_exceptions,
        return_executor=return_executor,
    )

generate_with_chunks

generate_with_chunks(
    chunks: Sequence[Any],
    testset_size: int,
    transforms: Optional[Transforms] = None,
    transforms_llm: Optional[InstructorBaseRagasLLM] = None,
    transforms_embedding_model: Optional[
        BaseRagasEmbedding
    ] = None,
    query_distribution: Optional[QueryDistribution] = None,
    run_config: Optional[RunConfig] = None,
    callbacks: Optional[Callbacks] = None,
    token_usage_parser: Optional[TokenUsageParser] = None,
    with_debugging_logs=False,
    raise_exceptions: bool = True,
    return_executor: bool = False,
    document_adapter: Optional[DocumentAdapter] = None,
    num_personas: int = 3,
    batch_size: Optional[int] = None,
) -> Union[Testset, Executor]

Generates an evaluation dataset based on provided pre-chunked documents.

This method allows users to skip the internal chunking process by providing documents that are already chunked. The input documents are treated as NodeType.CHUNK directly.

Parameters:

Name Type Description Default
chunks Sequence

The chunks: strings, objects with page_content and metadata attributes, or objects that document_adapter converts. Chunks with no text are skipped.

required
testset_size int

The number of test samples to generate

required
transforms Optional[Transforms]

Custom transforms to apply to the chunks, by default None

None
transforms_llm Optional[InstructorBaseRagasLLM]

LLM to use for transforms if different from instance LLM, by default None

None
transforms_embedding_model Optional[BaseRagasEmbedding]

Embedding model to use for transforms if different from instance model, by default None

None
query_distribution Optional[QueryDistribution]

Distribution of query types to generate, by default None

None
run_config Optional[RunConfig]

Configuration for the generation run, by default None

None
callbacks Optional[Callbacks]

Callbacks to use during generation, by default None

None
token_usage_parser Optional[TokenUsageParser]

Parse the LLMResult object and return a TokenUsage object.

None
with_debugging_logs bool

Whether to include debug logs, by default False

False
raise_exceptions bool

Whether to raise exceptions during generation, by default True

True
return_executor bool

If True, returns the Executor instance instead of running generation.

False
document_adapter Optional[Callable[[Any], DocumentLike]]

Converts each chunk that is not a string, as in :meth:generate_with_docs.

None
num_personas int

The number of personas to generate or use, as in :meth:generate.

3
batch_size Optional[int]

How large generation batches should be, as in :meth:generate. By default no batching is done.

None

Returns:

Type Description
Testset or Executor

If return_executor is False, returns the generated evaluation dataset. If return_executor is True, returns the Executor instance.

Source code in src/ragas/testset/synthesizers/generate.py
def generate_with_chunks(
    self,
    chunks: t.Sequence[t.Any],
    testset_size: int,
    transforms: t.Optional[Transforms] = None,
    transforms_llm: t.Optional[InstructorBaseRagasLLM] = None,
    transforms_embedding_model: t.Optional[BaseRagasEmbedding] = None,
    query_distribution: t.Optional[QueryDistribution] = None,
    run_config: t.Optional[RunConfig] = None,
    callbacks: t.Optional[Callbacks] = None,
    token_usage_parser: t.Optional[TokenUsageParser] = None,
    with_debugging_logs=False,
    raise_exceptions: bool = True,
    return_executor: bool = False,
    document_adapter: t.Optional[DocumentAdapter] = None,
    num_personas: int = 3,
    batch_size: t.Optional[int] = None,
) -> t.Union[Testset, Executor]:
    """
    Generates an evaluation dataset based on provided pre-chunked documents.

    This method allows users to skip the internal chunking process by providing
    documents that are already chunked. The input documents are treated as
    `NodeType.CHUNK` directly.

    Parameters
    ----------
    chunks : Sequence
        The chunks: strings, objects with ``page_content`` and
        ``metadata`` attributes, or objects that ``document_adapter``
        converts. Chunks with no text are skipped.
    testset_size : int
        The number of test samples to generate
    transforms : Optional[Transforms], optional
        Custom transforms to apply to the chunks, by default None
    transforms_llm : Optional[InstructorBaseRagasLLM], optional
        LLM to use for transforms if different from instance LLM, by default None
    transforms_embedding_model : Optional[BaseRagasEmbedding], optional
        Embedding model to use for transforms if different from instance model, by default None
    query_distribution : Optional[QueryDistribution], optional
        Distribution of query types to generate, by default None
    run_config : Optional[RunConfig], optional
        Configuration for the generation run, by default None
    callbacks : Optional[Callbacks], optional
        Callbacks to use during generation, by default None
    token_usage_parser : Optional[TokenUsageParser], optional
        Parse the LLMResult object and return a TokenUsage object.
    with_debugging_logs : bool, optional
        Whether to include debug logs, by default False
    raise_exceptions : bool, optional
        Whether to raise exceptions during generation, by default True
    return_executor : bool, optional
        If True, returns the Executor instance instead of running generation.
    document_adapter : Optional[Callable[[Any], DocumentLike]], optional
        Converts each chunk that is not a string, as in
        :meth:`generate_with_docs`.
    num_personas : int, default 3
        The number of personas to generate or use, as in :meth:`generate`.
    batch_size : Optional[int], optional
        How large generation batches should be, as in :meth:`generate`.
        By default no batching is done.

    Returns
    -------
    Testset or Executor
        If return_executor is False, returns the generated evaluation dataset.
        If return_executor is True, returns the Executor instance.
    """

    kg, transforms = self._build_kg(
        chunks,
        NodeType.CHUNK,
        document_adapter,
        transforms,
        transforms_llm,
        transforms_embedding_model,
    )
    return self._transform_and_generate(
        kg,
        transforms,
        testset_size=testset_size,
        query_distribution=query_distribution,
        num_personas=num_personas,
        run_config=run_config,
        batch_size=batch_size,
        callbacks=callbacks,
        token_usage_parser=token_usage_parser,
        with_debugging_logs=with_debugging_logs,
        raise_exceptions=raise_exceptions,
        return_executor=return_executor,
    )

generate

generate(
    testset_size: int,
    query_distribution: Optional[QueryDistribution] = None,
    num_personas: int = 3,
    run_config: Optional[RunConfig] = None,
    batch_size: Optional[int] = None,
    callbacks: Optional[Callbacks] = None,
    token_usage_parser: Optional[TokenUsageParser] = None,
    with_debugging_logs=False,
    raise_exceptions: bool = True,
    return_executor: bool = False,
) -> Union[Testset, Executor]

Generate an evaluation dataset based on given scenarios and parameters.

Parameters:

Name Type Description Default
testset_size int

The number of samples to generate.

required
query_distribution Optional[QueryDistribution]

A list of tuples containing scenario simulators and their probabilities. If None, default simulators will be used.

None
num_personas int

The number of personas to generate or use from the persona_list.

3
run_config Optional[RunConfig]

Configuration for running the generation process.

None
batch_size Optional[int]

How large should batches be. If set to None (default), no batching is done.

None
callbacks Optional[Callbacks]

Langchain style callbacks to use for the generation process. You can use this to log the generation process or add other metadata.

None
token_usage_parser Optional[TokenUsageParser]

Parse the LLMResult object and return a TokenUsage object. This is used to calculate the cost of the generation process.

None
with_debugging_logs bool

If True, enable debug logging for various components.

False
raise_exceptions bool

If True, raise exceptions during the generation process.

True
return_executor bool

If True, returns the Executor instance instead of running generation. The returned executor can be used to cancel execution by calling executor.cancel(). To get results, call executor.results().

False

Returns:

Type Description
Testset or Executor

If return_executor is False, returns a dataset containing the generated TestsetSamples. If return_executor is True, returns the Executor instance for cancellable execution.

Notes

This function performs the following steps: 1. Set up scenarios and debug logging if required. 2. Generate scenarios using an Executor. 3. Calculate split values for different scenario types. 4. Generate samples for each scenario. 5. Compile the results into a Testset.

Source code in src/ragas/testset/synthesizers/generate.py
def generate(
    self,
    testset_size: int,
    query_distribution: t.Optional[QueryDistribution] = None,
    num_personas: int = 3,
    run_config: t.Optional[RunConfig] = None,
    batch_size: t.Optional[int] = None,
    callbacks: t.Optional[Callbacks] = None,
    token_usage_parser: t.Optional[TokenUsageParser] = None,
    with_debugging_logs=False,
    raise_exceptions: bool = True,
    return_executor: bool = False,
) -> t.Union[Testset, Executor]:
    """
    Generate an evaluation dataset based on given scenarios and parameters.

    Parameters
    ----------
    testset_size : int
        The number of samples to generate.
    query_distribution : Optional[QueryDistribution], optional
        A list of tuples containing scenario simulators and their probabilities.
        If None, default simulators will be used.
    num_personas : int, default 3
        The number of personas to generate or use from the persona_list.
    run_config : Optional[RunConfig], optional
        Configuration for running the generation process.
    batch_size: int, optional
        How large should batches be.  If set to None (default), no batching is done.
    callbacks : Optional[Callbacks], optional
        Langchain style callbacks to use for the generation process. You can use
        this to log the generation process or add other metadata.
    token_usage_parser : Optional[TokenUsageParser], optional
        Parse the LLMResult object and return a TokenUsage object. This is used to
        calculate the cost of the generation process.
    with_debugging_logs : bool, default False
        If True, enable debug logging for various components.
    raise_exceptions : bool, default True
        If True, raise exceptions during the generation process.
    return_executor : bool, default False
        If True, returns the Executor instance instead of running generation.
        The returned executor can be used to cancel execution by calling executor.cancel().
        To get results, call executor.results().

    Returns
    -------
    Testset or Executor
        If return_executor is False, returns a dataset containing the generated TestsetSamples.
        If return_executor is True, returns the Executor instance for cancellable execution.

    Notes
    -----
    This function performs the following steps:
    1. Set up scenarios and debug logging if required.
    2. Generate scenarios using an Executor.
    3. Calculate split values for different scenario types.
    4. Generate samples for each scenario.
    5. Compile the results into a Testset.
    """
    return self._transform_and_generate(
        None,
        None,
        testset_size=testset_size,
        query_distribution=query_distribution,
        num_personas=num_personas,
        run_config=run_config,
        batch_size=batch_size,
        callbacks=callbacks,
        token_usage_parser=token_usage_parser,
        with_debugging_logs=with_debugging_logs,
        raise_exceptions=raise_exceptions,
        return_executor=return_executor,
    )