Skip to content

Google Gemini Integration Guide

Use Gemini models as ragas' evaluator and testset generator through LiteLLM. You need no Google SDK for the LLM.

Setup

Get an API key from Google AI Studio, then:

pip install ragas-ng
export GOOGLE_API_KEY="your-google-api-key"   # or GEMINI_API_KEY
from ragas.llms import llm_factory

llm = llm_factory("gemini/gemini-3.6-flash")

Vertex AI

With Google Cloud credentials (Application Default Credentials, e.g. gcloud auth application-default login):

llm = llm_factory(
    "vertex_ai/gemini-3.6-flash",
    vertex_project="my-project",
    vertex_location="us-central1",
)

Gemini's OpenAI-compatible endpoint

An OpenAI client pointed at Gemini's endpoint works too. The client's settings (headers, timeout, proxy) carry over:

import os

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GOOGLE_API_KEY"],
    base_url="https://generativelanguage.googleapis.com/v1beta/openai/",
)
llm = llm_factory("gemini-3.6-flash", client=client)

google-genai clients are no longer accepted for LLMs

llm_factory("gemini-...", provider="google", client=genai.Client(...)) raises an error naming llm_factory("gemini/gemini-...") instead (#182). Every call goes through LiteLLM, and instructor's native Gemini mode, with its HARM_CATEGORY_JAILBREAK safety-settings bug, isn't used any more. Embeddings still take a google-genai client (below).

Models

  • Flash: gemini-3.6-flash (stable, used on this page) and gemini-3.8-flash (Google's newest Flash model, with better reasoning for complex evaluations).
  • Flash-Lite: gemini-3.5-flash-lite, the fastest and cheapest, for simple or very high-volume metrics.

Older models are shut down (Gemini 1.x and 2.0) or shut down on 2026-12-01 (2.5). See Google's models and deprecations pages.

Embeddings

Metrics such as AnswerCorrectness, AnswerRelevancy and SemanticSimilarity also need embeddings. Use Google's:

from ragas.embeddings import GoogleEmbeddings

embeddings = GoogleEmbeddings(model="gemini-embedding-001")  # pip install google-genai
# or, with your own google-genai client:
# from google import genai
# embeddings = GoogleEmbeddings(client=genai.Client(), model="gemini-embedding-001")

Pass the embeddings to the metric

There are no default models: a metric that needs embeddings, such as AnswerCorrectness, takes them when you create it. With Gemini embeddings, as above, a Gemini-only setup needs no OPENAI_API_KEY. An LLM built on an OpenAI client pointed at Gemini's endpoint can't serve OpenAI embedding models, so use GoogleEmbeddings there too.

Example

from ragas import Dataset, metrics_experiment, summarize
from ragas.dataset_schema import SingleTurnSample
from ragas.embeddings import GoogleEmbeddings
from ragas.llms import llm_factory
from ragas.metrics.collections import AnswerCorrectness, ContextPrecision, Faithfulness

llm = llm_factory("gemini/gemini-3.6-flash")
embeddings = GoogleEmbeddings(model="gemini-embedding-001")

dataset = Dataset(
    "gemini_eval",
    "inmemory",
    data_model=SingleTurnSample,
    data=[
        SingleTurnSample(
            user_input="What is the capital of France?",
            response="Paris is the capital of France.",
            retrieved_contexts=["France is in Western Europe. Paris is its capital."],
            reference="Paris",
        )
    ],
)

run = metrics_experiment(
    [
        ContextPrecision(llm=llm),
        Faithfulness(llm=llm),
        AnswerCorrectness(llm=llm, embeddings=embeddings),
    ]
)
exp = run.run(dataset, name="gemini")
print(summarize(exp))

Or score one sample with a collections metric:

result = await AnswerCorrectness(llm=llm, embeddings=embeddings).ascore(
    user_input="What is the capital of France?",
    response="Paris",
    reference="Paris is the capital of France.",
)

Running the async examples

The examples on this page use await at the top level. That works as is in Jupyter, IPython and other notebooks. In a plain Python script, put the code in an async function and run it with asyncio.run:

import asyncio


async def main(): ...  # the example code, including its `await` lines


asyncio.run(main())

Metrics also have synchronous forms (metric.score(...), metric.batch_score(...)) that you can call directly from a script.

Model parameters

llm = llm_factory(
    "gemini/gemini-3.6-flash", temperature=0.5, max_tokens=2048, top_p=0.9
)

Without them, ragas sends near-deterministic defaults (temperature=0.01, top_p=0.1), which LiteLLM drops if the model rejects them. See LLMs (llm_factory).

Troubleshooting

  • Missing key: set GOOGLE_API_KEY or GEMINI_API_KEY, or pass api_key= to llm_factory.
  • Rate limits: ragas retries transient errors (rate limits, timeouts, 5xx) with backoff according to RunConfig. To stay under your quota, pass rate_limit=RateLimit(rpm=...) to llm_factory (see Requests and tokens per minute), or lower RunConfig(max_workers=...).
  • Model not available: check your region and quota, and that the model is on Google's current models page.

Migrating

Before Now
llm_factory("gemini-...", provider="google", client=genai.Client(api_key=...)) llm_factory("gemini/gemini-3.6-flash")
llm_factory("gemini-...", provider="google", client=genai.GenerativeModel(...)) (legacy SDK) llm_factory("gemini/gemini-3.6-flash")
a google-genai Vertex client llm_factory("vertex_ai/gemini-3.6-flash", vertex_project=..., vertex_location=...)
GoogleEmbeddings(client=genai.Client(), ...) unchanged

Resources