Google Gemini Integration Guide
Use Gemini models as ragas' evaluator and testset generator through LiteLLM. You need no Google SDK for the LLM.
Setup
Get an API key from Google AI Studio, then:
Vertex AI
With Google Cloud credentials (Application Default Credentials, e.g. gcloud auth application-default login):
llm = llm_factory(
"vertex_ai/gemini-3.6-flash",
vertex_project="my-project",
vertex_location="us-central1",
)
Gemini's OpenAI-compatible endpoint
An OpenAI client pointed at Gemini's endpoint works too. The client's settings (headers, timeout, proxy) carry over:
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["GOOGLE_API_KEY"],
base_url="https://generativelanguage.googleapis.com/v1beta/openai/",
)
llm = llm_factory("gemini-3.6-flash", client=client)
google-genai clients are no longer accepted for LLMs
llm_factory("gemini-...", provider="google", client=genai.Client(...)) raises an
error naming llm_factory("gemini/gemini-...") instead (#182). Every call goes
through LiteLLM, and instructor's native Gemini mode, with its
HARM_CATEGORY_JAILBREAK safety-settings bug, isn't used any more.
Embeddings still take a google-genai client (below).
Models
- Flash:
gemini-3.6-flash(stable, used on this page) andgemini-3.8-flash(Google's newest Flash model, with better reasoning for complex evaluations). - Flash-Lite:
gemini-3.5-flash-lite, the fastest and cheapest, for simple or very high-volume metrics.
Older models are shut down (Gemini 1.x and 2.0) or shut down on 2026-12-01 (2.5). See Google's models and deprecations pages.
Embeddings
Metrics such as AnswerCorrectness, AnswerRelevancy and SemanticSimilarity also need embeddings. Use Google's:
from ragas.embeddings import GoogleEmbeddings
embeddings = GoogleEmbeddings(model="gemini-embedding-001") # pip install google-genai
# or, with your own google-genai client:
# from google import genai
# embeddings = GoogleEmbeddings(client=genai.Client(), model="gemini-embedding-001")
Pass the embeddings to the metric
There are no default models: a metric that needs embeddings, such as
AnswerCorrectness, takes them when you create it. With Gemini
embeddings, as above, a Gemini-only setup needs no OPENAI_API_KEY. An
LLM built on an OpenAI client pointed at Gemini's endpoint can't serve
OpenAI embedding models, so use GoogleEmbeddings there too.
Example
from ragas import Dataset, metrics_experiment, summarize
from ragas.dataset_schema import SingleTurnSample
from ragas.embeddings import GoogleEmbeddings
from ragas.llms import llm_factory
from ragas.metrics.collections import AnswerCorrectness, ContextPrecision, Faithfulness
llm = llm_factory("gemini/gemini-3.6-flash")
embeddings = GoogleEmbeddings(model="gemini-embedding-001")
dataset = Dataset(
"gemini_eval",
"inmemory",
data_model=SingleTurnSample,
data=[
SingleTurnSample(
user_input="What is the capital of France?",
response="Paris is the capital of France.",
retrieved_contexts=["France is in Western Europe. Paris is its capital."],
reference="Paris",
)
],
)
run = metrics_experiment(
[
ContextPrecision(llm=llm),
Faithfulness(llm=llm),
AnswerCorrectness(llm=llm, embeddings=embeddings),
]
)
exp = run.run(dataset, name="gemini")
print(summarize(exp))
Or score one sample with a collections metric:
result = await AnswerCorrectness(llm=llm, embeddings=embeddings).ascore(
user_input="What is the capital of France?",
response="Paris",
reference="Paris is the capital of France.",
)
Running the async examples
The examples on this page use await at the top level. That works as is in Jupyter, IPython and other notebooks. In a plain Python script, put the code in an async function and run it with asyncio.run:
import asyncio
async def main(): ... # the example code, including its `await` lines
asyncio.run(main())
Metrics also have synchronous forms (metric.score(...), metric.batch_score(...)) that you can call directly from a script.
Model parameters
Without them, ragas sends near-deterministic defaults (temperature=0.01, top_p=0.1), which LiteLLM drops if the model rejects them. See LLMs (llm_factory).
Troubleshooting
- Missing key: set
GOOGLE_API_KEYorGEMINI_API_KEY, or passapi_key=tollm_factory. - Rate limits: ragas retries transient errors (rate limits, timeouts, 5xx) with backoff according to
RunConfig. To stay under your quota, passrate_limit=RateLimit(rpm=...)tollm_factory(see Requests and tokens per minute), or lowerRunConfig(max_workers=...). - Model not available: check your region and quota, and that the model is on Google's current models page.
Migrating
| Before | Now |
|---|---|
llm_factory("gemini-...", provider="google", client=genai.Client(api_key=...)) |
llm_factory("gemini/gemini-3.6-flash") |
llm_factory("gemini-...", provider="google", client=genai.GenerativeModel(...)) (legacy SDK) |
llm_factory("gemini/gemini-3.6-flash") |
| a google-genai Vertex client | llm_factory("vertex_ai/gemini-3.6-flash", vertex_project=..., vertex_location=...) |
GoogleEmbeddings(client=genai.Client(), ...) |
unchanged |