Skip to content

OCI Gen AI Integration

This guide shows how to evaluate with Oracle Cloud Infrastructure (OCI) Generative AI models in Ragas.

Ragas reaches OCI through LiteLLM's OCI provider: pass an oci/<model id> model string to llm_factory(). There is no OCI-specific wrapper and no OCI SDK to install.

Setup

1. Configure OCI credentials

LiteLLM signs each request with your OCI API key. Set these environment variables:

export OCI_USER=ocid1.user.oc1..example
export OCI_FINGERPRINT=your_fingerprint
export OCI_TENANCY=ocid1.tenancy.oc1..example
export OCI_KEY_FILE=~/.oci/private_key.pem   # or OCI_KEY with the PEM contents
export OCI_COMPARTMENT_ID=ocid1.compartment.oc1..example
export OCI_REGION=us-chicago-1               # defaults to us-ashburn-1

The values are the ones in your ~/.oci/config profile. You can also pass any of them to llm_factory() as keyword arguments (oci_user=, oci_fingerprint=, oci_tenancy=, oci_key_file=, oci_compartment_id=, oci_region=), which LiteLLM forwards with each request.

2. Pick a model

Use the model ID from the OCI Generative AI console, prefixed with oci/, for example oci/cohere.command-latest or oci/meta.llama-3.3-70b-instruct. Check the OCI documentation for the models available in your region.

Create the evaluator LLM

from ragas.llms import llm_factory

llm = llm_factory("oci/cohere.command-latest")

# Or pass the credentials explicitly instead of through the environment
llm = llm_factory(
    "oci/meta.llama-3.3-70b-instruct",
    oci_region="us-chicago-1",
    oci_compartment_id="ocid1.compartment.oc1..example",
)

Ragas asks the model for structured output through tool calling. If a model returns no tool call, use JSON mode instead:

import instructor

llm = llm_factory("oci/cohere.command-latest", mode=instructor.Mode.JSON)

For embeddings (needed by metrics such as AnswerRelevancy), use an OCI embedding model through LiteLLM as well:

from ragas.embeddings import LiteLLMEmbeddings

embeddings = LiteLLMEmbeddings(model="oci/cohere.embed-english-v3.0")

Score a single sample

from ragas.metrics.collections import Faithfulness

faithfulness = Faithfulness(llm=llm)

result = await faithfulness.ascore(
    user_input="What is the capital of France?",
    response="Paris is the capital of France.",
    retrieved_contexts=["France is a country in Europe. Its capital is Paris."],
)
print(result.value, result.reason)

Evaluate a dataset

from ragas import Dataset, metrics_experiment, summarize
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics.collections import (
    AnswerRelevancy,
    ContextPrecision,
    Faithfulness,
)

rows = [
    {
        "user_input": "What is the capital of France?",
        "response": "Paris is the capital of France.",
        "retrieved_contexts": ["France is a country in Europe. Its capital is Paris."],
        "reference": "Paris",
    },
    {
        "user_input": "Who wrote Romeo and Juliet?",
        "response": "William Shakespeare wrote Romeo and Juliet.",
        "retrieved_contexts": ["Romeo and Juliet is a play by William Shakespeare."],
        "reference": "William Shakespeare",
    },
]
dataset = Dataset(
    "oci_eval",
    "inmemory",
    data_model=SingleTurnSample,
    data=[SingleTurnSample(**row) for row in rows],
)

run = metrics_experiment(
    [
        Faithfulness(llm=llm),
        AnswerRelevancy(llm=llm, embeddings=embeddings),
        ContextPrecision(llm=llm),
    ]
)
exp = run.run(dataset, name="oci")
print(summarize(exp))

Rate limits

OCI Generative AI enforces per-model rate limits. Pace the requests with rate_limit=:

from ragas.rate_limit import RateLimit

llm = llm_factory("oci/cohere.command-latest", rate_limit=RateLimit(rpm=60))

Troubleshooting

  • Missing required parameters: oci_user, ...: one of the credential variables above is unset.
  • Model not found: the model is not offered in OCI_REGION, or your compartment has no access to it.
  • Permission errors: your user needs the IAM policies for Generative AI in the compartment.

Turn on LiteLLM's debug logging to see the request it sends:

import litellm

litellm._turn_on_debug()

See also: LLMs (llm_factory) and the OCI Generative AI docs.