Skip to content

Aligning LLM Evaluators with Human Judgment

Outputs on this page

Outputs of LLM calls on this page come from earlier runs, some with models that have since been retired, and the code now names current models. Scores and generated text from your run will differ.

This tutorial is part of a three-part series on how to use Vertex AI models with Ragas. It is recommended that you have gone through Getting Started: Ragas with Vertex AI, even if you have not, you can easily follow this. You can navigate to the Model Comparison tutorial using the link.

Overview

In this tutorial, you will learn how to train and align your own custom LLM-based metric using Ragas. While LLM-based evaluators offer a powerful means of scoring AI applications, they can sometimes produce judgments that diverge from human expectations due to differences in style, context, or subtle nuances. By following this guide, you will refine your metric so that it more accurately mirrors human judgment.

In this tutorial, you will:

  1. Define a model-based metric using Ragas.
  2. Construct a Dataset from the "helpful" subset of the HHH dataset.
  3. Run an initial evaluation to benchmark the metric’s performance.
  4. Review and annotate 15–20 evaluation examples.
  5. Train the metric using your annotated data.
  6. Reevaluate the metric to observe improvements in alignment with human judgments.

Getting Started

Install Dependencies

%pip install --upgrade --user --quiet google-cloud-aiplatform ragas-ng

Restart runtime

To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.

The restart might take a minute or longer. After it's restarted, continue to the next step.

import IPython

app = IPython.Application.instance()
app.kernel.do_shutdown(True)

Authenticate your notebook environment (Colab only)

If you're running this notebook on Google Colab, run the cell below to authenticate your environment.

import sys

if "google.colab" in sys.modules:
    from google.colab import auth

    auth.authenticate_user()

Set Google Cloud project information and initialize Vertex AI SDK

PROJECT_ID = "[your-project-id]"  # @param {type:"string"}
LOCATION = "us-central1"  # @param {type:"string"}

if not PROJECT_ID or PROJECT_ID == "[your-project-id]":
    raise ValueError("Please set your PROJECT_ID")


import vertexai

vertexai.init(project=PROJECT_ID, location=LOCATION)

Set up eval metrics

LLM-based metrics have tremendous potential but can sometimes misjudge responses compared to human evaluators. To bridge this gap, we align our model-based metric with human judgment using a feedback loop.

Define evaluator_llm

Import the required wrappers and define your evaluator LLM and embedder.

from ragas.embeddings import embedding_factory
from ragas.llms import llm_factory

evaluator_llm = llm_factory("vertex_ai/gemini-3.6-flash")
evaluator_embeddings = embedding_factory(
    "litellm", model="vertex_ai/gemini-embedding-2"
)

Ragas metrics

Ragas offers various model-based metrics that can be fine-tuned to align with human evaluators. For demonstration, we will use the Aspect Critic metricβ€”a user-defined, binary metric. For further details, please refer to the Aspect Critic documentation.

from ragas.metrics.collections import AspectCritic

helpfulness_critic = AspectCritic(
    name="helpfulness",
    definition="Evaluate how helpful the assistant's response is to the user's query.",
    llm=evaluator_llm,
)

You can preview the prompt that will be passed to the LLM (before alignment) by running:

print(helpfulness_critic.get_prompts()["prompt"].instruction)
Output
Evaluate the Input based on the criterial defined. Use only 'Yes' (1) and 'No' (0) as verdict.
Criteria Definition: Evaluate how helpful the assistant's response is to the user's query.

Defining Alignment Score

Since we are using a binary metric, we will measure the alignment using the F1-score. However, depending on the metric you are aligning, you can modify this function accordingly to use other methods to measure the alignment.

from typing import List
from sklearn.metrics import f1_score


def alignment_score(human_score: List[float], llm_score: List[float]) -> float:
    """
    Computes the alignment between human-annotated binary scores and LLM-generated binary scores
    using the F1-score metric.

    Args:
        human_score (List[int]): Binary labels from human evaluation (0 or 1).
        llm_score (List[int]): Binary labels from LLM predictions (0 or 1).

    Returns:
        float: The F1-score measuring alignment.
    """
    return f1_score(human_score, llm_score)

Prepare your dataset

The process_hhh_dataset function prepares data from the HHH dataset for use in training and aligning of the LLM evaluator. Alternate 0 and 1 scores (1 for helpful, 0 for non-helpful) are assigned to each example, indicating which response is preferred.

import numpy as np
from datasets import load_dataset  # pip install datasets
from ragas import Dataset
from ragas.dataset_schema import SingleTurnSample


def process_hhh_dataset(split: str = "helpful", total_count: int = 50):
    dataset = load_dataset(
        "HuggingFaceH4/hhh_alignment", split, split=f"test[:{total_count}]"
    )
    data = []
    expert_scores = []

    for idx, entry in enumerate(dataset):
        # Extract input and target details
        user_input = entry["input"]
        choices = entry["targets"]["choices"]
        labels = entry["targets"]["labels"]

        # Choose target based on whether the index is even or odd
        if idx % 2 == 0:
            target_label = 1
            score = 1
        else:
            target_label = 0
            score = 0

        label_index = labels.index(target_label)

        response = choices[label_index]

        data.append(SingleTurnSample(user_input=user_input, response=response))
        expert_scores.append(score)

    dataset = Dataset(
        f"hhh_{split}", "inmemory", data_model=SingleTurnSample, data=data
    )
    return dataset, expert_scores


eval_dataset, expert_scores = process_hhh_dataset()

Run evaluation

With the dataset and the helpfulness metric defined, you can now score the dataset with metrics_experiment:

from ragas import metrics_experiment

results = metrics_experiment([helpfulness_critic]).run(eval_dataset, name="before")
Running experiment: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 50/50 [00:00<?, ?it/s]

This initial run highlights the level of misalignment present in LLM-based evaluators, which the subsequent training will address.

Next, benchmark the metric's performance against the expert scores. The experiment's rows are in dataset order, so they line up with expert_scores:

human_score = expert_scores
llm_score = results.to_pandas()["helpfulness"].values

initial_score = alignment_score(human_score, llm_score)
initial_score
Output
0.8076923076923077

Review and Annotate

Now that you have obtained the evaluation results, it’s time to review and annotate them. As discussed in blog Aligning LLM as judge with human evaluators, collecting detailed feedback is essential for bridging the gap between LLM-based and human evaluations. Annotate at least 15–20 examples to capture diverse scenarios where the metric might be misaligned.

Here is a sample annotation for the above example. You can download and use it.

Training and Alignment

The next step is to train your metric using the annotated examples. This training process leverages a gradient-free prompt optimization approach that adjusts both instructions and few-shot demonstrations based on the annotated feedback. The optimized prompts replace the metric's own; the prompt objects it held before are left unchanged.

from ragas.config import InstructionConfig, DemonstrationConfig

demo_config = DemonstrationConfig(embedding=evaluator_embeddings)
inst_config = InstructionConfig(llm=evaluator_llm)

helpfulness_critic.train(
    path="annotated_data.json",
    instruction_config=inst_config,
    demonstration_config=demo_config,
)
Overall Progress: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 170/170 [00:00<?, ?it/s]

After training, review the updated instructions that have been optimized for the metric:

print(helpfulness_critic.get_prompts()["prompt"].instruction)
Output
You are provided with a user input and an assistant/model response. Your task is to evaluate the quality of the response based on how well it addresses the user input, considering all requests and constraints. Assign a score/verdict of 1 if the response is helpful, appropriate, and effective, and 0 if it is not. A good response should be accurate, complete, relevant, and provide a tangible improvement or solution, without omitting key information. Provide a brief explanation for your score/verdict.

Re-evaluate

Now that your metric has been aligned with human feedback, re-run the evaluation on your dataset. This step allows you to benchmark the improvements and quantify how well the alignment process has enhanced the metric’s reliability.

from ragas.embeddings import embedding_factory
from ragas.llms import llm_factory

evaluator_llm = llm_factory("vertex_ai/gemini-3.8-flash")
evaluator_embeddings = embedding_factory(
    "litellm", model="vertex_ai/gemini-embedding-2"
)

from ragas import metrics_experiment

results2 = metrics_experiment([helpfulness_critic]).run(eval_dataset, name="after")
Running experiment: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 50/50 [00:00<?, ?it/s]

Benchmark the updated results against the expert scores:

human_score = expert_scores
llm_score = results2.to_pandas()["helpfulness"].values

new_score = alignment_score(human_score, llm_score)
new_score
Output
0.8444444444444444

Checkout other tutorials of this series:

  • Ragas with Vertex AI: Learn how to use Vertex AI models with Ragas to evaluate your LLM workflows.
  • Model Comparison: Compare models provided by VertexAI on RAG-based Q&A task using Ragas metrics.