Cancelling Long-Running Tasks
Scoring a large dataset or generating a large testset can take a long time. This page shows how to stop either one cleanly, and how to bound a run with timeouts.
Overview
- Experiments, including
metrics_experiment:arun()is a coroutine. Cancel its task to stop the run. - Test set generation:
generate_with_docs(..., return_executor=True)returns an executor you can cancel.
Cancelling an experiment
Run arun() as a task and cancel the task:
import asyncio
from ragas import metrics_experiment
run = metrics_experiment(metrics)
task = asyncio.create_task(run.arun(dataset, name="long_run"))
# ... later, e.g. when a request is aborted
task.cancel()
try:
exp = await task
except asyncio.CancelledError:
print("The run was cancelled")
When the task is cancelled, the rows still running are cancelled too, and their metric and LLM calls stop at their next await. Nothing is saved: the experiment is written only when every row has finished.
From sync code, run.run(...) stops the same way on Ctrl+C: the KeyboardInterrupt cancels the run before it propagates.
A time limit for the whole run
try:
exp = await asyncio.wait_for(run.arun(dataset, name="nightly"), timeout=1800)
except asyncio.TimeoutError:
print("The run took longer than 30 minutes and was cancelled")
A time limit per metric call
RunConfig(timeout=...) bounds each metric call (and each LLM call's request). A call that takes longer raises TimeoutError; the metric scores NaN on that row and the run goes on, unless you set raise_exceptions=True:
from ragas.run_config import RunConfig
run = metrics_experiment(metrics, run_config=RunConfig(timeout=60))
See RunConfig.
Stopping on the first error
raise_exceptions=True stops the run at the first metric that raises: the rows still running are cancelled and that error is re-raised. Add save_partial=True to save the rows that finished first:
In a web application
Keep the task, and cancel it when the client asks:
import asyncio
import uuid
runs: dict[str, asyncio.Task] = {}
async def start(dataset):
run_id = str(uuid.uuid4())
runs[run_id] = asyncio.create_task(run.arun(dataset, name=run_id))
return run_id
def cancel(run_id: str) -> bool:
task = runs.pop(run_id, None)
if task is None or task.done():
return False
return task.cancel()
Cancelling test set generation
generate_with_docs() (and generate()) take return_executor=True. You get the executor instead of the testset, and can cancel it from another thread:
from ragas.testset.synthesizers.generate import TestsetGenerator
generator = TestsetGenerator(...)
executor = generator.generate_with_docs(
documents=documents,
testset_size=100,
return_executor=True,
)
# Elsewhere:
executor.cancel()
executor.is_cancelled() # True
# Blocks until the jobs finish or are cancelled
results = executor.results()
Limitations
- Cancellation is cooperative: a call stops at its next
await. A sync experiment function runs in a worker thread, which can't be stopped; its row finishes in the background and its result is discarded. - An LLM request already sent may still complete and be billed by the provider.