RateLimit
Requests- and tokens-per-minute limits for LLM and embedding calls (#171).
max_workers limits how many jobs run at once, not how many requests reach
the provider per minute: one metric or experiment row can make several LLM
calls, and instructor's validation retries add more. A RateLimit given to
llm_factory(rate_limit=...) (or LiteLLMEmbeddings(rate_limit=...))
paces every request those objects send, waiting -- never raising -- until the
provider's limit has room::
from ragas.rate_limit import RateLimit
limit = RateLimit(rpm=500, tpm=200_000)
judge = llm_factory("gpt-6-luna", rate_limit=limit)
# the same key's limit, shared: pass the same object
other = llm_factory("gpt-6-luna", temperature=0.7, rate_limit=limit)
Because it sits in the LLM layer, metrics, metrics_experiment and other
experiments, and testset generation all go through it. The counting is done by
pyrate-limiter (sliding windows;
thread- and event-loop-safe).
RateLimit
Requests per minute and, optionally, tokens per minute.
Args:
rpm: Requests per minute. Requests are also spread over the minute
(at most ceil(rpm / 60) per second) rather than sent in one
burst, since providers often enforce limits over shorter windows.
tpm: Tokens per minute. Each request is counted before it is sent as
its prompt's tokens (litellm.token_counter) plus the
max_tokens it asks for, if you set one -- how OpenAI counts
against its limit. Without max_tokens the output is not
counted, so leave room for it.
A request larger than tpm is counted as tpm.
One object is one limit. Share it between the LLMs and embeddings that use the same API key; give a different key its own. Copies of the metrics holding the LLM, such as the optimizer's candidates, share it.
Every request is counted, including instructor's validation retries and
ragas' own retries. Retries made inside a client (the OpenAI SDK's
max_retries, LiteLLM's num_retries) are not seen.
Source code in src/ragas/rate_limit.py
acquire
Block until a request of tokens tokens may be sent.
Source code in src/ragas/rate_limit.py
aacquire
async
Wait, without blocking the event loop, until the request may go.
Source code in src/ragas/rate_limit.py
estimate_chat_tokens
Prompt tokens plus the requested max_tokens of a completion call.
Source code in src/ragas/rate_limit.py
estimate_embedding_tokens
Tokens of the texts sent in one embedding request.
Source code in src/ragas/rate_limit.py
limit_completion_calls
limit_completion_calls(
fn: Callable[..., Any], limit: RateLimit
) -> Any
fn (litellm.acompletion, completion or a Router's), paced.
The wrapper keeps fn's sync/async nature: instructor decides how to
call it from that.