Rate limits

Rate limits cap how many requests and how many input tokens your organization can send per minute. This page explains how both are counted, how your tier raises them, how to read your remaining capacity from response headers, and how to handle a 429.

In short

• There are two limits: requests per minute (RPM) and tokens per minute (TPM). Whichever you reach first applies.
• Both belong to your organization and are shared by every project, key and model in it. Creating more keys or projects does not add capacity.
• Both are measured over a sliding 60-second window, not a calendar minute.
• They rise automatically with your usage tier, which depends on lifetime credit purchases and time since your first payment.
• Going over returns 429 rate_limit_exceeded with a Retry-After header in seconds. Refused requests are never charged.
• Separately, each project can have a monthly spend limit. Reaching it returns 429 project_spend_limit.

The two limits

A request must fit under both limits to run. Before it runs, the platform estimates its input tokens from the size of the request and checks that the estimate fits in what is left of your tokens for the window. The estimate is deliberately on the high side. Once the request finishes, the window holds the tokens it was actually billed for instead.

How the sliding window works

There is no minute boundary that resets your limits at :00. Every request stays in the window for 60 seconds after it was made, then drops out and frees its request slot and its tokens.

For example, Tier 1 allows 300 requests per minute:

• You send 300 requests between 10:00:00 and 10:00:05. All of them run.
• At 10:00:20 you send one more. It is refused with 429 rate_limit_exceeded and Retry-After: 40, because the oldest request leaves the window at 10:01:00.
• Between 10:01:00 and 10:01:05 the slots free up one by one as each of the first requests turns 60 seconds old.

Bursts up to the limit are allowed, but then you wait for the window to clear. Spreading the same traffic evenly avoids the wait: at 300 requests per minute, spacing requests a little more than 0.2 seconds apart keeps you under the limit.

Tokens work the same way: the tokens a request was billed for leave the window 60 seconds after that request. For tokens, Retry-After is the time until the oldest request in the window drops out. If that frees fewer tokens than your next request needs, you get another 429 with a new Retry-After, so retry in a loop rather than once.

How tokens per minute are counted

Tokens per minute count the input tokens you are billed for, so the number you see in usage.input_tokens is exactly what each request took from the window. Requests that fail or are refused bill nothing and take nothing from the window.

Fast and consistent mode

The mode field changes how many tokens a request bills, and so how many requests fit in your tokens per minute:

• fast (the default) reads the record once for all questions. It bills the record plus each question once.
• consistent reads the record again with each question. It bills the tokens it reads, roughly the record times the number of questions, and those tokens count toward tokens per minute.

Take a 1,500-token record with 5 questions of about 40 tokens each:

Consistent mode bills about 4.5 times as many tokens here, so each request costs about 4.5 times as much and takes that much more of your tokens per minute. About the most of these requests each tier can sustain, whichever limit is reached first:

Use consistent mode where answers must not shift with the question set (JevBench-mini: 83.6 consistent, 81.8 fast), and fast mode everywhere else. The response has the same shape in both modes.

Ask related questions in one request

Requests per minute count requests, not questions, and fast mode reads the record only once. With the example above, five questions in one request use one request slot and bill about 1,700 tokens. The same five questions as five separate requests use five slots and bill about 7,700 tokens, because the record is read five times. Put every question about one record into one request, up to your tier's questions per request.

Usage tiers

Your tier sets your limits. Tiers apply to the whole organization, across all of its projects, keys and models.

• Qualifies at: lifetime credit purchases, and for higher tiers the days since your organization's first payment. Both conditions must be met.
• RPM and TPM: requests and billed input tokens per sliding 60-second window.
• Questions / request: the most questions one request can carry. More returns 422 too_many_questions.
• Each top-up: the range for a single credit purchase on that tier. Top-ups start at $5.

How upgrades happen

You never apply for a higher tier. The moment your organization meets both conditions for a tier, it moves up automatically, with nothing to request. The API picks up the new limits within 60 seconds.

• A new organization is on Tier 0 until its first purchase. Tier 0 is sign-up only: it has no API access, and requests return 402 insufficient_credit.
• Any purchase from $5 moves it to Tier 1 straight away: there is no waiting period.
• Higher tiers also need time since your first payment, so they can arrive on a day you buy nothing: the time condition is met and the upgrade applies on its own.

For example, say you buy $5 on your first day and $45 more on day 3. You have now paid $50, but Tier 2 also needs 7 days since your first payment, so it applies 7 days after that first purchase, automatically.

When you move up a tier, the console shows a one-time notice. Limits in the console shows your current tier and limits, your progress toward the next tier on both conditions, and a tier history listing every move and what caused it: a credit purchase, time since your first payment or a refund. A move caused by time can take up to an hour to appear in the history and the notice; the new limits apply before that.

Rate limit headers

Responses from POST /v1/decisions and POST /v1/systemone tell you where you stand, so you can slow down before you reach a limit.

Two more headers are useful when debugging. x-request-id identifies the request (it is on errors too, and matches request_id in an error body), and saanora-processing-ms is the total time the request spent in the platform, in milliseconds.

To see the headers, send any request with curl -i, or read them from your HTTP client:

curl -i https://api.saanora.com/v1/decisions \
  -H "Authorization: Bearer $SAANORA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "clara-1", "state": "Customer writes: I was charged twice for my March invoice.", "questions": {"refund": {"type": "noul", "instructions": "Is the customer asking for money back?"}}}'

HTTP/2 200
content-type: application/json
x-request-id: req_8f2c41d09a7e4b6c1f3a5e70
x-ratelimit-limit-requests: 300
x-ratelimit-remaining-requests: 287
x-ratelimit-reset-requests: 12s
x-ratelimit-limit-tokens: 300000
x-ratelimit-remaining-tokens: 291604
saanora-processing-ms: 241

Read the headers defensively: not every error response carries all of them.

Handling 429 responses

Three different situations return 429. Branch on error.code, not on the message text:

All three have "type": "rate_limit_error". A rate limit response looks like this:

HTTP/2 429
content-type: application/json
retry-after: 9
x-request-id: req_3c9e1a7f5b2d4e8c0a6f1b94
x-ratelimit-limit-requests: 300
x-ratelimit-remaining-requests: 0
x-ratelimit-reset-requests: 9s

{
  "error": {
    "type": "rate_limit_error",
    "code": "rate_limit_exceeded",
    "message": "300 requests per minute on tier 1; lifetime top-ups raise it",
    "param": null
  },
  "request_id": "req_3c9e1a7f5b2d4e8c0a6f1b94"
}

A good client waits for Retry-After, adds a little random jitter so many workers do not retry at the same instant, caps the number of attempts, and stops at once on project_spend_limit. The same loop can retry 502 model_unavailable and 503 with exponential backoff. Retrying costs nothing: failed and refused requests are never charged.

import os
import random
import time

import requests

API_URL = "https://api.saanora.com"
API_KEY = os.environ["SAANORA_API_KEY"]


class SpendLimitReached(Exception):
    """The project reached its monthly spend limit. Retrying will not help until the limit is raised or the month ends."""


def decide(body: dict, max_attempts: int = 6) -> dict:
    """POST /v1/decisions, waiting out rate limits and retrying temporary errors."""
    request_id = None
    for attempt in range(max_attempts):
        r = requests.post(
            f"{API_URL}/v1/decisions",
            headers={"Authorization": f"Bearer {API_KEY}"},
            json=body,
            timeout=120,  # the first request after an idle period can take a while
        )
        if r.status_code == 200:
            return r.json()

        try:
            err = r.json().get("error") or {}
        except ValueError:
            err = {}
        code = err.get("code")
        request_id = r.headers.get("x-request-id")

        if r.status_code == 429 and code == "project_spend_limit":
            raise SpendLimitReached(err.get("message"))
        if r.status_code == 429:
            wait = float(r.headers.get("Retry-After", "1"))  # seconds until capacity frees up
        elif r.status_code in (502, 503):
            wait = min(2 ** attempt, 30)                     # 1, 2, 4, 8, 16 seconds
        else:
            # 400, 401, 402, 404, 413, 422: the same request will fail again
            raise RuntimeError(f"{r.status_code} {code}: {err.get('message')} ({request_id})")

        if attempt + 1 < max_attempts:
            time.sleep(wait + random.uniform(0, 1))         # jitter keeps clients from retrying in lockstep

    raise RuntimeError(f"gave up after {max_attempts} attempts ({request_id})")


if __name__ == "__main__":
    result = decide({
        "model": "clara-1",
        "state": "Customer writes: I was charged twice for my March invoice.",
        "questions": {
            "refund": {"type": "noul", "instructions": "Is the customer asking for money back?"},
        },
    })
    print(result["answers"]["refund"]["noul"], result["usage"])

Retrying a 429 immediately, without waiting, only gets another 429. Other errors (400, 401, 402, 404, 413, 422) will fail again until you change the request or your account; see Errors.

Stay under the limits

• Pace requests evenly. Space requests a little more than 60 ÷ RPM seconds apart (0.2 seconds on Tier 1), or run a queue with a fixed number of workers.
• Watch the headers. When x-ratelimit-remaining-requests or x-ratelimit-remaining-tokens gets close to zero, slow down before you get a 429.
• Batch questions. Put every question about one record into one request.
• Prefer fast mode unless you need consistent answers; it bills far fewer tokens per request.
• Throttle background jobs. Every project shares the organization's limits, so a backfill in a staging project can use up the requests production needs. Give batch jobs their own lower rate in your code.

Per-request limits

These apply to each request on its own, whatever your remaining rate limit:

Project spend limits

Rate limits cap speed. Spend limits cap money. Each project can have its own monthly spend limit, so a runaway job in one project cannot use up the credit every project shares.

• Set it in the console under Projects: enter a monthly limit in US dollars, or leave it empty for no limit. Owners and admins can change it.
• The month is the UTC calendar month. Spend counts from 00:00 UTC on the 1st.
• At the limit, every key in that project gets 429 project_spend_limit until the next UTC month starts or someone raises the limit (the API picks up a changed limit within 60 seconds). Other projects keep working.
• Near the limit, a request is refused if its worst-case cost would take the project over the limit. Before each request a hold covering its worst case is placed, and the spend check uses that hold, so a request can be refused slightly before spend reaches the exact limit.
• Track it on Limits, which shows each project's spend this month against its limit, and in Usage.

A spend limit does not change the rate limit: a project with a limit still shares the organization's requests and tokens per minute. It is also separate from 402 insufficient_credit, which means the organization's prepaid credit cannot cover a request. See Organizations and projects and Billing and credit.

Beyond Tier 5

If you need more than 5,000 requests or 5M tokens per minute, or dedicated capacity, write to support@saanora.com. Tell us the peak requests and tokens per minute you expect.

We can set dedicated limits for your organization: requests and tokens per minute, questions per request and the size of a single top-up, each above what your tier allows. They apply to every key in the organization within a minute, replace the tier's values only where set, and show in the console under Limits with a “Dedicated limits” label.