Limits

Every request must fit several independent limits before it reaches a provider. This page lists each limit, the error it produces, and where to check your own values. Values may change.

Rate limits

Inference requests are rate-limited per key, per account and per key creator, and a request must fit all three. Each scope limits both the number of requests and their token demand: your prompt, plus the full max_completion_tokens you set, plus media inputs. A large output limit therefore uses up your allowance faster. Short bursts are absorbed; sustained traffic is held to a steady rate.

When a limit is reached, the response is 429 rate_limited with the message rate limit reached and a Retry-After header in whole seconds. Wait that long before the next request with the same key. Renaming a key or changing its spending limit doesn't reset its rate limits. A rate-limited request doesn't use up its X-Request-Id.

Request body size

A larger body returns 413 payload_too_large, whether its size is declared in Content-Length or it's sent chunked.

Media inputs

These limits apply to images, PDFs and audio in chat messages, embedding inputs, image references and transcription uploads:

A media item that breaks one of these rules, can't be fetched promptly, or points to a non-public destination returns 400 invalid_request. data: URLs also count toward the route's body limit, and base64 adds about a third, so a 10 MiB body carries about 7.5 MiB of inline media. Use HTTPS URLs for more. A transcription upload must fit both the per-item limit and the route's body limit.

Structural limits

| Hosted tool configuration | 16 settings, 32 KiB | | max_tool_calls | 1 to 128 | | max_completion_tokens, max_tokens | 1 to 131072 |

| session_id, X-Session-Id, prompt_cache_key, user | 256 characters | | X-Request-Id | 128 characters from A-Z a-z 0-9 . _ - | | provider.order, only, ignore, quantizations | 100 entries of up to 256 characters | | provider.regions | 64 unique entries | | Embeddings input | 2048 items; each string up to 4 MiB | | Embeddings dimensions | 1 to 65536 | | Images n | 1 to 10 | | Images input_references | 16 | | Images prompt | 1 MiB | | Images size | auto or WxH, each side 64 to 8192 pixels | | Speech input | 4096 bytes | | Transcription multipart form | 32 parts, one file, other fields up to 8192 bytes |

The number of tools and the size of a function's description and parameters have no limit of their own; they count toward the body limit. Values outside a range return 400; they aren't clamped. A model's maximum output is in top_provider.max_completion_tokens from GET /models.

Timeouts

Streams send no keep-alive frames, so set your client's read timeout to at least 10 minutes.

Credits and reservations

Before a request runs, an upper bound of its cost is held against your balance and spending limits; unused funds are released when the request completes. For token-priced models the hold covers the endpoint's full context and output limits, so a smaller max_completion_tokens doesn't reduce it. The request runs only if the hold fits your available balance, which is your balance minus holds for other running requests. Many concurrent requests can use up the available balance through holds alone, even when the balance covers their actual cost.

Balances are per currency: a USD key draws only on the USD balance, and a BTC key only on the BTC balance. When the hold doesn't fit, the response is 402 insufficient_credits. GET /generation is the authoritative record of what you were charged. See Credits and payments.

Key spending limits

An inference key can carry a spending limit in its currency, with an optional limit_reset:

A request runs only if the key's spend in the current period, plus holds for its running requests, plus this request's hold stays within the limit. Set the limit in Keys or with PATCH /keys/{hash}. GET /key returns the calling key's limit, limit_reset and limit_remaining; limit_remaining doesn't subtract holds for running requests. See API keys.

Account spending limits

Owners and administrators can set one spending limit per currency for the whole account in Billing. It uses the same periods and rule as key limits, applied to all of the account's keys together.

Next steps