Fallbacks and reliability

When an endpoint fails before it has sent you anything, Numes tries another endpoint of the same model. You see one response and pay only for the output that reached you. Your client handles what's left: a model with no working endpoint, your own rate limits and network failures.

How health is tracked

Health belongs to one model on one endpoint. A failing endpoint doesn't make the model look unhealthy elsewhere, and one failing model doesn't penalize other models on the same endpoint.

For every request, routing uses only endpoints that are connected and ready, have room for the request, and aren't failing repeatedly. If a model keeps failing on an endpoint, routing stops sending it requests until it passes verification again. Failures caused by you or by the platform, such as your cancellations, invalid requests and parameters an endpoint doesn't declare, aren't counted against the provider.

Among healthy endpoints, price and measured reliability and performance shape the default order. See Provider routing. Each model's page publishes per-endpoint success rate, time to first token and throughput; see Endpoint statistics.

How automatic fallback works

Each request starts with the ordered list of eligible endpoints described in Provider routing. The router then:

1. Holds an upper bound of the request's cost against your balance and limits, and sends the request to the first endpoint.

2. If that endpoint fails before any output reached you, releases the hold.

3. Re-checks the remaining endpoints against current approval, verification, health, capacity, price and your filters.

4. Sends the request to the next eligible endpoint, until one delivers output or the limited number of tries is used up.

Fallback stays within the requested model. Every try respects your max_price, only, ignore, regions and data filters, and the features your request needs.

Which failures fall back

The last rule protects you from paying twice: while a failed endpoint may still be working on your request, the platform doesn't start a second copy elsewhere. After an upstream account problem, routing also skips that endpoint's other listings for the rest of the request.

When the platform confirms that a request was rejected before execution, it releases that request's capacity. If execution is uncertain, capacity stays reserved until the original request deadline and recovery period have passed, or the platform confirms that the request stopped.

When every try fails, the response carries the last error. For example, a final timeout returns 408 timeout, a final provider rate limit returns 429 rate_limited without Retry-After, a malformed provider response returns 502 provider_unavailable, and exhausted capacity returns 503 provider_overloaded. See Errors.

The delivery boundary

Fallback happens only before output is visible to you. After that, routing never switches endpoints and never splices a second provider's output into the first.

• Streaming. Once output has started, a later failure ends the stream with an error chunk: the HTTP status stays 200, the chunk has finish_reason: "error" and an error object, and data: [DONE] follows without a usage chunk. You're charged for the output delivered up to that point.

• Non-streaming. Nothing is visible until the complete response is written, so a failure partway through generation can still fall back.

data: {"id":"0f5b5c9a-3a7e-4c6f-9b1d-2e8a7c4d5f60","object":"chat.completion.chunk","created":1790000000,"model":"openai/gpt-6-luna","choices":[{"index":0,"delta":{"role":"assistant","content":"The first"},"finish_reason":null}]}​data: {"error":{"code":503,"message":"provider became unavailable","metadata":{"error_type":"provider_unavailable"}},"choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}],"id":"0f5b5c9a-3a7e-4c6f-9b1d-2e8a7c4d5f60","object":"chat.completion.chunk","created":1790000004,"model":"openai/gpt-6-luna"}​data: [DONE]

If you need a complete answer, treat finish_reason: "error" as a failed request and retry it yourself. The retry is a new request and is billed separately.

Turn fallbacks off

Set allow_fallbacks: false to make exactly one try. With order, it also drops every unlisted endpoint, so the request goes to the first listed endpoint that's eligible:

curl -X POST "https://api.numes.ai/api/v1/chat/completions" \  -H "Authorization: Bearer $NUMES_API_KEY" \  -H "Content-Type: application/json" \  -d '{    "model": "openai/gpt-6-luna",    "messages": [      {        "role": "user",        "content": "Hello"      }    ],    "provider": {      "order": [        "example-provider/eu-west"      ],      "allow_fallbacks": false    }  }'

Use this to benchmark one endpoint, or when a contract requires a named processor. Expect more 503 responses in exchange.

What fallback costs

Failed tries cost nothing, and their hold is released in full. Only the endpoint that delivered output is charged, at its own price, and the provider of a failed try earns nothing.

A fallback can land on a more expensive endpoint than the first choice, within your max_price. Its hold must also fit your available balance; if it doesn't, routing may continue to a cheaper endpoint.

Inspect the attempts

GET /generation lists every endpoint that was tried, with its outcome, failure class and cost. Use the generation ID from the id of a response or stream chunk, or from the X-Generation-Id header, which is present once output has started.

curl "https://api.numes.ai/api/v1/generation?id=0f5b5c9a-3a7e-4c6f-9b1d-2e8a7c4d5f60" \  -H "Authorization: Bearer $NUMES_API_KEY"

{  "data": {    "id": "0f5b5c9a-3a7e-4c6f-9b1d-2e8a7c4d5f60",    "model": "openai/gpt-6-luna",    "outcome": "success",    "provider_slug": "another-provider",    "endpoint": "another-provider/fast-gpu",    "delivered_output": true,    "total_cost": 0.000154,    "attempts": [      {        "ordinal": 1,        "provider_slug": "example-provider",        "endpoint": "example-provider/eu-west",        "outcome": "failed",        "failure_class": "timeout",        "delivered_output": false,        "total_cost": 0      },      {        "ordinal": 2,        "provider_slug": "another-provider",        "endpoint": "another-provider/fast-gpu",        "outcome": "success",        "delivered_output": true,        "total_cost": 0.000154      }    ]  }}

failure_class says why a try failed, for example provider_unavailable, rate_limited, timeout or cancelled.

A request that fails before any output returns no X-Generation-Id. Find it in Logs: each entry's Route panel lists the endpoints tried, in order, with their outcome, timing and whether they were charged.

Model fallback

Server-side fallback never changes the model. The models array accepts exactly one entry, so OpenRouter-style multi-model fallback is rejected with 400. Substituting another model would change your output and your price without your consent.

To use a second model when the first is unavailable, do it in your client:

Switching models doesn't help with your own 429: traffic limits apply to your key and account, whatever the model. Give every try a fresh X-Request-Id, or omit the header; see Request and generation IDs.

import osimport randomimport timeimport uuid​from openai import APIConnectionError, APIStatusError, OpenAI​client = OpenAI(    base_url="https://api.numes.ai/api/v1",    api_key=os.environ["NUMES_API_KEY"],    max_retries=0,  # this loop owns retries    timeout=630,    # above the server's request limit)​MODELS = ["openai/gpt-6-luna", "author/backup-model"]STOP = {400, 401, 402, 403, 413, 415}SAME_MODEL = {"routing_unavailable", "service_unavailable"}​​def error_type(error: APIStatusError) -> str:    try:        return error.response.json()["error"]["metadata"]["error_type"]    except Exception:        return ""​​def backoff(attempt: int, retry_after: str | None = None) -> float:    if retry_after and retry_after.isdigit():        return float(retry_after)    return random.uniform(0, min(8.0, 0.5 * 2**attempt))​​def complete(messages: list[dict]):    last_error: Exception | None = None    for model in MODELS:        for attempt in range(3):            try:                return client.chat.completions.create(                    model=model,                    messages=messages,                    extra_headers={"X-Request-Id": uuid.uuid4().hex},                )            except APIStatusError as error:                last_error = error                kind = error_type(error)                if error.status_code in STOP or kind == "cancelled":                    raise                if error.status_code == 429 or kind in SAME_MODEL:                    time.sleep(backoff(attempt, error.response.headers.get("retry-after")))                    continue                break  # 502, 503 provider_*, 408 timeout: next model            except APIConnectionError as error:                last_error = error                time.sleep(backoff(attempt))    raise last_error

With streaming, decide how to handle an interrupted stream before you retry. Output already delivered was billed, and a retry on another model starts from the beginning. A common approach is to retry only when no content has arrived yet, and otherwise surface the partial answer.

Build resilient clients

• Retries. Retry only status codes that can succeed on a second try, with exponential backoff and jitter, honoring Retry-After. See Retry strategy.

• Timeouts. Set your client timeout above the server's request limit, and don't use a short idle timeout before the first chunk. See Timeouts.

• Rate limits. Extra keys don't raise your ceiling; smooth bursts with a client-side queue. See Rate limits.

• Faster endpoints. provider.sort: "latency" or "throughput" puts the fastest measured endpoints first. Keep allow_fallbacks at true unless you must pin one endpoint.

• Interrupted requests. A dropped connection or platform maintenance can interrupt a request that already delivered output. Don't repeat actions your tools performed, and check GET /generation for what was delivered and charged before you retry.

Next steps