# Fallbacks and reliability

> How Numes keeps failing endpoints out of routing, fails over between endpoints of a model before any output, and how to add model fallback.

Source: https://0.0.0.0:3000/docs/routing/fallbacks
Documentation index: https://0.0.0.0:3000/docs/llms.txt

When an endpoint fails before it has sent you anything, Numes tries another endpoint of the same model. You see one response and pay only for the output that reached you. Your client handles what's left: a model with no working endpoint, your own rate limits and network failures.

## How health is tracked

Health belongs to one model on one endpoint. A failing endpoint doesn't make the model look unhealthy elsewhere, and one failing model doesn't penalize other models on the same endpoint.

For every request, routing uses only endpoints that are connected and ready, have room for the request, and aren't failing repeatedly. If a model keeps failing on an endpoint, routing stops sending it requests until it passes [verification](/docs/routing/verification) again. Failures caused by you or by the platform, such as your cancellations, invalid requests and parameters an endpoint doesn't declare, aren't counted against the provider.

Among healthy endpoints, price and measured reliability and performance shape the default order. See [Provider routing](/docs/routing/provider-routing#default-routing). Each model's page publishes per-endpoint success rate, time to first token and throughput; see [Endpoint statistics](/docs/models#endpoint-statistics).

## How automatic fallback works

Each request starts with the ordered list of eligible endpoints described in [Provider routing](/docs/routing/provider-routing). The router then:

1. Holds an upper bound of the request's cost against your balance and limits, and sends the request to the first endpoint.
2. If that endpoint fails before any output reached you, releases the hold.
3. Re-checks the remaining endpoints against current approval, verification, health, capacity, price and your filters.
4. Sends the request to the next eligible endpoint, until one delivers output or the limited number of tries is used up.

Fallback stays within the requested model. Every try respects your `max_price`, `only`, `ignore`, `regions` and data filters, and the features your request needs.

## Which failures fall back

| Falls back to the next endpoint | Ends the request |
| --- | --- |
| The endpoint is unreachable, disconnects or times out | You cancel or disconnect |
| The provider reports a rate limit or runs out of capacity | Output was already delivered to you |
| The response is malformed or empty | Your key or account spending limit is reached |
| The model is missing at the provider | The request itself is invalid |
| The provider's upstream account is out of credit, rejects its key or reached a usage cap | `allow_fallbacks` is `false` |
| The endpoint rejects your prompt as too long, and an endpoint with a larger context window is eligible | No larger context window is left after such a rejection |
| The endpoint refuses an undeclared parameter or cannot represent the supplied sampler, format or conversation | Every endpoint refused the input (`400`) |
| The endpoint's content policy blocks the input | Your account's request policy blocks it before dispatch, or the last endpoint blocks it (`403`) |
| Your balance can't cover this endpoint, but a cheaper one may fit | The platform can't confirm the failed endpoint stopped working on your request |

The last rule protects you from paying twice: while a failed endpoint may still be working on your request, the platform doesn't start a second copy elsewhere. After an upstream account problem, routing also skips that endpoint's other listings for the rest of the request.

When the platform confirms that a request was rejected before execution, it releases that request's capacity. If execution is uncertain, capacity stays reserved until the original request deadline and recovery period have passed, or the platform confirms that the request stopped.

When every try fails, the response carries the last error. For example, a final timeout returns `408 timeout`, a final provider rate limit returns `429 rate_limited` without `Retry-After`, a malformed provider response returns `502 provider_unavailable`, and exhausted capacity returns `503 provider_overloaded`. See [Errors](/docs/api-reference/errors).

## The delivery boundary

Fallback happens only before output is visible to you. After that, routing never switches endpoints and never splices a second provider's output into the first.

- **Streaming.** Once output has started, a later failure ends the stream with an error chunk: the HTTP status stays `200`, the chunk has `finish_reason: "error"` and an `error` object, and `data: [DONE]` follows without a usage chunk. You're charged for the output delivered up to that point.
- **Non-streaming.** Nothing is visible until the complete response is written, so a failure partway through generation can still fall back.

```sse title="Stream interrupted after output"
data: {"id":"0f5b5c9a-3a7e-4c6f-9b1d-2e8a7c4d5f60","object":"chat.completion.chunk","created":1790000000,"model":"openai/gpt-6-luna","choices":[{"index":0,"delta":{"role":"assistant","content":"The first"},"finish_reason":null}]}

data: {"error":{"code":503,"message":"provider became unavailable","metadata":{"error_type":"provider_unavailable"}},"choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}],"id":"0f5b5c9a-3a7e-4c6f-9b1d-2e8a7c4d5f60","object":"chat.completion.chunk","created":1790000004,"model":"openai/gpt-6-luna"}

data: [DONE]
```

If you need a complete answer, treat `finish_reason: "error"` as a failed request and retry it yourself. The retry is a new request and is billed separately.

## Turn fallbacks off

Set `allow_fallbacks: false` to make exactly one try. With `order`, it also drops every unlisted endpoint, so the request goes to the first listed endpoint that's eligible:

```request
{
  "method": "POST",
  "path": "/chat/completions",
  "title": "Exactly one try on one endpoint",
  "body": {
    "model": "openai/gpt-6-luna",
    "messages": [{ "role": "user", "content": "Hello" }],
    "provider": {
      "order": ["example-provider/eu-west"],
      "allow_fallbacks": false
    }
  }
}
```

Use this to benchmark one endpoint, or when a contract requires a named processor. Expect more `503` responses in exchange.

## What fallback costs

Failed tries cost nothing, and their hold is released in full. Only the endpoint that delivered output is charged, at its own price, and the provider of a failed try earns nothing.

A fallback can land on a more expensive endpoint than the first choice, within your `max_price`. Its hold must also fit your available balance; if it doesn't, routing may continue to a cheaper endpoint.

## Inspect the attempts

[`GET /generation`](/docs/api-reference/chat/get-generation) lists every endpoint that was tried, with its outcome, failure class and cost. Use the generation ID from the `id` of a response or stream chunk, or from the `X-Generation-Id` header, which is present once output has started.

```request
{
  "method": "GET",
  "path": "/generation",
  "query": { "id": "0f5b5c9a-3a7e-4c6f-9b1d-2e8a7c4d5f60" }
}
```

```json title="Response (abridged)"
{
  "data": {
    "id": "0f5b5c9a-3a7e-4c6f-9b1d-2e8a7c4d5f60",
    "model": "openai/gpt-6-luna",
    "outcome": "success",
    "provider_slug": "another-provider",
    "endpoint": "another-provider/fast-gpu",
    "delivered_output": true,
    "total_cost": 0.000154,
    "attempts": [
      {
        "ordinal": 1,
        "provider_slug": "example-provider",
        "endpoint": "example-provider/eu-west",
        "outcome": "failed",
        "failure_class": "timeout",
        "delivered_output": false,
        "total_cost": 0
      },
      {
        "ordinal": 2,
        "provider_slug": "another-provider",
        "endpoint": "another-provider/fast-gpu",
        "outcome": "success",
        "delivered_output": true,
        "total_cost": 0.000154
      }
    ]
  }
}
```

`failure_class` says why a try failed, for example `provider_unavailable`, `rate_limited`, `timeout` or `cancelled`.

A request that fails before any output returns no `X-Generation-Id`. Find it in [Logs](/dashboard/logs): each entry's **Route** panel lists the endpoints tried, in order, with their outcome, timing and whether they were charged.

## Model fallback

Server-side fallback never changes the model. The `models` array accepts exactly one entry, so OpenRouter-style multi-model fallback is rejected with `400`. Substituting another model would change your output and your price without your consent.

To use a second model when the first is unavailable, do it in your client:

| Response | Meaning | Client action |
| --- | --- | --- |
| `503` `provider_unavailable` or `provider_overloaded` | No endpoint can serve the request, including for unknown, currency-mismatched and inaccessible models | Try the next model |
| `502` `provider_unavailable` | The last endpoint returned an invalid response | Try the next model |
| `408` `timeout` | The request exceeded its time limit | Try the next model |
| `429` `rate_limited` | Your traffic limit (with `Retry-After`), or every provider rate-limited the request (without it) | Wait for `Retry-After` when present, otherwise back off, then retry the same model |
| `503` `routing_unavailable` or `service_unavailable` | A transient platform condition | Back off and retry the same model |
| `400`, `401`, `402`, `403`, `413`, `415` | A request, key, balance or permission problem | Stop; a retry fails the same way |

Switching models doesn't help with your own `429`: traffic limits apply to your key and account, whatever the model. Give every try a fresh `X-Request-Id`, or omit the header; see [Request and generation IDs](/docs/api-reference/errors#request-and-generation-ids).

:::code-group
```python title="Python"
import os
import random
import time
import uuid

from openai import APIConnectionError, APIStatusError, OpenAI

client = OpenAI(
    base_url="https://api.numes.ai/api/v1",
    api_key=os.environ["NUMES_API_KEY"],
    max_retries=0,  # this loop owns retries
    timeout=630,    # above the server's request limit
)

MODELS = ["openai/gpt-6-luna", "author/backup-model"]
STOP = {400, 401, 402, 403, 413, 415}
SAME_MODEL = {"routing_unavailable", "service_unavailable"}


def error_type(error: APIStatusError) -> str:
    try:
        return error.response.json()["error"]["metadata"]["error_type"]
    except Exception:
        return ""


def backoff(attempt: int, retry_after: str | None = None) -> float:
    if retry_after and retry_after.isdigit():
        return float(retry_after)
    return random.uniform(0, min(8.0, 0.5 * 2**attempt))


def complete(messages: list[dict]):
    last_error: Exception | None = None
    for model in MODELS:
        for attempt in range(3):
            try:
                return client.chat.completions.create(
                    model=model,
                    messages=messages,
                    extra_headers={"X-Request-Id": uuid.uuid4().hex},
                )
            except APIStatusError as error:
                last_error = error
                kind = error_type(error)
                if error.status_code in STOP or kind == "cancelled":
                    raise
                if error.status_code == 429 or kind in SAME_MODEL:
                    time.sleep(backoff(attempt, error.response.headers.get("retry-after")))
                    continue
                break  # 502, 503 provider_*, 408 timeout: next model
            except APIConnectionError as error:
                last_error = error
                time.sleep(backoff(attempt))
    raise last_error
```

```typescript title="TypeScript"
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.numes.ai/api/v1",
  apiKey: process.env.NUMES_API_KEY,
  maxRetries: 0, // this loop owns retries
  timeout: 630_000, // above the server's request limit
});

const MODELS = ["openai/gpt-6-luna", "author/backup-model"];
const STOP = new Set([400, 401, 402, 403, 413, 415]);
const SAME_MODEL = new Set(["routing_unavailable", "service_unavailable"]);

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

function backoffMs(attempt: number, retryAfter?: string | null): number {
  if (retryAfter && /^\d+$/.test(retryAfter)) return Number(retryAfter) * 1000;
  return Math.random() * Math.min(8000, 500 * 2 ** attempt);
}

function errorType(error: InstanceType<typeof OpenAI.APIError>): string {
  const body = error.error as { metadata?: { error_type?: string } } | undefined;
  return body?.metadata?.error_type ?? "";
}

export async function complete(messages: OpenAI.Chat.ChatCompletionMessageParam[]) {
  let lastError: unknown;
  for (const model of MODELS) {
    for (let attempt = 0; attempt < 3; attempt++) {
      try {
        return await client.chat.completions.create(
          { model, messages },
          { headers: { "X-Request-Id": crypto.randomUUID() } },
        );
      } catch (error) {
        lastError = error;
        if (error instanceof OpenAI.APIError && error.status !== undefined) {
          const kind = errorType(error);
          if (STOP.has(error.status) || kind === "cancelled") throw error;
          if (error.status === 429 || SAME_MODEL.has(kind)) {
            await sleep(backoffMs(attempt, error.headers?.get("retry-after")));
            continue;
          }
          break; // 502, 503 provider_*, 408 timeout: next model
        }
        await sleep(backoffMs(attempt)); // network error
      }
    }
  }
  throw lastError;
}
```
:::

With streaming, decide how to handle an interrupted stream before you retry. Output already delivered was billed, and a retry on another model starts from the beginning. A common approach is to retry only when no content has arrived yet, and otherwise surface the partial answer.

## Build resilient clients

- **Retries.** Retry only status codes that can succeed on a second try, with exponential backoff and jitter, honoring `Retry-After`. See [Retry strategy](/docs/api-reference/errors#retry-strategy).
- **Timeouts.** Set your client timeout above the server's request limit, and don't use a short idle timeout before the first chunk. See [Timeouts](/docs/api-reference/limits#timeouts).
- **Rate limits.** Extra keys don't raise your ceiling; smooth bursts with a client-side queue. See [Rate limits](/docs/api-reference/limits#rate-limits).
- **Faster endpoints.** `provider.sort: "latency"` or `"throughput"` puts the fastest measured endpoints first. Keep `allow_fallbacks` at `true` unless you must pin one endpoint.
- **Interrupted requests.** A dropped connection or platform maintenance can interrupt a request that already delivered output. Don't repeat actions your tools performed, and check [`GET /generation`](/docs/api-reference/chat/get-generation) for what was delivered and charged before you retry.

## Next steps

:::cards
- {network} [Provider routing](/docs/routing/provider-routing) Order, filters and sorting that shape the list of endpoints.
- {shield-check} [Verification and trust](/docs/routing/verification) What makes an endpoint eligible.
- {alert} [Errors](/docs/api-reference/errors) Every status code, `error_type` and the retry strategy.
- {receipt} [Usage accounting](/docs/account/usage-accounting) How delivered output is charged.
:::