# Prompt caching

> Reuse repeated prompt prefixes through provider caches, read cache usage, and understand discount-only cache pricing.

Source: https://0.0.0.0:3000/docs/features/prompt-caching
Documentation index: https://0.0.0.0:3000/docs/llms.txt

Many providers can cache the processed form of a prompt prefix, such as a long system prompt, a document or a conversation history, and reuse it on the next request. Cached input is faster and can be cheaper. Numes makes cache hits more likely by keeping related requests on the same endpoint, and passes explicit cache directives to endpoints that support them.

Caching happens at the provider. A directive or key makes a hit more likely but never guarantees one, and cached input never costs more than ordinary input.

| Mechanism | What you send | Effect |
| --- | --- | --- |
| Endpoint affinity | Nothing, a `session_id`, or a `prompt_cache_key` | Follow-up requests go back to the endpoint that served the conversation, where its cache lives. See [Sticky sessions](/docs/features/sessions). |
| Cache key | `prompt_cache_key` | The same affinity, plus an opaque cache hint for endpoints that accept one. |
| Cache directives | `cache_control`, `prompt_cache_options`, `prompt_cache_breakpoint` | Explicit instructions about what to cache and for how long. |
| Provider defaults | Nothing | Many providers cache repeated prefixes automatically. |

Put stable content first and changing content last. Caches match on an exact prefix, so a timestamp or request ID at the top of the system prompt defeats every mechanism on this page.

## Prompt cache key

`prompt_cache_key` groups requests that share a prefix. Your value is never forwarded: it's used, scoped to your account and the model, for endpoint affinity, and endpoints that accept a cache key receive only an opaque hint with no content or account identifiers. `prompt_cache_key` takes precedence over `session_id` for affinity.

```request
{
  "method": "POST",
  "path": "/chat/completions",
  "body": {
    "model": "openai/gpt-6-luna",
    "prompt_cache_key": "support-bot-v3",
    "messages": [
      { "role": "system", "content": "You are the support assistant for an internal ticketing system. Follow the style guide below.\n\n<long, stable style guide>" },
      { "role": "user", "content": "How do I reopen a closed ticket?" }
    ]
  }
}
```

:::fields
- `prompt_cache_key` `string` — 1–256 bytes, not blank, no control characters.
:::

:::note The key never limits routing
`prompt_cache_key` is a hint, not a requirement. Endpoints that don't list `prompt_cache_key` in `supported_parameters` can still serve the request; they get the affinity but no cache hint.
:::

Use one key per reusable prefix, such as one per system prompt version or per document, rather than one per user or per request.

## Cache control

`cache_control` marks content as cacheable, in the Anthropic and OpenRouter style. It accepts one shape:

:::fields cache_control
- `type` `string` required — `ephemeral`.
- `ttl` `string` — `5m` or `1h`. Omit it for the provider's default lifetime.
:::

You can place it in one of two ways, never both in the same request.

### At the top level

A top-level directive asks the endpoint to cache the prompt automatically:

```request
{
  "method": "POST",
  "path": "/chat/completions",
  "body": {
    "model": "openai/gpt-6-luna",
    "cache_control": { "type": "ephemeral", "ttl": "1h" },
    "messages": [
      { "role": "system", "content": "<long, stable instructions>" },
      { "role": "user", "content": "Summarize the instructions in one sentence." }
    ]
  }
}
```

### On messages or text parts

On a message or on a `text` content part, the directive marks the end of a cacheable prefix. Mark the last block of the stable content:

```request
{
  "method": "POST",
  "path": "/chat/completions",
  "body": {
    "model": "openai/gpt-6-luna",
    "messages": [
      {
        "role": "system",
        "content": [
          { "type": "text", "text": "You answer questions about the attached handbook." },
          { "type": "text", "text": "<full handbook text>", "cache_control": { "type": "ephemeral" } }
        ]
      },
      { "role": "user", "content": "What is the parental leave policy?" }
    ]
  }
}
```

Placement rules, each enforced with `400 invalid_request`:

- A top-level `cache_control` cannot be combined with any message- or part-level `cache_control`.
- A message and its own text parts cannot both carry `cache_control`.
- A text part cannot carry both `cache_control` and `prompt_cache_breakpoint`.

`cache_control` on an image, file, audio or unfamiliar content part is accepted as the same hint as on a `text` part.

A function tool may also carry `cache_control`, as some clients add one by default, either on the tool or inside `function`. It's validated but not sent to the endpoint. Mark a system or message block instead: the cached prefix includes the tools before it.

## Cache options and breakpoints

Some upstream APIs use a different vocabulary: request-level cache options and explicit breakpoints.

:::fields prompt_cache_options
- `mode` `string` — `implicit` or `explicit`: the upstream's caching mode.
- `ttl` `string` — `30m`: requested retention.
:::

At least one of the two keys is required; `{}` is rejected.

`prompt_cache_breakpoint` marks an explicit cache boundary on a `text` part. Its only form is `{"mode": "explicit"}`:

```json
{
  "model": "openai/gpt-6-luna",
  "prompt_cache_options": { "mode": "explicit", "ttl": "30m" },
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "<long reference document>", "prompt_cache_breakpoint": { "mode": "explicit" } },
        { "type": "text", "text": "Which sections mention data retention?" }
      ]
    }
  ]
}
```

## Directives narrow routing

These directives are always honored. Requests go only to endpoints that support each one, and the directive is never dropped to find an endpoint:

| You send | The endpoint must list in `supported_parameters` |
| --- | --- |
| Top-level `cache_control` | `cache_control` |
| `prompt_cache_options` | `prompt_cache_options` |
| `prompt_cache_breakpoint` | `prompt_cache_breakpoint` |

If no eligible endpoint supports all of them, the request fails with `503 provider_unavailable`.

`cache_control` on messages or content parts, including media parts, is a hint instead. Endpoints that don't list `cache_control` still serve the request without the markers. To send marked requests only to endpoints that support them, set `provider.require_parameters: true`.

Endpoints support different subsets of the directives, so check each endpoint's `supported_parameters`:

```request
{
  "method": "GET",
  "path": "/models/openai/gpt-6-luna/endpoints"
}
```

Omitting directives never narrows routing, and affinity through `session_id` works on every endpoint. Some endpoints add an automatic cache hint when your request has no cache directive; any directive you send replaces it.

## Read cache usage

When the provider reports how much of the prompt it read from or wrote to its cache, the counts appear in `usage`:

```json title="Usage excerpt"
{
  "prompt_tokens": 18432,
  "completion_tokens": 120,
  "total_tokens": 18552,
  "prompt_tokens_details": { "cached_tokens": 17920, "cache_write_tokens": 0 },
  "native_tokens": {
    "prompt_tokens": 18432,
    "completion_tokens": 120,
    "cache_read_tokens": 17920,
    "cache_write_tokens": 0
  },
  "token_count_basis": "provider_native",
  "cost": 0.001704,
  "cost_details": { "upstream_inference_cost": 0.001704 }
}
```

:::fields Cache fields
- `prompt_tokens_details.cached_tokens` `integer` — Prompt tokens read from the cache, as the provider reported them.
- `prompt_tokens_details.cache_write_tokens` `integer` — Prompt tokens written to the cache.
- `native_tokens.cache_read_tokens` `integer` — The same read count within the provider's native token evidence.
- `native_tokens.cache_write_tokens` `integer` — The same write count within the provider's native token evidence.
:::

Absent and zero mean different things: a missing field means the provider didn't report that count, not that no caching happened; `0` means it reported none. Cache counts are part of `prompt_tokens`, never added on top. Counts can appear even when the endpoint has no cache rate; those tokens are then billed at the ordinary prompt rate.

[`GET /generation`](/docs/api-reference/chat/get-generation) additionally reports `cache_discount`: the ordinary prompt charge minus the charge actually applied to the classified prompt. It is always zero or positive.

## Pricing

Endpoints can publish separate rates for cached input in their `pricing` object:

:::fields pricing
- `prompt` `string` — Ordinary input rate per token, in your key's billing currency.
- `input_cache_read` `string` — Rate for prompt tokens read from the cache. Optional.
- `input_cache_write` `string` — Rate for prompt tokens written to the cache. Optional.
:::

- **Discount only.** Cache rates, including the cache-write rate, are never higher than the endpoint's ordinary prompt rate.
- **Absent is not zero.** Without `input_cache_read`, cached tokens are billed at the ordinary `prompt` rate. An explicit `"0"` means cache reads are free.
- **How the discount applies.** Cache discounts are based on the usage the platform accepts from the provider's response: cached prompt tokens are billed at the cache rate and the rest at the ordinary rate.
- **Price ceilings.** A `provider.max_price.prompt` ceiling also applies to an endpoint's cache-write rate.

## Stickiness and caching

A cache lives on the endpoint that built it. Affinity sends a conversation back to the endpoint that last served it, as long as that endpoint is still eligible and among the lowest-priced. It applies only when you set neither `provider.order` nor `provider.sort`; with either, your explicit preference decides and cache hits become less likely. [Sticky sessions](/docs/features/sessions) explains the rules.

## Best practices

- **Order prompts for reuse**: tools and system instructions first, then long reference content, then the changing conversation.
- **Keep the prefix byte-identical.** Whitespace, ordering of tool definitions and dynamic values all change it.
- **Prefer affinity or message-level markers over the other directives** when you want caching without restricting which endpoints can serve you.
- **Use one `session_id` or `prompt_cache_key` per conversation.** Either keeps it on one endpoint; the key also passes a cache hint to endpoints that accept one.
- **Check `cached_tokens` over several requests** to confirm hits; a missing field means "not reported".
- **Compare `input_cache_read` rates** across endpoints for cache-heavy workloads.

## Next steps

:::cards
- {tree} [Sticky sessions](/docs/features/sessions) How requests stay on one endpoint.
- {receipt} [Usage accounting](/docs/account/usage-accounting) Every usage field, including cache counts.
- {network} [Provider routing](/docs/routing/provider-routing) How `order` and `sort` interact with affinity.
- {boxes} [Models](/docs/models) Read endpoint pricing and supported parameters.
:::