Prompt caching

Many providers can cache the processed form of a prompt prefix, such as a long system prompt, a document or a conversation history, and reuse it on the next request. Cached input is faster and can be cheaper. Numes makes cache hits more likely by keeping related requests on the same endpoint, and passes explicit cache directives to endpoints that support them.

Caching happens at the provider. A directive or key makes a hit more likely but never guarantees one, and cached input never costs more than ordinary input.

Put stable content first and changing content last. Caches match on an exact prefix, so a timestamp or request ID at the top of the system prompt defeats every mechanism on this page.

Prompt cache key

prompt_cache_key groups requests that share a prefix. Your value is never forwarded: it's used, scoped to your account and the model, for endpoint affinity, and endpoints that accept a cache key receive only an opaque hint with no content or account identifiers. prompt_cache_key takes precedence over session_id for affinity.

curl -X POST "https://api.numes.ai/api/v1/chat/completions" \  -H "Authorization: Bearer $NUMES_API_KEY" \  -H "Content-Type: application/json" \  -d '{    "model": "openai/gpt-6-luna",    "prompt_cache_key": "support-bot-v3",    "messages": [      {        "role": "system",        "content": "You are the support assistant for an internal ticketing system. Follow the style guide below.\n\n<long, stable style guide>"      },      {        "role": "user",        "content": "How do I reopen a closed ticket?"      }    ]  }'

Use one key per reusable prefix, such as one per system prompt version or per document, rather than one per user or per request.

Cache control

cache_control marks content as cacheable, in the Anthropic and OpenRouter style. It accepts one shape:

You can place it in one of two ways, never both in the same request.

At the top level

A top-level directive asks the endpoint to cache the prompt automatically:

curl -X POST "https://api.numes.ai/api/v1/chat/completions" \  -H "Authorization: Bearer $NUMES_API_KEY" \  -H "Content-Type: application/json" \  -d '{    "model": "openai/gpt-6-luna",    "cache_control": {      "type": "ephemeral",      "ttl": "1h"    },    "messages": [      {        "role": "system",        "content": "<long, stable instructions>"      },      {        "role": "user",        "content": "Summarize the instructions in one sentence."      }    ]  }'

On messages or text parts

On a message or on a text content part, the directive marks the end of a cacheable prefix. Mark the last block of the stable content:

curl -X POST "https://api.numes.ai/api/v1/chat/completions" \  -H "Authorization: Bearer $NUMES_API_KEY" \  -H "Content-Type: application/json" \  -d '{    "model": "openai/gpt-6-luna",    "messages": [      {        "role": "system",        "content": [          {            "type": "text",            "text": "You answer questions about the attached handbook."          },          {            "type": "text",            "text": "<full handbook text>",            "cache_control": {              "type": "ephemeral"            }          }        ]      },      {        "role": "user",        "content": "What is the parental leave policy?"      }    ]  }'

Placement rules, each enforced with 400 invalid_request:

• A top-level cache_control cannot be combined with any message- or part-level cache_control.

• A message and its own text parts cannot both carry cache_control.

• A text part cannot carry both cache_control and prompt_cache_breakpoint.

cache_control on an image, file, audio or unfamiliar content part is accepted as the same hint as on a text part.

A function tool may also carry cache_control, as some clients add one by default, either on the tool or inside function. It's validated but not sent to the endpoint. Mark a system or message block instead: the cached prefix includes the tools before it.

Cache options and breakpoints

Some upstream APIs use a different vocabulary: request-level cache options and explicit breakpoints.

At least one of the two keys is required; {} is rejected.

prompt_cache_breakpoint marks an explicit cache boundary on a text part. Its only form is {"mode": "explicit"}:

{  "model": "openai/gpt-6-luna",  "prompt_cache_options": { "mode": "explicit", "ttl": "30m" },  "messages": [    {      "role": "user",      "content": [        { "type": "text", "text": "<long reference document>", "prompt_cache_breakpoint": { "mode": "explicit" } },        { "type": "text", "text": "Which sections mention data retention?" }      ]    }  ]}

Directives narrow routing

These directives are always honored. Requests go only to endpoints that support each one, and the directive is never dropped to find an endpoint:

If no eligible endpoint supports all of them, the request fails with 503 provider_unavailable.

cache_control on messages or content parts, including media parts, is a hint instead. Endpoints that don't list cache_control still serve the request without the markers. To send marked requests only to endpoints that support them, set provider.require_parameters: true.

Endpoints support different subsets of the directives, so check each endpoint's supported_parameters:

curl "https://api.numes.ai/api/v1/models/openai/gpt-6-luna/endpoints" \  -H "Authorization: Bearer $NUMES_API_KEY"

Omitting directives never narrows routing, and affinity through session_id works on every endpoint. Some endpoints add an automatic cache hint when your request has no cache directive; any directive you send replaces it.

Read cache usage

When the provider reports how much of the prompt it read from or wrote to its cache, the counts appear in usage:

{  "prompt_tokens": 18432,  "completion_tokens": 120,  "total_tokens": 18552,  "prompt_tokens_details": { "cached_tokens": 17920, "cache_write_tokens": 0 },  "native_tokens": {    "prompt_tokens": 18432,    "completion_tokens": 120,    "cache_read_tokens": 17920,    "cache_write_tokens": 0  },  "token_count_basis": "provider_native",  "cost": 0.001704,  "cost_details": { "upstream_inference_cost": 0.001704 }}

Absent and zero mean different things: a missing field means the provider didn't report that count, not that no caching happened; 0 means it reported none. Cache counts are part of prompt_tokens, never added on top. Counts can appear even when the endpoint has no cache rate; those tokens are then billed at the ordinary prompt rate.

GET /generation additionally reports cache_discount: the ordinary prompt charge minus the charge actually applied to the classified prompt. It is always zero or positive.

Pricing

Endpoints can publish separate rates for cached input in their pricing object:

• Discount only. Cache rates, including the cache-write rate, are never higher than the endpoint's ordinary prompt rate.

• Absent is not zero. Without input_cache_read, cached tokens are billed at the ordinary prompt rate. An explicit "0" means cache reads are free.

• How the discount applies. Cache discounts are based on the usage the platform accepts from the provider's response: cached prompt tokens are billed at the cache rate and the rest at the ordinary rate.

• Price ceilings. A provider.max_price.prompt ceiling also applies to an endpoint's cache-write rate.

Stickiness and caching

A cache lives on the endpoint that built it. Affinity sends a conversation back to the endpoint that last served it, as long as that endpoint is still eligible and among the lowest-priced. It applies only when you set neither provider.order nor provider.sort; with either, your explicit preference decides and cache hits become less likely. Sticky sessions explains the rules.

Best practices

• Order prompts for reuse: tools and system instructions first, then long reference content, then the changing conversation.

• Keep the prefix byte-identical. Whitespace, ordering of tool definitions and dynamic values all change it.

• Prefer affinity or message-level markers over the other directives when you want caching without restricting which endpoints can serve you.

• Use one session_id or prompt_cache_key per conversation. Either keeps it on one endpoint; the key also passes a cache hint to endpoints that accept one.

• Check cached_tokens over several requests to confirm hits; a missing field means "not reported".

• Compare input_cache_read rates across endpoints for cache-heavy workloads.

Next steps