# Streaming

> Stream chat output as Server-Sent Events, read the final usage chunk, cancel, handle mid-stream errors, and parse the wire format, including image and speech streams.

Source: https://0.0.0.0:3000/docs/features/streaming
Documentation index: https://0.0.0.0:3000/docs/llms.txt

Set `stream: true` on a chat completion to receive output as it's generated. The response is a Server-Sent Events (SSE) stream of OpenAI-compatible `chat.completion.chunk` objects that ends with a usage chunk carrying the request's cost.

Text, reasoning, argument fragments, transcripts and citation text preserve control characters through JSON escaping. Decode each JSON event before using its text, and escape it for HTML or terminal display. Response size and structure limits still apply.

Citation URLs may use HTTP or HTTPS and are limited to 8,192 UTF-8 bytes. A
citation is a reference; receiving it does not fetch its contents.

## Start a stream

Send the usual request body with `stream: true`. Choose **Try it** to watch the events arrive.

```request
{
  "method": "POST",
  "path": "/chat/completions",
  "body": {
    "model": "openai/gpt-6-luna",
    "stream": true,
    "messages": [
      { "role": "user", "content": "Explain Server-Sent Events in two sentences." }
    ]
  }
}
```

A successful stream returns `200` with `Content-Type: text/event-stream`, `Cache-Control: no-cache`, `X-Request-Id` and `X-Generation-Id`. The generation ID is also every chunk's `id`; keep it to look up the cost later.

```sse
data: {"id":"6f1c2b8e-3d4a-4f5b-9c7d-2e8a1b0c9d3f","object":"chat.completion.chunk","created":1790000000,"model":"openai/gpt-6-luna","choices":[{"index":0,"delta":{"role":"assistant","content":"Server-Sent"},"finish_reason":null}]}

data: {"id":"6f1c2b8e-3d4a-4f5b-9c7d-2e8a1b0c9d3f","object":"chat.completion.chunk","created":1790000000,"model":"openai/gpt-6-luna","choices":[{"index":0,"delta":{"content":" Events keep one HTTP response open"},"finish_reason":null}]}

data: {"id":"6f1c2b8e-3d4a-4f5b-9c7d-2e8a1b0c9d3f","object":"chat.completion.chunk","created":1790000000,"model":"openai/gpt-6-luna","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"6f1c2b8e-3d4a-4f5b-9c7d-2e8a1b0c9d3f","object":"chat.completion.chunk","created":1790000000,"model":"openai/gpt-6-luna","choices":[],"usage":{"prompt_tokens":16,"completion_tokens":38,"total_tokens":54,"cost":0.000017,"cost_details":{"upstream_inference_cost":0.000017}}}

data: [DONE]
```

A successful stream always arrives in this order:

1. **Output chunks.** The first chunk's `delta` normally carries `role: "assistant"` along with its content; there's no separate role-only chunk.
2. **Finish chunk.** `delta` is empty, or carries a reasoning terminator, and `finish_reason` is set. See [Finish reasons](/docs/api-reference/overview#finish-reasons).
3. **Usage chunk.** `choices` is an empty array and `usage` holds the final usage and cost.
4. **`data: [DONE]`.**

### Delta fields

A delta carries only the fields produced by that event:

:::fields delta
- `role` `string` — `assistant`, on the first chunk at most.
- `content` `string` — The next fragment of the answer.
- `reasoning` `string` — The next fragment of visible reasoning.
- `reasoning_details` `object[]` — Structured reasoning. A text entry repeats `reasoning` from the same chunk.
- `tool_calls` `object[]` — Function-call fragments keyed by `index`.
- `annotations` `object[]` — Citations from [hosted tools](/docs/features/hosted-tools), with `type` `url_citation` or `file_citation`.
- `artifacts` `object[]` — Inline files from hosted tools: `type` (`file`, `image` or `audio`), `filename`, `media_type` and base64 `data`.
- `images` `object[]` — Chat image output, from endpoints that support it.
- `audio` `object` — Chat audio output, from endpoints that support it.
:::

### Reasoning deltas

Each visible reasoning fragment appears **twice in the same chunk**: as `delta.reasoning` and as a `reasoning.text` entry in `delta.reasoning_details`.

```sse
data: {"id":"6f1c2b8e-3d4a-4f5b-9c7d-2e8a1b0c9d3f","object":"chat.completion.chunk","created":1790000000,"model":"openai/gpt-6-luna","choices":[{"index":0,"delta":{"role":"assistant","reasoning":"The user wants brevity.","reasoning_details":[{"type":"reasoning.text","text":"The user wants brevity."}]},"finish_reason":null}]}
```

:::warning Read one representation, not both
Concatenating `reasoning` and the text in `reasoning_details` doubles the reasoning. Display `reasoning`, and accumulate `reasoning_details` only to send reasoning state back on a later turn.
:::

- When a provider signs a reasoning block after its text has streamed, a later `reasoning_details` entry carries the `signature` (and `id` or `format`) without repeating the text. Accumulate the entries in order to rebuild the signed block.
- The finish chunk has an empty delta. Provider continuation details are preserved without adding an unsigned terminator.
- `reasoning.encrypted` and `reasoning.summary` entries can also appear. Send them back unchanged.
- With `reasoning: {"exclude": true}`, neither representation is sent, so expect a longer pause before the first chunk.

[Reasoning](/docs/features/reasoning) covers the controls and continuation.

### Tool call deltas

Function calls stream in `delta.tool_calls`. Accumulate them by each entry's `index`, not by position in the array.

```sse
data: {"id":"6f1c2b8e-3d4a-4f5b-9c7d-2e8a1b0c9d3f","object":"chat.completion.chunk","created":1790000000,"model":"openai/gpt-6-luna","choices":[{"index":0,"delta":{"role":"assistant","tool_calls":[{"index":0,"id":"call_7f3a9c","type":"function","function":{"name":"get_weather"}}]},"finish_reason":null}]}

data: {"id":"6f1c2b8e-3d4a-4f5b-9c7d-2e8a1b0c9d3f","object":"chat.completion.chunk","created":1790000000,"model":"openai/gpt-6-luna","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"city\":\"Lisbon\"}"}}]},"finish_reason":null}]}

data: {"id":"6f1c2b8e-3d4a-4f5b-9c7d-2e8a1b0c9d3f","object":"chat.completion.chunk","created":1790000000,"model":"openai/gpt-6-luna","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}
```

- The fragment that opens a call carries `id`, `type: "function"` and `function.name`. Later fragments for the same `index` may omit them.
- Some endpoints send a call's full `arguments` in one fragment; others split them. Append fragments in arrival order; fragments of different calls can interleave.
- Parse arguments and run tools only after the finish chunk. Some endpoints report `stop` instead of `tool_calls`, so check for accumulated calls too.

See [Tool calling](/docs/features/tool-calling) for the full loop.

## Final usage chunk

Every successful stream ends with a usage chunk. You don't need `stream_options: {"include_usage": true}`: it's accepted and ignored, as is `usage: {"include": true}`, and setting either to `false` doesn't suppress the chunk.

`usage.cost` is the final charge in your key's currency (satoshis for BTC keys). With `X-OpenRouter-Metadata: enabled`, the usage chunk also carries `openrouter_metadata`. [Usage accounting](/docs/account/usage-accounting) explains every usage field.

A stream that's cancelled or ends with an error has no usage chunk; read its cost from [`GET /generation`](/docs/api-reference/chat/get-generation) with the generation ID.

## Cancel a stream

Close the connection to stop the request; it's the only cancellation signal. With the OpenAI SDKs, call `stream.close()` in Python or `stream.controller.abort()` in TypeScript.

A cancelled stream has no usage chunk. [`GET /generation`](/docs/api-reference/chat/get-generation) is the authoritative record of what was delivered and charged; it can show `outcome: "in_progress"` briefly while the request finalizes.

```request
{
  "method": "GET",
  "path": "/generation",
  "query": { "id": "6f1c2b8e-3d4a-4f5b-9c7d-2e8a1b0c9d3f" }
}
```

[Encrypted models](/docs/inference/encrypted#billing-and-limits) differ: after output starts, closing the connection stops delivery to you, but the request runs to completion and is billed on the provider's final usage.

## Errors

### Errors before the first chunk

If the request fails before any chunk is written, you get an ordinary non-`200` response with the JSON [error envelope](/docs/api-reference/errors) and no event stream: for example `400` for an invalid body, `402` for insufficient credits, `429` for a rate limit or `503` when no endpoint can serve the request. Before any output, the request may already have been retried on other endpoints. Check the status before you parse events.

### Errors after streaming starts

Once the stream has started, the status is already `200`. A later failure arrives as one final chunk with a top-level `error` object and `finish_reason: "error"`, followed by `data: [DONE]`. No finish chunk or usage chunk is sent.

```sse
data: {"id":"6f1c2b8e-3d4a-4f5b-9c7d-2e8a1b0c9d3f","object":"chat.completion.chunk","created":1790000004,"model":"openai/gpt-6-luna","error":{"code":503,"message":"provider became unavailable","metadata":{"error_type":"provider_unavailable"}},"choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}]}

data: [DONE]
```

Server interruption is distinct from your own cancellation: the former uses `503 provider_unavailable`, the latter `408 cancelled`. A client retry is a new request; handle partial output and tool effects before restarting. See the [retry guidance](/docs/api-reference/errors#errors-in-streams).

`error.code` is the HTTP status the error would have had, and `error.metadata.error_type` is the stable category:

| `code` | `error_type` | Cause |
| --- | --- | --- |
| `408` | `timeout` | The request reached its time limit |
| `408` | `cancelled` | The caller cancelled the request or response delivery failed |
| `429` | `rate_limit_exceeded` | The provider rate-limited the request |
| `502` | `provider_unavailable` | The provider sent an invalid response |
| `503` | `provider_unavailable` | The provider failed or server execution was interrupted while the caller remained connected |
| `503` | `routing_unavailable` | The service is unavailable |
| `503` | `bitcoin_reference_price_unavailable` | A BTC price couldn't be confirmed; no output was delivered |

Errors from your account's [request policy](/docs/privacy/request-policy) can also arrive in-band with the codes listed in [Errors and debugging](/docs/api-reference/errors#status-codes-and-error-types).

Output never switches to another provider once it has reached you. The failed generation may have a charge; check its generation record. If your client retries, the new request has separate usage and cost, and must handle the partial response as described above.

A stream can also end without `[DONE]` when the connection drops. Treat a missing terminator as an incomplete response and check the generation record.

## Timeouts

A request, including its whole stream, has a 10-minute time limit. Before any output it fails with a `408 timeout` error response; after output starts you get the in-band `timeout` chunk.

The stream has no keep-alive frames, so long silent gaps are normal: before the first chunk, while a model reasons, or while a hosted tool runs. Set your client's read timeout to at least the request limit, and make sure proxies between you and the API don't close idle connections sooner.

## Stream with the OpenAI SDK

The official SDKs parse the stream for you. Fields outside the OpenAI types, such as `reasoning` and `usage.cost`, are still present on each chunk. An in-band error chunk raises an `APIError`.

:::code-group
```python title="Python"
import os
from openai import OpenAI, APIError

client = OpenAI(
    base_url="https://api.numes.ai/api/v1",
    api_key=os.environ["NUMES_API_KEY"],
    timeout=360,  # seconds; at least the request time limit
)

stream = client.chat.completions.create(
    model="openai/gpt-6-luna",
    messages=[{"role": "user", "content": "Explain Server-Sent Events in two sentences."}],
    stream=True,
)

generation_id, usage = None, None
try:
    for chunk in stream:
        generation_id = chunk.id
        if chunk.usage is not None:
            usage = chunk.usage  # usage chunk; choices is empty
        if chunk.choices and chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)
except APIError as error:
    print(f"\nstream failed: {error.message}")
finally:
    stream.close()

if usage is not None:
    print(f"\ncost: {getattr(usage, 'cost', None)}")
else:
    print(f"\nno usage chunk; look up generation {generation_id}")
```

```typescript title="TypeScript"
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.numes.ai/api/v1",
  apiKey: process.env.NUMES_API_KEY,
  timeout: 360_000, // milliseconds; at least the request time limit
});

const stream = await client.chat.completions.create({
  model: "openai/gpt-6-luna",
  messages: [{ role: "user", content: "Explain Server-Sent Events in two sentences." }],
  stream: true,
});

let generationId: string | undefined;
let usage: (OpenAI.CompletionUsage & { cost?: number }) | undefined;
try {
  for await (const chunk of stream) {
    generationId = chunk.id;
    if (chunk.usage) usage = chunk.usage; // usage chunk; choices is empty
    const content = chunk.choices[0]?.delta.content;
    if (content) process.stdout.write(content);
  }
} catch (error) {
  if (!(error instanceof OpenAI.APIError)) throw error;
  console.error("\nstream failed:", error.message);
}

console.log(usage ? `\ncost: ${usage.cost}` : `\nno usage chunk; look up generation ${generationId}`);
```
:::

## Wire format

Read this section when you write or debug your own parser.

### Frame syntax

- Each event is one `data: ` line holding a JSON object with no line breaks, followed by an empty line: `data: {...}\n\n`.
- Frames use only the `data` field; there are no `event:`, `id:` or `retry:` fields.
- Each frame is flushed as soon as it's written.
- The stream ends with the literal frame `data: [DONE]`, which isn't JSON.
- Lines starting with `:` are SSE comments. The API doesn't send them, but your parser must ignore them.
- JSON key order inside a chunk isn't significant.

### Chunk shape

```typescript title="ChatCompletionChunk"
type ChatCompletionChunk = {
  id: string; // generation ID, constant for the stream
  object: "chat.completion.chunk";
  created: number; // Unix seconds
  model: string;
  choices: [{ index: 0; delta: Delta; finish_reason: string | null }] | []; // [] on the usage chunk
  usage?: Usage; // usage chunk only
  openrouter_metadata?: Record<string, unknown>; // usage chunk, when requested
  error?: { code: number; message: string; metadata: { error_type: string } }; // error chunk only
};

type Delta = {
  role?: "assistant"; // first chunk only
  content?: string;
  reasoning?: string;
  reasoning_details?: ReasoningDetail[];
  tool_calls?: {
    index: number;
    id?: string; // first fragment of a call
    type?: "function";
    function: { name?: string; arguments?: string };
  }[];
  annotations?: Annotation[];
  artifacts?: { type: "file" | "image" | "audio"; filename: string; media_type: string; data: string }[];
  images?: { index: number; type: "image_url"; image_url: { url: string } }[];
  audio?: { data: string; transcript: string; format: string };
};
```

`Usage`, `ReasoningDetail` and `Annotation` have the same shapes as in a buffered response; see [Create chat completion](/docs/api-reference/chat/create-chat-completion). Chunks have no `provider` field; send `X-OpenRouter-Metadata: enabled` to get the serving provider on the usage chunk.

### Image streams

[`POST /images`](/docs/api-reference/media/create-image) with `stream: true` uses the same frame syntax, `Content-Type: text/event-stream` and `X-Generation-Id`. Each frame is a JSON object with a `type`:

| `type` | Fields | Meaning |
| --- | --- | --- |
| `image_generation.partial` | `index`, `b64_json`, `media_type` | A progress preview |
| `image_generation.completed` | `index`, `b64_json`, `media_type` | A final image |
| `image_generation.usage` | `usage` | Usage and `cost`, sent once after the images |
| `image_generation.error` | `error` | An error after streaming started |

```sse
data: {"type":"image_generation.partial","index":0,"b64_json":"iVBORw0KGgo...","media_type":"image/png"}

data: {"type":"image_generation.completed","index":0,"b64_json":"iVBORw0KGgo...","media_type":"image/png"}

data: {"type":"image_generation.usage","usage":{"input_images":0,"output_images":1,"input_audio_seconds":0,"output_audio_seconds":0,"input_characters":0,"prompt_tokens":0,"completion_tokens":0,"total_tokens":0,"cost":0.04}}

data: [DONE]
```

`index` identifies the image when you request several with `n`. Images are always base64; there are no image URLs. An image error frame differs from the chat shape: `error.code` is the error type and `error.status` is the HTTP status.

```sse
data: {"type":"image_generation.error","error":{"code":"provider_unavailable","message":"provider became unavailable","status":503}}

data: [DONE]
```

### Speech streams

[`POST /audio/speech`](/docs/api-reference/media/create-speech) doesn't use SSE. It streams raw audio bytes with the audio format as `Content-Type` and `X-Generation-Id` set when the first bytes are sent; write the body straight to a file or player. Errors before the first byte use the JSON error envelope. After audio starts there's no in-band error: a failure ends the response early. Check the generation's `outcome` with `GET /generation` to confirm a complete result.

### Write a robust parser

1. **Check the status first.** Only a `200` with `Content-Type: text/event-stream` is a stream; anything else is a JSON error.
2. **Buffer across reads.** A read can end mid-line or mid-character. Decode UTF-8 with a streaming decoder and split on `\n` or `\r\n` only after buffering.
3. **An empty line ends an event.** Join its `data:` lines with `\n`.
4. **Ignore everything else**, including `:` comment lines and fields other than `data`.
5. **Stop at `[DONE]`** without passing it to a JSON parser.
6. **Check every chunk for `error`** before reading `choices`, and remember the usage chunk has `choices: []`.
7. **Reassemble tool calls by `index`**, and read one reasoning representation.

:::code-group
```python title="Python"
import json
import os

import requests


def sse_data(lines):
    """Yield the data payload of each SSE event."""
    data = []
    for line in lines:
        if line == "":  # a blank line ends the event
            if data:
                yield "\n".join(data)
                data = []
        elif not line.startswith(":"):  # skip comments
            field, _, value = line.partition(":")
            if field == "data":
                data.append(value[1:] if value.startswith(" ") else value)
    if data:
        yield "\n".join(data)


response = requests.post(
    "https://api.numes.ai/api/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['NUMES_API_KEY']}"},
    json={
        "model": "openai/gpt-6-luna",
        "stream": True,
        "messages": [{"role": "user", "content": "What is the weather in Oslo?"}],
    },
    stream=True,
    timeout=(10, 360),  # connect, read
)
if response.status_code != 200:
    raise RuntimeError(response.json()["error"])
response.encoding = "utf-8"  # event streams are UTF-8

text, calls, usage = [], {}, None
try:
    for payload in sse_data(response.iter_lines(decode_unicode=True)):
        if payload == "[DONE]":
            break
        chunk = json.loads(payload)
        if "error" in chunk:
            raise RuntimeError(f"stream failed: {chunk['error']} (generation {chunk['id']})")
        if chunk.get("usage"):
            usage = chunk["usage"]
        for choice in chunk["choices"]:
            delta = choice["delta"]
            text.append(delta.get("content") or "")
            for call in delta.get("tool_calls", []):
                slot = calls.setdefault(call["index"], {"id": None, "name": None, "arguments": ""})
                slot["id"] = call.get("id", slot["id"])
                function = call.get("function", {})
                slot["name"] = function.get("name", slot["name"])
                slot["arguments"] += function.get("arguments", "")
finally:
    response.close()  # closing early cancels the request

print("".join(text))
for call in calls.values():
    print(call["name"], json.loads(call["arguments"] or "{}"))
print("cost:", usage["cost"] if usage else "unknown")
```

```typescript title="TypeScript"
async function* sseData(body: ReadableStream<Uint8Array>): AsyncGenerator<string> {
  const reader = body.pipeThrough(new TextDecoderStream()).getReader();
  let buffer = "";
  let data: string[] = [];
  for (;;) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += value;
    let newline: number;
    while ((newline = buffer.indexOf("\n")) >= 0) {
      const line = buffer.slice(0, newline).replace(/\r$/, "");
      buffer = buffer.slice(newline + 1);
      if (line === "") {
        if (data.length > 0) yield data.join("\n");
        data = [];
      } else if (!line.startsWith(":")) {
        const colon = line.indexOf(":");
        const field = colon < 0 ? line : line.slice(0, colon);
        const raw = colon < 0 ? "" : line.slice(colon + 1);
        if (field === "data") data.push(raw.startsWith(" ") ? raw.slice(1) : raw);
      }
    }
  }
  if (data.length > 0) yield data.join("\n");
}

// Aborting the controller closes the connection and cancels the request.
const controller = new AbortController();
const response = await fetch("https://api.numes.ai/api/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.NUMES_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "openai/gpt-6-luna",
    stream: true,
    messages: [{ role: "user", content: "What is the weather in Oslo?" }],
  }),
  signal: controller.signal,
});
if (!response.ok || !response.body) {
  throw new Error(JSON.stringify(await response.json()));
}

type Call = { id?: string; name?: string; arguments: string };
const calls = new Map<number, Call>();
let text = "";
let usage: { cost: number } | undefined;

for await (const payload of sseData(response.body)) {
  if (payload === "[DONE]") break;
  const chunk = JSON.parse(payload);
  if (chunk.error) {
    throw new Error(`stream failed: ${chunk.error.metadata.error_type} (generation ${chunk.id})`);
  }
  if (chunk.usage) usage = chunk.usage;
  for (const choice of chunk.choices) {
    const delta = choice.delta;
    if (delta.content) text += delta.content;
    for (const call of delta.tool_calls ?? []) {
      const slot = calls.get(call.index) ?? { arguments: "" };
      if (call.id) slot.id = call.id;
      if (call.function?.name) slot.name = call.function.name;
      slot.arguments += call.function?.arguments ?? "";
      calls.set(call.index, slot);
    }
  }
}

console.log(text);
for (const call of calls.values()) console.log(call.name, JSON.parse(call.arguments || "{}"));
console.log("cost:", usage?.cost);
```
:::

## Next steps

:::cards
- {wrench} [Tool calling](/docs/features/tool-calling) Accumulate streamed calls and return results to the model.
- {brain} [Reasoning](/docs/features/reasoning) Effort, budgets, exclusion and reasoning state across turns.
- {receipt} [Usage accounting](/docs/account/usage-accounting) Every usage field and how cost is calculated.
- {alert} [Errors and debugging](/docs/api-reference/errors) Status codes, error types and retry strategy.
:::