# Multimodal

> Send images, PDFs and audio to chat models, and generate images, speech, transcripts and embeddings through dedicated endpoints.

Source: https://0.0.0.0:3000/docs/multimodal/overview
Documentation index: https://0.0.0.0:3000/docs/llms.txt

Media inputs travel inside ordinary Chat Completions requests as typed content parts in a `user` message. Media outputs come from dedicated endpoints that follow the OpenAI request shapes. All of them use the same inference key, routing preferences and billing as chat.

## Capabilities

| Capability | How to send or request it | Guide |
| --- | --- | --- |
| Image understanding | `POST /chat/completions` with an `image_url` content part | [Image inputs](/docs/multimodal/images) |
| PDF input | `POST /chat/completions` with a `file` content part | [PDFs and files](/docs/multimodal/pdfs) |
| Audio understanding | `POST /chat/completions` with an `input_audio` content part | [Audio inputs](/docs/multimodal/audio) |
| Image generation | `POST /images` with a `prompt` and optional reference images | [Image generation](/docs/multimodal/image-generation) |
| Text to speech | `POST /audio/speech` with `input` text and a `voice` | [Speech and transcription](/docs/multimodal/speech#text-to-speech) |
| Speech to text | `POST /audio/transcriptions` with a multipart upload or base64 JSON | [Speech and transcription](/docs/multimodal/speech#transcription) |
| Embeddings | `POST /embeddings` with strings, token arrays or content-part arrays | [Embeddings](/docs/multimodal/embeddings) |

Chat completions return text. A chat request that asks for `image` or `audio` output in `modalities` fails with 503 `provider_unavailable` when no endpoint produces that output; use the dedicated image and speech endpoints instead. Video isn't accepted on any endpoint.

## Content parts

A chat message's `content` can contain text and structured parts within the
request's aggregate size and JSON-complexity limits. Send media using the shapes
below so it passes acquisition, modality and resource checks. Unrecognized
media forms, including `video_url` and native inline/file-source objects, are
rejected. Extra media locators beside a supported part are also rejected.

Use `user` messages for media across native providers. A native adapter returns
`unsupported_capability` for media on another role when its translation cannot
preserve that content. Matching Chat Completions providers receive the admitted
role and media unchanged.

| Part | Shape | Source |
| --- | --- | --- |
| Text | `{"type": "text", "text": "..."}` | Inline text |
| Image | `{"type": "image_url", "image_url": {"url": "..."}}` | HTTPS URL or base64 `data:` URL |
| PDF | `{"type": "file", "file": {"filename": "...", "file_data": "..."}}` | HTTPS URL or base64 `data:` URL |
| Audio | `{"type": "input_audio", "input_audio": {"data": "...", "format": "wav"}}` | Bare base64 only |

```request
{
  "method": "POST",
  "path": "/chat/completions",
  "body": {
    "model": "openai/gpt-6-luna",
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "What is in this picture?" },
          { "type": "image_url", "image_url": { "url": "https://example.com/images/street.jpg" } }
        ]
      }
    ]
  }
}
```

The model receives your text and your media as separate ordered lists, so refer to media by position ("the second image") rather than by where it sits between sentences.

## Sending media

Providers never receive your URLs, cookies or headers; they receive only the validated media.

- **Remote media** must be a publicly reachable `https://` URL that returns the file uncompressed with a 2xx status. URLs that can't be fetched promptly, or that point to non-public destinations, are rejected with 400.
- **Inline media** uses `data:<media type>;base64,<standard base64>`. Use `image/jpeg`, not `image/jpg`.
- **The bytes must match the declared type.** A file declared as `image/png` that is really a JPEG is rejected. A remote file served as `application/octet-stream`, or without a `Content-Type`, is accepted when its bytes are a supported format.

| Kind | Formats |
| --- | --- |
| Images | PNG, JPEG, GIF, WebP |
| Documents | PDF |
| Audio | WAV, MP3, FLAC, M4A, Ogg, WebM, AAC, AIFF, raw 24 kHz mono PCM |

| Limit | Value |
| --- | --- |
| One media item | 8 MiB decoded |
| All media in one request | 24 MiB decoded |
| All audio in one request | 1 hour |
| Request body | 10 MiB for chat, images and embeddings; 25 MiB for transcription; 1 MiB for speech |

Base64 adds about a third to a file's size, and inline media counts toward the request body. HTTPS URLs don't, so use them for large images and PDFs. See [Limits](/docs/api-reference/limits) for rate limits.

## Find models that support a modality

Each model row in `GET /models` has an `architecture` object:

```json title="Excerpt of a model row"
{
  "id": "author/vision-model",
  "architecture": {
    "modality": "text+image->text",
    "input_modalities": ["text", "image"],
    "output_modalities": ["text"]
  },
  "embeddings": false
}
```

Filter by output with `output_modalities`, a comma-separated list. Values are `text`, `image`, `embeddings`, `speech` and `transcription`:

```request
{
  "method": "GET",
  "path": "/models",
  "query": { "output_modalities": "image" }
}
```

A model row describes one representative endpoint, and endpoints for the same model can accept different inputs. Before you depend on image, file or audio input, check each endpoint's `input_modalities` in [`GET /models/{author}/{slug}/endpoints`](/docs/api-reference/models/list-model-endpoints). The [catalogue](/models) offers the same filters.

Routing sends a request only to endpoints that support every modality it uses, and never drops a media part to make a request fit. If no endpoint qualifies, the request fails with 503 `provider_unavailable`.

## Billing by modality

Each endpoint prices media either in its own units or as prompt tokens. You're charged for the units in your request's final usage at the endpoint's rates. Failed requests that deliver no output aren't charged.

| Unit | What it counts |
| --- | --- |
| `input_images` | Images you send, including reference images for image generation |
| `output_images` | Generated images delivered to you |
| `input_audio_seconds` | Seconds of audio you send |
| `output_audio_seconds` | Seconds of speech delivered to you |
| `input_characters` | Characters of text sent for speech |
| Tokens and requests | As for any chat request |

`pricing.image` and `pricing.audio` in `GET /models` show the lowest image and audio rate of a model's representative endpoint. Before a request runs, an upper bound of its cost is held against your balance; for image generation and speech, that bound covers every requested image and the longest possible audio. `usage.cost` is the exact charge, and [`GET /generation`](/docs/api-reference/chat/get-generation) is the authoritative record. See [Usage accounting](/docs/account/usage-accounting).

## Next steps

:::cards
- {image} [Image inputs](/docs/multimodal/images) Send photos, screenshots and diagrams to vision models.
- {file} [PDFs and files](/docs/multimodal/pdfs) Ask questions about PDF documents.
- {audio} [Audio inputs](/docs/multimodal/audio) Send recorded audio to chat models.
- {sparkles} [Image generation](/docs/multimodal/image-generation) Create images from a prompt and reference images.
- {waves} [Speech and transcription](/docs/multimodal/speech) Turn text into audio and audio into text.
- {layers} [Embeddings](/docs/multimodal/embeddings) Create vectors for search, clustering and retrieval.
:::