# Audio inputs

> Send recorded audio to audio-capable chat models with the input_audio content part.

Source: https://0.0.0.0:3000/docs/multimodal/audio
Documentation index: https://0.0.0.0:3000/docs/llms.txt

Audio-capable chat models can answer questions about a call, summarize a voice memo or follow spoken instructions. Add an `input_audio` content part to a `user` message with the audio as base64 and its format. The examples use `openai/gpt-6-luna`; replace it with a model that has an endpoint whose `input_modalities` include `audio`. If you only need the spoken words, use [transcription](/docs/multimodal/speech#transcription).

## Send an audio file

```json title="Request body"
{
  "model": "openai/gpt-6-luna",
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "What does the caller ask for, and what tone do they use?" },
        { "type": "input_audio", "input_audio": { "data": "UklGRiQAAABXQVZFZm10...", "format": "wav" } }
      ]
    }
  ]
}
```

Read the file, encode it with standard base64 and pick the `format` that matches its container:

:::code-group
```python title="Python"
import base64
import os
from pathlib import Path

from openai import OpenAI

FORMATS = {".wav": "wav", ".mp3": "mp3", ".flac": "flac", ".m4a": "m4a", ".ogg": "ogg",
           ".webm": "webm", ".aac": "aac", ".aiff": "aiff", ".aif": "aiff"}

def audio_part(path: str) -> dict:
    file = Path(path)
    data = base64.b64encode(file.read_bytes()).decode("ascii")
    return {"type": "input_audio", "input_audio": {"data": data, "format": FORMATS[file.suffix.lower()]}}

client = OpenAI(base_url="https://api.numes.ai/api/v1", api_key=os.environ["NUMES_API_KEY"])

completion = client.chat.completions.create(
    model="openai/gpt-6-luna",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Summarize this voice memo as a to-do list."},
                audio_part("memo.mp3"),
            ],
        }
    ],
)
print(completion.choices[0].message.content)
```

```typescript title="TypeScript"
import { readFile } from "node:fs/promises";
import { extname } from "node:path";
import OpenAI from "openai";

const FORMATS: Record<string, string> = {
  ".wav": "wav", ".mp3": "mp3", ".flac": "flac", ".m4a": "m4a", ".ogg": "ogg",
  ".webm": "webm", ".aac": "aac", ".aiff": "aiff", ".aif": "aiff",
};

async function audioPart(path: string) {
  const format = FORMATS[extname(path).toLowerCase()];
  if (!format) throw new Error(`Unsupported audio type: ${path}`);
  const data = (await readFile(path)).toString("base64");
  // The SDK types list only wav and mp3; the API accepts every format above.
  return { type: "input_audio" as const, input_audio: { data, format: format as "wav" | "mp3" } };
}

const client = new OpenAI({ baseURL: "https://api.numes.ai/api/v1", apiKey: process.env.NUMES_API_KEY });

const completion = await client.chat.completions.create({
  model: "openai/gpt-6-luna",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Summarize this voice memo as a to-do list." },
        await audioPart("memo.mp3"),
      ],
    },
  ],
});
console.log(completion.choices[0]?.message.content);
```
:::

`data` is the bare base64 string: no `data:audio/...;base64,` prefix, and no URL. Audio is always inline, so it counts toward the 10 MiB chat body limit after base64 adds about a third. Keep the audio in one request under about 7.5 MiB, compress it (for example mono Ogg or low-bitrate MP3), or use [transcription](/docs/multimodal/speech#transcription) for long recordings.

## The input_audio part

:::fields Content part
- `type` `"input_audio"` required — Identifies an audio part.
- `input_audio` `object` required — The audio clip.
  - `data` `string` required — Standard base64 of the complete audio file. Must not be empty.
  - `format` `string` required — `wav`, `mp3`, `flac`, `m4a`, `ogg`, `webm`, `aac`, `aiff`, `pcm16` or `pcm24`.
:::

You can attach several clips to one message; they reach the model in the order you list them.

## Supported formats

The `format` you declare must match the file's bytes. A mismatch, such as `mp3` for an M4A file, is rejected.

| `format` | Media type | Notes |
| --- | --- | --- |
| `wav` | `audio/wav` | RIFF WAVE |
| `mp3` | `audio/mpeg` | With or without an ID3 tag |
| `flac` | `audio/flac` | |
| `m4a` | `audio/mp4` | MP4 audio container |
| `ogg` | `audio/ogg` | Opus or Vorbis |
| `webm` | `audio/webm` | Some browser recordings are rejected; see below |
| `aac` | `audio/aac` | ADTS stream |
| `aiff` | `audio/aiff` | AIFF or AIFF-C |
| `pcm16` | `audio/pcm` | Raw PCM, 24 kHz, mono, 16-bit samples |
| `pcm24` | `audio/pcm` | Raw PCM, 24 kHz, mono, 24-bit samples |

Raw PCM has no header, so it's read as 24 kHz mono, and its length must be a whole number of samples. Resample other PCM to 24 kHz first, or wrap it in a WAV header.

Every clip must have a measurable duration, and all audio in one request may last at most 1 hour. Audio that can't be parsed, is truncated or exceeds the limit is rejected with 400. Some browser recordings, such as WebM files from `MediaRecorder`, can't be measured; re-mux them or send Ogg or WAV.

## Route to audio-capable endpoints

Routing considers only endpoints whose `input_modalities` include `audio`. Check a model's endpoints first:

```request
{
  "method": "GET",
  "path": "/models/author/audio-model/endpoints"
}
```

## Billing

Endpoints price input audio either per second, reported as `input_audio_seconds` in `usage`, or as prompt tokens. Text tokens and any per-request price apply as usual. `pricing.audio` in `GET /models` shows the lowest audio rate of the model's representative endpoint. See [Usage accounting](/docs/account/usage-accounting).

## Chat audio or transcription

| | Chat with `input_audio` | [Transcription](/docs/multimodal/speech#transcription) |
| --- | --- | --- |
| Result | A model's answer about the audio | The spoken words as text |
| Upload | Base64 inside the JSON body | Multipart upload or base64 JSON |
| Practical size | About 7.5 MiB of audio per request | Up to 8 MiB per file |
| Text instructions | Yes | No |

To produce speech, use [text to speech](/docs/multimodal/speech#text-to-speech).

## Troubleshooting

Audio problems return 400 `invalid_request`. On chat completions the message names the failing part, such as `messages[0].content[1].input_audio`.

| Symptom | Cause and fix |
| --- | --- |
| 400 with a correct-looking file | `format` doesn't match the bytes, for example an `.mp3` file that is really M4A. |
| 400 with a `data:` prefix or a URL in `data` | Send bare base64 of the file itself. |
| 400 for a browser WebM recording | Re-mux it or send Ogg or WAV. |
| 400 for PCM | The data length isn't a whole number of 16-bit or 24-bit samples. |
| 400 for long audio | A clip exceeds 8 MiB or all audio exceeds 1 hour. Split it or use transcription. |
| 413 `payload_too_large` | The base64 audio pushed the body over 10 MiB. |
| 503 `provider_unavailable` | No endpoint for the model accepts audio input. |

## Next steps

:::cards
- {waves} [Speech and transcription](/docs/multimodal/speech) Transcribe recordings and generate speech.
- {image} [Image inputs](/docs/multimodal/images) Send images to vision models.
- {boxes} [Multimodal](/docs/multimodal/overview) Media rules, limits and billing for every modality.
- {receipt} [Usage accounting](/docs/account/usage-accounting) Read usage and cost.
:::