Audio Input
PCM format, sample rate, channels, chunk size, and latency tradeoffs.
The realtime API accepts PCM16LE audio. This page covers format requirements, frame size, and latency tradeoffs.
Format requirements
| Property | Requirement |
|---|---|
| Encoding | PCM16LE (16-bit signed little-endian) |
| Sample rate | 16000 Hz |
| Channels | 1 (mono) |
Audio not matching this format will be rejected or produce incorrect results. Convert with ffmpeg first.
Converting audio
ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav
Transport methods
Text frames (base64)
{ "type": "input_audio_buffer.append", "audio": "<PCM16LE base64>" }
Has ~33% base64 overhead. Good for debugging and quick integration.
Binary frames (raw PCM)
Send raw ArrayBuffer directly. No encoding overhead. Recommended for production.
Frame size and latency
| Frame size | Interval | Data size | Latency impact |
|---|---|---|---|
| 50ms | 50ms | 1600 bytes | Lowest latency, higher CPU overhead |
| 100ms | 100ms | 3200 bytes | Recommended balance |
| 200ms | 200ms | 6400 bytes | Increased latency, fewer frames |
| 500ms | 500ms | 16000 bytes | Noticeable latency |
100ms frames (3200 bytes) is the recommended sweet spot.
Max frame size
1 MiB (1,048,576 bytes). Exceeding this returns audio_frame_too_large and closes the connection (1009).
Browser capture
const ctx = new AudioContext({ sampleRate: 16000 });
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const source = ctx.createMediaStreamSource(stream);
// Use AudioWorklet or ScriptProcessorNode to convert float samples to PCM16LE
// Ensure 16000 Hz sample rate and mono
Server-side capture
- Use
Authorization: Bearer sk-...header - Send binary PCM frames directly (no base64 needed)
- Keep sending to avoid idle timeout
Backpressure
When the upstream cannot keep up:
| Condition | Behavior |
|---|---|
| Soft threshold (1 MiB in-flight) | Drop frames, send lanson.throttled |
| Hard threshold (8 MiB in-flight) | Close connection (1013) |
Monitor lanson.throttled events and reduce send rate accordingly.
Related
- Client Messages — audio message format
- Connection Lifecycle — idle timeout
- Realtime Quickstart — complete example