Ox API Documentation

Voice · developer preview

Create an Ox Voice Agent session

Exchange your project bearer key for a short-lived client secret and configure the voice pipeline on your server. Ox owns streaming audio, turn detection, transcription, reasoning, speech, barge-in, and stage metrics. Your product owns telephony, CRM, prompts, workflow, and tool execution.

1 · Create session

curl -sS "https://ox.inc/v1/realtime/sessions" \
  -X POST \
  -H "Authorization: Bearer $PANDAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ox/voice-agent",
    "session": {
      "instructions": "Answer accurately and ask one question at a time.",
      "language": "en",
      "voice": "Ryan",
      "reasoning": {
        "model": "deepseek/deepseek-v4-flash-0731",
        "temperature": 0.3,
        "max_output_tokens": 512
      },
      "endpointing": {"eagerness": "medium"},
      "tools": [{
        "type": "function",
        "name": "lookup_product",
        "description": "Read customer-owned product data.",
        "parameters": {"type": "object", "properties": {}}
      }]
    }
  }'

Response

{
  "object": "realtime.session",
  "model": "ox/voice-agent",
  "url": "wss://ox.inc/v1/realtime?session_id=rts_123&model=ox%2Fvoice-agent",
  "client_secret": {
    "value": "short_lived_token",
    "expires_at": 1786646400
  },
  "audio": {
    "input": {"format": "pcm16", "sample_rate_hz": 16000, "frame_ms": 80},
    "output": {"format": "pcm16", "sample_rate_hz": 24000, "packet_ms": 20}
  },
  "server_configured": true,
  "routing": {
    "strategy": "single_provider",
    "candidates": [{"provider": "ox", "model": "ox/voice-agent"}]
  }
}

2 · Create on server, connect from client

// Run this fetch on your server. Never ship PANDAN_API_KEY to a browser.
const session = await fetch(
  "https://www.pandanlabs.co/v1/realtime/sessions",
  {
    method: "POST",
    headers: { Authorization: `Bearer ${PANDAN_API_KEY}` },
  },
).then((response) => response.json());

// Return only the short-lived session object to your authenticated client.
const socket = new WebSocket(session.url, [
  "pandan-realtime",
  `auth.${session.client_secret.value}`,
]);
socket.binaryType = "arraybuffer";

// Send exact 80 ms PCM16 mono frames as binary ArrayBuffers.
// Binary messages received from Ox are 24 kHz PCM16 output packets.
// Execute response.function_call_arguments.done in your application and return
// conversation.item.create with a matching function_call_output.

The browser playground at /voice-agent implements the same audio and interruption contract. The native speech-to-speech model remains available separately at /voice.

Voice protocol

OpenAI-Realtime-like events

Direction	Event	Purpose
Client	`session.update`	Configure instructions, audio formats, and tools after `session.created`.
Client	binary PCM16	Append exactly one 80 ms, 16 kHz mono input frame without base64 overhead.
Server	binary PCM16	Stream 24 kHz mono output audio in 20 ms packets.
Server	`output.transcript.*`	Stream assistant transcript deltas and the completed transcript.
Server	`input.transcript.*`	Stream revisable user partials and the final transcript.
Server	`input.speech_started`	Confirm an interruption so clients can immediately clear queued playback.
Server	`response.function_call_arguments.done`	Ask the customer application to execute an allow-listed tool.
Server	`trace.stage`	Report endpoint, reasoner, tool, TTS, turn, and cancellation latency.
Either	`session.close` / `session.end`	Request and confirm a bounded clean shutdown.

Preview limits: two minutes per session, no availability SLA, and English-first testing. The current default stack is Silero VAD 6.2, Smart Turn v3.2, Qwen3-ASR 0.6B, a customer-selected OpenAI-compatible reasoning model, and Qwen3-TTS 1.7B. A preview is not a quality or cost guarantee; promote it only after a shadow evaluation against the incumbent agent traffic.

Voice operations

Terminal session webhooks

Register a project HTTPS endpoint to receive completed and failed session events. Delivery uses a durable outbox with retries. Verify the signature against the exact raw request body before parsing it; tool calls stay on the live connection and do not wait for a webhook.

Signature input

HMAC-SHA256(
  webhook_secret,
  `${webhook_id}.${webhook_timestamp}.${raw_body}`
)

// Request headers
webhook-id: evt_...
webhook-timestamp: 1786646400
webhook-signature: v1,<base64-hmac>

Authentication

Use a bearer key

Every /v1 request requires your Ox API key.

Environment

export PANDAN_API_BASE="https://ox.inc"
export PANDAN_API_KEY="your_api_key"

Quickstart

Ask about a video

Use the OpenAI client or call /v1/chat/completions directly.

Python · OpenAI SDK

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["PANDAN_API_KEY"],
    base_url="https://pandan--pandan-api-api.modal.run/v1",
    timeout=600,
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-30B-A3B-Instruct",
    messages=[{"role": "user", "content": [
        {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}},
        {"type": "text", "text": "Summarize the sequence of events."},
    ]}],
    temperature=0,
)

print(response.choices[0].message.content)

curl

curl --max-time 600 "$PANDAN_API_BASE/v1/chat/completions" \
  -H "Authorization: Bearer $PANDAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-VL-30B-A3B-Instruct",
    "messages": [{"role": "user", "content": [
      {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}},
      {"type": "text", "text": "Summarize the sequence of events."}
    ]}],
    "temperature": 0
  }'

Non-streaming answers are returned in choices[0].message.content.

Video

Choose a workflow

Workflow	Best for	How it works
Video URL	One question	Send `video_url` in a chat completion.
Reusable media	Multiple questions	Upload once and reuse the returned `media_id`.

Reusable media

Upload once. Reuse the ID.

Use reusable media when you will ask more than one question about the same file. Uploads are limited to 256 MiB.

Type	Formats	Content-Type
Images	JPEG, PNG, WebP	`image/jpeg`, `image/png`, `image/webp`
Video	MP4, WebM, QuickTime	`video/mp4`, `video/webm`, `video/quicktime`

1. Upload the file

POST /v1/media

curl --max-time 600 "$PANDAN_API_BASE/v1/media" \
  -H "Authorization: Bearer $PANDAN_API_KEY" \
  -H "Content-Type: video/mp4" \
  --data-binary @video.mp4

Response

{
  "id": "media_<sha256>_mp4",
  "object": "media",
  "content_type": "video/mp4",
  "bytes": 1234567,
  "created": true
}

Persist id with your record. Uploading identical bytes returns the same ID with created: false.

2. Ask with the returned ID

POST /v1/chat/completions

curl --max-time 600 "$PANDAN_API_BASE/v1/chat/completions" \
  -H "Authorization: Bearer $PANDAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-VL-30B-A3B-Instruct",
    "media_ids": ["media_<sha256>_mp4"],
    "messages": [{
      "role": "user",
      "content": "Which person enters the room first?"
    }],
    "temperature": 0,
    "max_tokens": 128
  }'

3. Ask follow-up questions

Reuse media_ids. Include prior messages only when the next question depends on them.

media_id is a content-addressed file reference, not a semantic index.

Streaming

Receive tokens as they are generated

Set stream: true. The response uses server-sent events and ends with data: [DONE].

curl · SSE

curl -N --max-time 600 "$PANDAN_API_BASE/v1/chat/completions" \
  -H "Authorization: Bearer $PANDAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-VL-30B-A3B-Instruct",
    "media_ids": ["media_<sha256>_mp4"],
    "messages": [{"role": "user", "content": "Describe the main event."}],
    "stream": true,
    "stream_options": {"include_usage": true},
    "temperature": 0
  }'

Read each data: event and concatenate choices[0].delta.content. With include_usage, the final JSON event includes token usage.

Best practices

Practical defaults

Reuse media IDs

Lower upload overhead and preserve warm cache reuse.

Ask precisely

Name the event, object, person, or time relationship.

Use `temperature: 0`

Best for extraction and repeatable evaluations.

Expect cold starts

An idle first request can take several minutes.

Operations

Runtime and limits

Behavior	Current beta
Client timeout	Allow up to 600 seconds for requests that may include a cold start.
Idle capacity	Compute scales to zero after about five idle minutes. The next request may take several minutes while the model starts.
Media reuse	A `media_id` references the stored upload. Processor and prefix-cache acceleration apply while the serving container remains warm.
Capacity	Private-beta capacity is bounded and has no availability SLA. Contact us before sustained or high-concurrency traffic.

API reference

Endpoints

Method	Path	Purpose
GET	/health	Returns 200 after the model server is ready. Public; may wake the GPU.
GET	/v1/models	Lists the currently served model. Bearer authentication required.
POST	/v1/media	Stores image or video bytes and returns a content-addressed ID.
HEAD	/v1/media/{media_id}	Checks whether an uploaded media ID exists.
POST	/v1/chat/completions	Runs OpenAI-compatible text, image, or video inference. Streaming supported.
POST	/v1/realtime/sessions	Exchanges an Ox bearer key for a short-lived native VoiceChat or configurable Ox Voice Agent session.
GET	/v1/realtime/health	Reports non-secret voice configuration, durable-state readiness, and admission candidates.
GET	/v1/realtime/sessions/{id}	Returns durable lifecycle state and finalized transport usage.
POST	/v1/realtime/calls	Starts an outbound E.164 call through the configured LiveKit SIP trunk.
POST	/v1/projects/{project_id}/api-keys	Control plane: issues a project-scoped key and returns its value once.
DELETE	/v1/projects/{project_id}/api-keys/{key_id}	Control plane: revokes a managed API key.
GET / POST	/v1/projects/{project_id}/webhooks	Lists or registers signed terminal-session webhooks.
GET	/v1/projects/{project_id}/usage	Returns daily session and audio usage aggregates.
GET	/v1/projects/{project_id}/audit-logs	Returns append-only project security and operations events.
WSS	/v1/realtime	Runs one full-duplex voice session with binary PCM media and JSON control events.

Reference

Errors

400Invalid request body, model input, or unknown media_id.
401Missing or invalid bearer key.
404Uploaded media was not found when checking it with HEAD.
413Empty upload or media larger than 256 MiB.
415Unsupported media content type.
5xxModel startup or inference failure. Retry with bounded exponential backoff.

Pricing

Private beta

$0within assigned limits

Invite only. No SLA during beta. Production rates will be published before billing begins. Request access.