FAQ

Benchmarks run hourly via a scheduled cron job on Modal. Each run sweeps all configured models across LLM, TTS, and STT providers.

All benchmarks execute in Modal's us-east region, co-located with the LiveKit us-east agents cluster. This minimizes network variability and reflects the latency a real voice agent would experience.

The "inference" route sends requests through LiveKit's inference gateway — the same path a production LiveKit agent uses. The "direct" route hits the provider API directly, bypassing the gateway. Comparing both reveals the overhead (or lack thereof) introduced by the gateway. Note: Anthropic models are direct-only as they do not yet have LiveKit Inference support.

First-turn only. Each benchmark iteration sends a system prompt, an agent greeting, and a single user message, then measures the time to generate one completion response. This mirrors the most latency-sensitive moment in a voice agent conversation — the first reply.

Synthetic voice-agent conversations spanning 7 real-world use cases: restaurant reservations, debt collection, customer support, appointment reminders, doctor appointments, payment reminders, and user surveys. Conversations vary in complexity (simple, moderate, complex) and direction (inbound, outbound). System prompts are detailed Jinja2 templates reflecting production agent configurations.

Provider defaults for temperature and max tokens — no overrides are applied. Connection timeout is 5 seconds (10 seconds for Google). There are no retries on API errors (max_retry=0); failed iterations are excluded from results rather than retried, to avoid inflating latency numbers.

TTFT measures the time from when a request is sent to when the first token of the response is received. This is the primary latency metric for LLM providers — it directly determines how long a user waits before the agent starts speaking.

TTFB measures the time from when text is submitted for synthesis to when the first audio byte is returned. This is the primary latency metric for TTS providers — it determines the delay before audio playback can begin.

Transcription delay measures the time from the end of audio input (end_input signal) to the final transcript being received. This is the gating metric for STT providers — it represents the latency that STT adds to the voice agent pipeline after the user finishes speaking.

Tokens per second measures LLM output throughput — how many tokens the model generates per second after streaming begins. Higher values mean faster overall response generation.

Mean is the arithmetic average across all successful iterations. P95 (95th percentile) shows the latency at which 95% of requests complete — it captures tail latency and worst-case behavior. For production voice agents, P95 is often more important than mean because occasional slow responses degrade user experience.

10 iterations per model per hourly run. Each iteration uses a different conversation from the test dataset, providing variance across prompt lengths and complexities.

Failed iterations (API errors, timeouts) are recorded but excluded from aggregate statistics. There are no retries — if a request fails, it fails. This means the metrics you see reflect only successful completions. Models with high failure rates will have lower sample counts visible in the leaderboard.

Benchmarks run hourly and write results to the database immediately. The dashboard uses a 5-minute cache (ISR), so new results appear within minutes of a run completing.

LLM: OpenAI (gpt-4o, gpt-4o-mini, gpt-4.1, gpt-4.1-mini, gpt-4.1-nano), Anthropic (claude-sonnet-4-5, claude-haiku-4-5), Google (gemini-2.5-flash). TTS: Cartesia (sonic-2, sonic-turbo), ElevenLabs (eleven_turbo_v2_5, eleven_flash_v2_5), Deepgram (aura-2-andromeda-en, aura-2-athena-en). STT: Deepgram (nova-3, flux-general-en), AssemblyAI (u3-rt-pro, universal-streaming-english), Cartesia (ink-whisper), ElevenLabs (scribe_v2_realtime).

Benchmarks execute on Modal (serverless compute) in the us-east region. Results are stored in Supabase (hosted PostgreSQL). The dashboard is a Next.js app deployed on Vercel. All three services are independent — the dashboard is read-only and cannot modify benchmark data.

Each provider (e.g., OpenAI, Anthropic, Deepgram) runs as a separate Modal container in parallel. Within a single provider, models are benchmarked sequentially to avoid contention and rate limiting. This means all providers are tested near-simultaneously each hour.

All benchmark runs originate from the same Modal region (us-east), eliminating geographic variance. The hourly cadence over days and weeks smooths out transient network conditions. Time-series charts make it easy to spot and discount anomalous spikes.