ASR + TTS · evidence board

Economics demonstrated. Parity is not.

Ox is evaluating focused replacements for Scribe v2 Realtime and Eleven v3. Independent public benchmarks screen the field; Ox measurements show deployment cost and behavior. Different harnesses stay visibly separate.

Public baseline snapshot · August 17, 2026 · no blended metrics

01 / Public benchmark baseline

What we can establish without calling ElevenLabs.

Same-harness third-party results are the baseline. Provider studies are shown separately and labeled.

Independent · same streaming harness

Scribe owns accuracy. Open weights own the speed axis.

Artificial Analysis AA-WER Streaming snapshot · August 17, 2026

Model	Final WER	After-speech latency	Price / 1k audio min	Readout
ElevenLabs Scribe v2 RealtimeReference	3.59%	141 ms final · 134 ms first	$6.50	Best accuracy of this shortlist.
NVIDIA Nemotron 3.5 0.6B · 80 msOpen weights	8.38%	73 ms final · 41 ms first	Self-hosted	1.93× faster final and 3.28× faster first event, with 2.33× the WER.
Mistral Voxtral Mini RealtimeOpen weights + hosted	5.24%	682 ms final · 320 ms first	$6.00	8% lower API price, but 46% more errors and 4.83× final latency.

AA streams roughly eight hours of audio in realtime and measures from Silero VAD-detected end of speech. Its current Nemotron entry is the newer multilingual 3.5 checkpoint—not the English checkpoint in the Ox pilot.

Artificial Analysis streaming leaderboard ↗

Independent · provider voice arena

The best open-weight TTS trade is speed and price—not an overall quality win.

Artificial Analysis Speech Arena snapshot · August 17, 2026

Model	Quality Elo	Generation speed	API price / 1M chars	Readout
ElevenLabs Eleven v3Standalone reference	1,177 ±12	41.0 chars / sec	$100	Independent quality leader over the open-weight shortlist.
Fish Audio S2 ProOpen weights · commercial license required	1,124 ±13	94.7 chars / sec	$15	2.31× faster and 6.67× lower API price, 53 Elo behind Eleven v3.
Mistral Voxtral TTSOpen weights · noncommercial license	1,081 ±13	88.8 chars / sec	$16	2.17× faster and 6.25× lower API price, 96 Elo behind Eleven v3.
Qwen-Audio 3.0 TTS PlusHosted control · not open weights	1,237 ±15	Not reported here	$27.60	The public quality-and-price winner: higher Elo and 3.62× lower price than v3.

AA uses each provider's native voices here. The controlled-voice arena still ranks Eleven v3 above Fish and Voxtral. Neither arena covers the Agents-only eleven_v3_conversational model or Ox's exact Qwen CustomVoice checkpoint.

Artificial Analysis TTS leaderboard ↗

Fish-run direct preference testVendor reported

60% vs 40%

S2 Pro beat Eleven v3 across 581 direct blind pairs in Fish Audio's production study. Useful directional evidence, but provider-run and contradicted by AA's overall Elo ordering.

Fish Audio study ↗

Voxtral implicit emotion steeringVendor reported

55.4% win rate

Mistral's native-listener study narrowly preferred Voxtral over Eleven v3 when emotion had to be inferred from text; explicit steering was 51.0%. Released weights are CC BY-NC.

Voxtral TTS paper ↗

Controlled-voice reality checkThird-party

1,059 vs 1,004

AA's same-eight-voice arena scores Eleven v3 at 1,059 ±13 and Fish S2 Pro at 1,004 ±12. Public quality evidence is mixed, not parity.

AA controlled-voice arena ↗

02 / Ox replacement scorecard

Where the case stands today

Candidate measurements and published baselines remain separate until a paired customer trial.

01 / Speech recognition

Scribe v2 Realtime replacement

2.21× raw-cost gap at c8

Ox challengerNemotron EN 0.6B native c8 · public Nemotron 3.5 reference

Reference stackElevenLabs Scribe v2 Realtime

Ox demonstrated a lower raw-compute denominator on the English checkpoint. Separately, AA shows the newer open-weight Nemotron 3.5 at 80 ms finishing 1.93× faster than Scribe after speech end, but with 2.33× the WER. These are distinct experiments—not a combined parity result.

Nemotron raw compute / audio minOx modeled

$0.00294

Modeled from measured A10 c8 realtime density; excludes idle, HA, networking, and operations.

Native c8 artifact ↗

Modeled active-compute gapOx modeled

2.21× lower

At eight continuously occupied streams. Scribe list price divided by raw active compute; the modeled advantage disappears below four occupied streams and is not an invoice comparison.

Native c8 artifact ↗

Nemotron synthetic entity recallOx measured

10 / 10

Exact normalized entity matches across five single-voice synthetic medical utterances; not a customer-corpus result.

Native c8 artifact ↗

Nemotron synthetic corpus WEROx measured

10.1%

Five single-voice synthetic medical utterances; not comparable to Scribe's third-party corpus.

Native c8 artifact ↗

Nemotron first text p95Ox measured

809 ms

Server-side native emission at 80 ms cadence and A10 c8; excludes network and is not directly comparable to Scribe's published boundary.

Native c8 artifact ↗

Nemotron c8 compute cadenceOx measured

0 misses

42.9 ms p95 compute step against the 80 ms realtime deadline; eight submitted streams.

Native c8 artifact ↗

Parakeet first text p95Ox measured

353 ms

Server-side native emission at c8 and 80 ms cadence; excludes network and cannot be compared with AA's after-speech boundary.

Parakeet c8 artifact ↗

Parakeet tagged-string hitsOx measured

7 / 10

Favorable substring scoring across five synthetic utterances; medical smoke gate failed.

Parakeet c8 artifact ↗

Scribe realtime latencyVendor reported

~150 ms

Published model latency; not yet reproduced on the Ox corpus.

ElevenLabs models ↗

Scribe realtime list priceVendor reported

$0.39 / hr

Published API price before negotiated volume terms.

ElevenLabs pricing ↗

Scribe realtime WERThird-party

3.64%

Third-party streaming benchmark; use the paired customer corpus for the decision.

AA streaming benchmark ↗

Scribe final after speech endThird-party

~140 ms

Third-party streaming result; timing definitions must match in the Ox run.

AA streaming benchmark ↗

Next proof gate

Use AA to avoid a broad Eleven baseline run. Only after domain tuning, run identical customer audio through the surviving candidate and Scribe to establish medical-term accuracy and end-of-speech latency.

02 / Speech generation

Eleven v3 component replacement

26.7× raw-compute denominator gap

Ox challengerQwen3-TTS 12Hz 0.6B · Fish S2 Pro public control

Reference stackEleven v3 standalone · v3 Conversational in Agents

Qwen 0.6B demonstrated strong self-hosted throughput and a low same-unit raw-compute denominator versus the standalone Eleven v3 API. Public AA data identifies Fish S2 Pro as the strongest open-weight quality candidate, with 2.31× generation speed and a 6.67× lower hosted price—but Eleven v3 still leads independent quality Elo.

Qwen first PCM p95Ox measured

458 ms

Warm L40S, c4, n=20; identical 228-character medical-device script.

Candidate summary ↗

Qwen protocol completionsOx measured

20 / 20

One warm, single-prompt c4 pilot; this is not a production reliability result.

Candidate summary ↗

Qwen aggregate goodputOx measured

15.5 audio-s / s

Generated audio seconds per wall second across four concurrent clients.

Candidate summary ↗

Qwen raw compute / 1k charsOx modeled

$0.00375

Derived from the measured 4,560-character c4 pilot; excludes idle capacity, HA, relay, and operations.

Corrected cost derivation ↗

Eleven v3 direct API priceVendor reported

$0.10 / 1k chars

Standalone v3 list price. This is the matched billing unit for the Qwen raw-compute comparison.

ElevenLabs API pricing ↗

Modeled active-compute gapOx modeled

26.7× lower

Standalone v3 API list price divided by Qwen's script-specific raw compute per 1k characters—not customer savings.

Cost derivation ↗

v3 Conversational public TTFAInconclusive

Not published

The Agents-only model is distinct from standalone v3; no numeric public benchmark was found for its first playable audio.

ElevenLabs Expressive Mode ↗

Next proof gate

Use AA and published studies for screening. For the final customer claim, test only the selected finalist against the customer's actual v3 Conversational voice on medical pronunciation, blind preference, first audio, and interruption behavior.

03 / Claim contract

A win means every gate passes

Accuracy, quality, latency, reliability, and cost move independently. We publish the whole scorecard.

ASR quality

WER non-inferior; critical-entity error no worse than baseline

A cheap transcript is not useful if model names, quantities, or dates change.

Realtime behavior

Stable-partial and finalization p95 at or below the baseline

Median-only numbers hide the pauses that make a sales call feel broken.

Voice quality

Blind preference is statistically non-inferior; pronunciation passes

Synthetic-speech speed does not establish trust, clarity, or brand fit.

Economics

At least 2× lower all-in cost at the customer concurrency profile

Includes warm-idle capacity, retries, relay, storage, and failed calls.

Reliability

Success rate and p95 remain inside the gate under sustained load

A four-request burst is not production capacity evidence.

Methodology notes

Read the boundary, not just the number.

Latency boundaries must match. First ASR partial, finalization, and first playable TTS PCM are reported separately.
Raw compute is not a production quote. Warm-idle spend, relay, storage, egress, HA, and failure cost are excluded.
Small samples are directional. These results are engineering evidence, not production percentiles or an SLO.
Quality is a separate gate. Low cost and latency do not establish ASR accuracy or blind TTS preference.