An independent benchmark of 17 speech-to-text systems on whether the transcript preserves exact structured values in English workplace speech - callback numbers, email addresses, command-line flags, file paths, URLs, account identifiers, dates, and amounts. 300 human-recorded segments, 1,482 audited target entities across 26 types - the besimple-ai/voice-code-bench dataset by Besimple AI, released MIT on Hugging Face.
Every system gets the same raw audio and nothing else. A versioned LLM verifier (gpt-5.6-sol) judges each target entity, held constant across systems. The headline metrics are TSR (recordings with every value correct) and CTEM (values recovered exactly), with format-invariant WER as the broad diagnostic.
On this snapshot, Deepgram Nova-3 leads recording-level exactness (TSR 71.0%) and ElevenLabs Scribe v2 leads value-level recovery (CTEM 91.8%). The lowest-WER system, AssemblyAI Universal 3.5 Pro (3.9% WER), gets every structured value right in only 50.3% of recordings - clean-sounding transcripts are not the same thing as usable ones.
“Most accurate STT” has no single answer. Ask what consumes the transcript - an automated system that parses values, a human who skims prose, a realtime agent - then route to the metric that prices that failure mode.
Deepgram Nova-3
When the transcript is parsed, routed, stored, or executed downstream, one wrong digit fails the task. Task Success Rate is the metric that prices that risk: every target value in the recording must be exactly recoverable.
ElevenLabs Scribe v2
If a human reviews the output and partial recovery still saves time, rank by CTEM - the fraction of individual structured values recovered exactly, independent of whether the rest of the recording was perfect.
Deepgram Nova-3
Voice agents and live captioning can't wait for batch. Streaming modes are scored as separate systems on realtime-paced audio; this is the current streaming exactness leader.
AssemblyAI Universal 3.5 Pro
For meeting notes and summaries where structured values are incidental, format-invariant WER is the right diagnostic - but note this leader recovers every structured value in only 50.3% of recordings.
The router above is the recommendation surface. This section keeps the measured TSR / CTEM / WER facts crawlable and breaks entity recovery down across all 26 structured types.
| Rank | System | Mode | TSR | CTEM | WER |
|---|---|---|---|---|---|
| 1 | Deepgram Nova-3 | batch | 71.0% | 91.4% | 8.2% |
| 2 | ElevenLabs Scribe v2 | batch | 68.0% | 91.8% | 6.8% |
| 3 | Deepgram Nova-3 | streaming | 61.0% | 88.3% | 9.5% |
| 4 | ElevenLabs Scribe v2 Realtime | streaming | 59.0% | 88.1% | 5.1% |
| 5 | Google Cloud Chirp 3 | batch | 58.0% | 88.7% | 5.8% |
| 6 | OpenAI GPT Realtime (Whisper) | streaming | 57.3% | 88.7% | 8.6% |
| 7 | Google Cloud Chirp 3 | streaming | 56.3% | 87.3% | 6.8% |
| 8 | Inkling (Thinking Machines) · managed endpoint via Modal | batch | 54.7% | 85.3% | 5.0% |
| 9 | OpenAI GPT-4o Transcribe | batch | 54.0% | 87.0% | 4.6% |
| 10 | Whisper large-v3 | batch | 53.3% | 87.3% | 5.9% |
| 11 | AssemblyAI Universal 3.5 Pro | batch | 50.3% | 86.2% | 3.9% |
| 12 | Inworld STT 1 | batch | 48.3% | 85.4% | 8.2% |
| 13 | NVIDIA Parakeet TDT 0.6B v3 · self-hosted via Modal | batch | 39.3% | 79.8% | 11.4% |
| 14 | Amazon Transcribe | streaming | 35.0% | 78.8% | 7.5% |
| 15 | Meta OmniASR LLM Unlimited 7B v2 · self-hosted via Modal | batch | 32.7% | 76.9% | 10.5% |
| 16 | AssemblyAI Universal 3 Pro | streaming | 24.7% | 71.9% | 7.5% |
| 17 | Inworld STT 1 | streaming | 7.3% | 34.0% | 58.2% |
--dry-run, DATABASE_URL, a callback number - that downstream software needs verbatim.| Entity type | Mean CTEM | Best system | Best CTEM |
|---|---|---|---|
| Command | 48.1% | OpenAI GPT Realtime (Whisper) · streaming | 74.0% |
| URL | 49.3% | Deepgram Nova-3 · batch | 69.3% |
| File Path | 51.6% | OpenAI GPT Realtime (Whisper) · streaming | 66.7% |
| Email Address | 55.2% | Whisper large-v3 · batch | 78.5% |
| Postal Address | 56.0% | ElevenLabs Scribe v2 · batch | 72.5% |
| Environment Variable | 78.7% | ElevenLabs Scribe v2 · batch | 97.1% |
| CLI Flag | 80.5% | OpenAI GPT Realtime (Whisper) · streaming | 95.5% |
| IP Address | 82.4% | Deepgram Nova-3 · batch | 96.0% |
| System | Email Address | Phone Number | Phone Extension | Person Or Team Name | Postal Address | URL | IP Address | Port Number | Command | CLI Flag | File Path | Environment Variable | Code Symbol | Version | Reference ID | Product Code | Account Or Record Number | Currency Amount | Percentage | Measurement | Plain Number | Date | Time | Acronym Or Initialism | Spelled Sequence | Domain Term |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Deepgram Nova-3 · batch | 77% | 98% | 100% | 100% | 70% | 69% | 96% | 97% | 56% | 91% | 61% | 86% | 86% | 93% | 95% | 96% | 97% | 100% | 100% | 100% | 100% | 100% | 97% | 99% | 98% | 90% |
| ElevenLabs Scribe v2 · batch | 69% | 100% | 100% | 100% | 73% | 65% | 96% | 100% | 72% | 93% | 65% | 97% | 97% | 93% | 93% | 91% | 94% | 100% | 100% | 100% | 100% | 99% | 98% | 96% | 98% | 100% |
| Deepgram Nova-3 · streaming | 69% | 98% | 100% | 100% | 68% | 63% | 92% | 90% | 60% | 89% | 53% | 80% | 77% | 93% | 93% | 90% | 92% | 96% | 100% | 98% | 98% | 96% | 97% | 95% | 96% | 90% |
| ElevenLabs Scribe v2 Realtime · streaming | 71% | 98% | 100% | 100% | 60% | 47% | 96% | 97% | 50% | 84% | 47% | 54% | 97% | 100% | 95% | 92% | 92% | 92% | 100% | 100% | 100% | 100% | 98% | 99% | 98% | 100% |
| Google Cloud Chirp 3 · batch | 66% | 98% | 100% | 100% | 57% | 61% | 96% | 97% | 54% | 91% | 59% | 89% | 91% | 90% | 91% | 92% | 94% | 97% | 98% | 100% | 100% | 100% | 93% | 96% | 92% | 90% |
| OpenAI GPT Realtime (Whisper) · streaming | 65% | 90% | 93% | 94% | 53% | 58% | 84% | 97% | 74% | 95% | 67% | 91% | 94% | 90% | 93% | 93% | 91% | 89% | 100% | 98% | 98% | 100% | 98% | 98% | 91% | 95% |
| Google Cloud Chirp 3 · streaming | 68% | 97% | 100% | 100% | 55% | 55% | 96% | 97% | 54% | 86% | 63% | 83% | 83% | 87% | 89% | 96% | 88% | 93% | 98% | 98% | 98% | 100% | 92% | 94% | 94% | 95% |
| Inkling (Thinking Machines) · batch | 57% | 92% | 100% | 89% | 57% | 55% | 84% | 83% | 58% | 80% | 59% | 89% | 83% | 97% | 88% | 83% | 86% | 95% | 98% | 100% | 97% | 100% | 95% | 92% | 93% | 95% |
| OpenAI GPT-4o Transcribe · batch | 63% | 100% | 100% | 94% | 53% | 45% | 80% | 97% | 56% | 86% | 57% | 94% | 89% | 93% | 90% | 91% | 88% | 100% | 100% | 100% | 98% | 100% | 98% | 94% | 91% | 85% |
| Whisper large-v3 · batch | 78% | 95% | 100% | 94% | 60% | 42% | 92% | 90% | 46% | 89% | 53% | 89% | 91% | 93% | 93% | 89% | 91% | 97% | 100% | 100% | 95% | 97% | 90% | 96% | 97% | 85% |
| AssemblyAI Universal 3.5 Pro · batch | 55% | 90% | 97% | 100% | 68% | 47% | 88% | 97% | 46% | 86% | 63% | 83% | 100% | 83% | 91% | 86% | 88% | 92% | 98% | 100% | 97% | 100% | 98% | 93% | 95% | 100% |
| Inworld STT 1 · batch | 60% | 90% | 97% | 89% | 55% | 47% | 72% | 90% | 58% | 84% | 67% | 89% | 91% | 90% | 86% | 89% | 91% | 96% | 96% | 95% | 98% | 98% | 93% | 99% | 89% | 90% |
| NVIDIA Parakeet TDT 0.6B v3 · batch | 29% | 83% | 100% | 94% | 45% | 47% | 76% | 83% | 38% | 86% | 37% | 83% | 83% | 93% | 81% | 81% | 82% | 85% | 100% | 100% | 98% | 97% | 93% | 94% | 90% | 85% |
| Amazon Transcribe · streaming | 28% | 98% | 90% | 83% | 53% | 40% | 88% | 87% | 12% | 61% | 37% | 57% | 77% | 87% | 89% | 86% | 86% | 85% | 98% | 90% | 98% | 99% | 98% | 95% | 87% | 95% |
| Meta OmniASR LLM Unlimited 7B v2 · batch | 35% | 73% | 100% | 89% | 48% | 50% | 72% | 87% | 32% | 70% | 24% | 77% | 80% | 73% | 88% | 86% | 85% | 75% | 50% | 90% | 98% | 94% | 92% | 99% | 96% | 80% |
| AssemblyAI Universal 3 Pro · streaming | 38% | 78% | 90% | 94% | 50% | 21% | 60% | 97% | 34% | 73% | 31% | 69% | 77% | 80% | 63% | 72% | 63% | 73% | 96% | 97% | 95% | 90% | 92% | 91% | 79% | 95% |
| Inworld STT 1 · streaming | 9% | 18% | 17% | 72% | 30% | 27% | 32% | 33% | 18% | 23% | 35% | 29% | 31% | 57% | 55% | 31% | 60% | 12% | 6% | 41% | 40% | 46% | 25% | 29% | 53% | 10% |
template (script with entity placeholders), acoustic (what the speaker says aloud), and canonical(the written value a downstream application needs). “double dash dry dash run” maps to --dry-run; “all caps database underscore URL” maps to DATABASE_URL.CTEM · Canonical Token/Entity Match. Correct target entities ÷ target entities. An entity is correct only when the exact canonical value is recoverable - formatting differences are fine, wrong or missing characters are not.TSR · Task Success Rate. Recordings with all target entities correct ÷ recordings. The strictest headline metric: one corrupted digit fails the whole recording.WER · format-invariant Word Error Rate. Every target entity span accepts either the documented acoustic rendering or the canonical rendering as reference, so 212-555-0100and “two one two five five five zero one zero zero” score identically. Non-entity words use ordinary word-level edit distance.per-entity CTEM · CTEM restricted to one of the 26entity types - the full matrix above, useful for per-entity risk analysis (e.g. “can I trust this system with spoken email addresses?”).Every system is evaluated under a raw-audio-only protocol: it receives the audio file and nothing else. Benchmark-specific prompts, target entity lists, domain labels, custom vocabulary, grammar constraints, candidate values, and post-ASR correction are all excluded from the main setting. This measures what the model does on its own, not what a tuned integration could extract from it.
Batch systems receive each recording as a complete file at the provider's documented input format. Streaming systems receive headerless PCM paced in realtime (100 ms chunks) over the provider's websocket protocol, with VAD, turn detection, and vendor canonicalization disabled where the API allows. Batch and streaming endpoints of the same vendor are tracked as separate systems because their error profiles differ substantially.
For each recording, every target entity is checked against the system transcript by a versioned LLM verifier: openai_gpt_5_6_sol_group10_sync_v1 (gpt-5.6-sol, 10 benchmark datapoints grouped per request, reasoning effort medium). The verifier sees the transcript plus each entity's acoustic and canonical forms, and marks an entity present only when enough evidence exists to recover the exact canonical value. Casing, punctuation, spacing, and formatting differences are acceptable only when the same value remains recoverable; wrong, missing, extra, or substituted characters that change the value are rejected.
The verifier's prompt, response schema, grouping, and inference settings are frozen into a config digest that ships with every entity-match artifact, and the same verifier scores every system. For present entities it must cite an exact transcript substring as evidence; for absent ones it records the closest corrupted substring and a reason - so every one of the 25,194 entity decisions behind this page can be audited in seconds.
Everything behind this table is released in openbenchmarks-labs/voice-code-bench: the audio (data/audio/*.wav), transcripts and entity annotations (data/metadata.jsonl), per-system transcripts plus entity-match decisions (baselines/predictions/*.json), the versioned verifier configs, and the scoring CLI. The dataset itself is authored by Besimple AI and also published on Hugging Face as besimple-ai/voice-code-bench (MIT).
The release script rescores the published artifacts and rewrites baselines/results.csv - the exact table rendered on this page - without any provider credentials. Re-running a system live (vcb-transcribe → vcb-extract-entities → vcb-score-entities) requires only that provider's API key plus an OpenAI key for the verifier.
openai_gpt_5_6_sol_group10_sync_v1 (10 datapoints per request, reasoning effort medium), replacing the earlier asynchronous Batch API workflow.