--- title: "Independent Structured Speech-to-Text (STT/ASR) Benchmark: Emails, Phones, IDs (2026) | Openbenchmarks" description: "Independent speech-to-text (STT/ASR) benchmark for exact values in downstream tasks. Emails, phones, paths, and IDs. 17 systems, 300 English recordings. Open data." url: https://openbenchmarks.com/structured-speech-to-text-benchmark --- [](https://openbenchmarks.com/) [← all benchmarks](https://openbenchmarks.com/) [benchmarks](https://openbenchmarks.com/) / structured-speech-to-text-benchmark 300 recordings · 17 systems · updated Aug 7, 2026 # Independent Structured Speech-to-Text (STT/ASR) Benchmark ## Why did we build this benchmark? As a standard metric, Word Error Rate (WER) treats every word as equal. A transcript can sound clean and still corrupt the callback number, email or URL that a downstream task needs verbatim. We built this benchmark on a dataset provided by [Besimple AI](https://huggingface.co/besimple-ai) to measure that failure mode and to show that WER and task success can decouple. On this snapshot **AssemblyAI Universal 3.5 Pro** leads WER at 3.9% but ranks 11 of 17 on Task Success Rate (50.3%), while Deepgram Nova-3 and ElevenLabs Scribe v2 are statistically tied at the top on Task Success Rate (71.0% vs 68.0%, within roughly ±2.6pp sampling noise on 300 recordings). ## Definition & problem This is an ASR/STT benchmark of 17 systems, including Deepgram, ElevenLabs, Google Cloud, OpenAI, and Thinking Machines Lab, on 300 human-recorded segments (5.587 hours, 85 speakers) with 1,482 audited target entities across 26 types. Every system gets the same raw audio and nothing else: no prompts, custom vocabulary, or post-ASR cleanup. A versioned LLM verifier (`gpt-5.6-sol`) judges each entity. Headline metrics: **TSR** (recordings with every value correct) and **CTEM** (values recovered exactly), with format-invariant WER as the broad diagnostic. ## How to pick the best STT for your workflow * ### Booking appointments or support callbacks The task must capture a phone number, email, date, and time exactly, since one wrong digit fails the run. TSR is a fair first screen for that job; confirm on the phone, email, date, and time rows in the entity matrix before you ship. * ### Voice coding tasks Do not pick from headline TSR. Rank the command, CLI-flag, file-path, and environment-variable rows in the entity matrix (those leaders are often not the TSR leaders). * ### CRM or form filling from a call Account numbers, reference IDs, and postal addresses are the failure modes that matter, so read those entity rows rather than only the headline rank. * ### Live / realtime tasks vs post-call batch If realtime looks fine in demos but production streams drop digits, compare the same vendor's batch and streaming rows. For one decision guide that combines both meanings of accuracy, see the [best speech-to-text models by use case](https://openbenchmarks.com/structured-speech-to-text-benchmark/most-accurate-speech-to-text-model). It keeps the WER and structured-value scores separate instead of blending incompatible benchmarks into one rank. ## Is there a best STT model? No, not as a single ranking. “Best STT” only makes sense once you name the use case and the failure mode you care about. * ### Downstream task or software that needs every field right Booking, callbacks, form fill: use Task Success Rate on this board. The top systems are often statistically tied within sampling noise; confirm on the entity rows for phones, emails, dates, and IDs. * ### Human-read transcripts, cheapest, or fastest That is a raw Word Error Rate and latency question. Use the [Speech-to-Text benchmark](https://openbenchmarks.com/speech-to-text-benchmark-by-coval), not this page. Changelog Last updated Aug 7, 2026 · 2026 Q3 snapshot: 17 systems (batch + streaming), including Inworld STT 1 and self-hosted Parakeet / OmniASR baselines; all re-verified with gpt-5.6-sol. [open data + code Audio, annotations, per-system transcripts, entity-match decisions, verifier configs, and the scoring CLI are released in openbenchmarks-labs/voice-code-bench · recompute any number end-to-end. github → ](https://github.com/openbenchmarks-labs/voice-code-bench "Open the public data + code mirror on GitHub") recordings 300 systems 17 entity types 26 [Deepgramspeech models](https://deepgram.com "Open Deepgram in a new tab") [ElevenLabsvoice AI platform](https://elevenlabs.io "Open ElevenLabs in a new tab") Google Cloud — [OpenAIspeech models](https://openai.com "Open OpenAI in a new tab") Thinking Machines Lab — [AssemblyAIspeech models](https://www.assemblyai.com "Open AssemblyAI in a new tab") [Inworld AIspeech models](https://inworld.ai "Open Inworld AI in a new tab") NVIDIA — Amazon Web Services — Meta — \[01\] results ## Leaderboard and per-entity-type breakdown Measured TSR, CTEM, and format-invariant WER on identical raw audio. Batch and streaming endpoints are separate rows. No opinions in this section: numbers only. leaderboard All 17 tracked systems ranked by Task Success Rate. Streaming receives realtime-paced audio and cannot revisit finalized segments. Top 8 per metric __Speech-to-text systems ranked by exact structured-value recovery__ | Rank | System | Vendor | Mode | Deployment | TSR | CTEM | WER | | --- | --- | --- | --- | --- | --- | --- | --- | | 1 | [Deepgram Nova-3](https://deepgram.com "Open Deepgram in a new tab") | Deepgram | batch | hosted API | **71.0%** | 91.4% | 8.2% | | 2 | [ElevenLabs Scribe v2](https://elevenlabs.io "Open ElevenLabs in a new tab") | ElevenLabs | batch | hosted API | **68.0%** | 91.8% | 6.8% | | 3 | [Deepgram Nova-3](https://deepgram.com "Open Deepgram in a new tab") | Deepgram | streaming | hosted API | **61.0%** | 88.3% | 9.5% | | 4 | [ElevenLabs Scribe v2 Realtime](https://elevenlabs.io "Open ElevenLabs in a new tab") | ElevenLabs | streaming | hosted API | **59.0%** | 88.1% | 5.1% | | 5 | [Google Cloud Chirp 3](https://cloud.google.com/speech-to-text "Open Google Cloud in a new tab") | Google Cloud | batch | hosted API | **58.0%** | 88.7% | 5.8% | | 6 | [OpenAI GPT Realtime (Whisper)](https://openai.com "Open OpenAI in a new tab") | OpenAI | streaming | hosted API | **57.3%** | 88.7% | 8.6% | | 7 | [Google Cloud Chirp 3](https://cloud.google.com/speech-to-text "Open Google Cloud in a new tab") | Google Cloud | streaming | hosted API | **56.3%** | 87.3% | 6.8% | | 8 | [Inkling (Thinking Machines)](https://thinkingmachines.ai "Open Thinking Machines Lab in a new tab") | Thinking Machines Lab | batch | managed endpoint via Modal | **54.7%** | 85.3% | 5.0% | | 9 | [OpenAI GPT-4o Transcribe](https://openai.com "Open OpenAI in a new tab") | OpenAI | batch | hosted API | **54.0%** | 87.0% | 4.6% | | 10 | [Whisper large-v3](https://openai.com "Open OpenAI in a new tab") | OpenAI | batch | hosted API | **53.3%** | 87.3% | 5.9% | | 11 | [AssemblyAI Universal 3.5 Pro](https://www.assemblyai.com "Open AssemblyAI in a new tab") | AssemblyAI | batch | hosted API | **50.3%** | 86.2% | 3.9% | | 12 | [Inworld STT 1](https://inworld.ai "Open Inworld in a new tab") | Inworld | batch | hosted API | **48.3%** | 85.4% | 8.2% | | 13 | [NVIDIA Parakeet TDT 0.6B v3](https://www.nvidia.com "Open NVIDIA in a new tab") | NVIDIA | batch | self-hosted via Modal | **39.3%** | 79.8% | 11.4% | | 14 | [Amazon Transcribe](https://aws.amazon.com/transcribe/ "Open Amazon Web Services in a new tab") | Amazon Web Services | streaming | hosted API | **35.0%** | 78.8% | 7.5% | | 15 | [Meta OmniASR LLM Unlimited 7B v2](https://ai.meta.com "Open Meta in a new tab") | Meta | batch | self-hosted via Modal | **32.7%** | 76.9% | 10.5% | | 16 | [AssemblyAI Universal 3 Pro](https://www.assemblyai.com "Open AssemblyAI in a new tab") | AssemblyAI | streaming | hosted API | **24.7%** | 71.9% | 7.5% | | 17 | [Inworld STT 1](https://inworld.ai "Open Inworld in a new tab") | Inworld | streaming | hosted API | **7.3%** | 34.0% | 58.2% | benchmarks/voice-code-bench/2026-q3 17 systems · 26 entity types __Per-entity-type CTEM for every benchmarked speech-to-text system__ | System | Email Address | Phone Number | Phone Extension | Person Or Team Name | Postal Address | URL | IP Address | Port Number | Command | CLI Flag | File Path | Environment Variable | Code Symbol | Version | Reference ID | Product Code | Account Or Record Number | Currency Amount | Percentage | Measurement | Plain Number | Date | Time | Acronym Or Initialism | Spelled Sequence | Domain Term | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Deepgram Nova-3 · batch | 77% | 98% | 100% | 100% | 70% | 69% | 96% | 97% | 56% | 91% | 61% | 86% | 86% | 93% | 95% | 96% | 97% | 100% | 100% | 100% | 100% | 100% | 97% | 99% | 98% | 90% | | ElevenLabs Scribe v2 · batch | 69% | 100% | 100% | 100% | 73% | 65% | 96% | 100% | 72% | 93% | 65% | 97% | 97% | 93% | 93% | 91% | 94% | 100% | 100% | 100% | 100% | 99% | 98% | 96% | 98% | 100% | | Deepgram Nova-3 · streaming | 69% | 98% | 100% | 100% | 68% | 63% | 92% | 90% | 60% | 89% | 53% | 80% | 77% | 93% | 93% | 90% | 92% | 96% | 100% | 98% | 98% | 96% | 97% | 95% | 96% | 90% | | ElevenLabs Scribe v2 Realtime · streaming | 71% | 98% | 100% | 100% | 60% | 47% | 96% | 97% | 50% | 84% | 47% | 54% | 97% | 100% | 95% | 92% | 92% | 92% | 100% | 100% | 100% | 100% | 98% | 99% | 98% | 100% | | Google Cloud Chirp 3 · batch | 66% | 98% | 100% | 100% | 57% | 61% | 96% | 97% | 54% | 91% | 59% | 89% | 91% | 90% | 91% | 92% | 94% | 97% | 98% | 100% | 100% | 100% | 93% | 96% | 92% | 90% | | OpenAI GPT Realtime (Whisper) · streaming | 65% | 90% | 93% | 94% | 53% | 58% | 84% | 97% | 74% | 95% | 67% | 91% | 94% | 90% | 93% | 93% | 91% | 89% | 100% | 98% | 98% | 100% | 98% | 98% | 91% | 95% | | Google Cloud Chirp 3 · streaming | 68% | 97% | 100% | 100% | 55% | 55% | 96% | 97% | 54% | 86% | 63% | 83% | 83% | 87% | 89% | 96% | 88% | 93% | 98% | 98% | 98% | 100% | 92% | 94% | 94% | 95% | | Inkling (Thinking Machines) · batch | 57% | 92% | 100% | 89% | 57% | 55% | 84% | 83% | 58% | 80% | 59% | 89% | 83% | 97% | 88% | 83% | 86% | 95% | 98% | 100% | 97% | 100% | 95% | 92% | 93% | 95% | | OpenAI GPT-4o Transcribe · batch | 63% | 100% | 100% | 94% | 53% | 45% | 80% | 97% | 56% | 86% | 57% | 94% | 89% | 93% | 90% | 91% | 88% | 100% | 100% | 100% | 98% | 100% | 98% | 94% | 91% | 85% | | Whisper large-v3 · batch | 78% | 95% | 100% | 94% | 60% | 42% | 92% | 90% | 46% | 89% | 53% | 89% | 91% | 93% | 93% | 89% | 91% | 97% | 100% | 100% | 95% | 97% | 90% | 96% | 97% | 85% | | AssemblyAI Universal 3.5 Pro · batch | 55% | 90% | 97% | 100% | 68% | 47% | 88% | 97% | 46% | 86% | 63% | 83% | 100% | 83% | 91% | 86% | 88% | 92% | 98% | 100% | 97% | 100% | 98% | 93% | 95% | 100% | | Inworld STT 1 · batch | 60% | 90% | 97% | 89% | 55% | 47% | 72% | 90% | 58% | 84% | 67% | 89% | 91% | 90% | 86% | 89% | 91% | 96% | 96% | 95% | 98% | 98% | 93% | 99% | 89% | 90% | | NVIDIA Parakeet TDT 0.6B v3 · batch | 29% | 83% | 100% | 94% | 45% | 47% | 76% | 83% | 38% | 86% | 37% | 83% | 83% | 93% | 81% | 81% | 82% | 85% | 100% | 100% | 98% | 97% | 93% | 94% | 90% | 85% | | Amazon Transcribe · streaming | 28% | 98% | 90% | 83% | 53% | 40% | 88% | 87% | 12% | 61% | 37% | 57% | 77% | 87% | 89% | 86% | 86% | 85% | 98% | 90% | 98% | 99% | 98% | 95% | 87% | 95% | | Meta OmniASR LLM Unlimited 7B v2 · batch | 35% | 73% | 100% | 89% | 48% | 50% | 72% | 87% | 32% | 70% | 24% | 77% | 80% | 73% | 88% | 86% | 85% | 75% | 50% | 90% | 98% | 94% | 92% | 99% | 96% | 80% | | AssemblyAI Universal 3 Pro · streaming | 38% | 78% | 90% | 94% | 50% | 21% | 60% | 97% | 34% | 73% | 31% | 69% | 77% | 80% | 63% | 72% | 63% | 73% | 96% | 97% | 95% | 90% | 92% | 91% | 79% | 95% | | Inworld STT 1 · streaming | 9% | 18% | 17% | 72% | 30% | 27% | 32% | 33% | 18% | 23% | 35% | 29% | 31% | 57% | 55% | 31% | 60% | 12% | 6% | 41% | 40% | 46% | 25% | 29% | 53% | 10% | \[02\] methodology and what is not measured+ \[02.a\] dataset and task ### What each system is asked to do 1. The dataset is 300 human-recorded English WAV segments (5.587 hours, 85 anonymized speakers) across 8 workplace workflow domains: support callbacks, deployments, order handling, and similar. 2. Each item carries three transcript layers: `template` (script with entity placeholders), `acoustic` (what the speaker says aloud), and `canonical`(the written value a downstream application needs). “double dash dry dash run” maps to `--dry-run`; “all caps database underscore URL” maps to `DATABASE_URL`. 3. 1,482 audited target entities span 26 structured types: emails, phone numbers, URLs, IP addresses, ports, shell commands, CLI flags, file paths, environment variables, code symbols, versions, reference IDs, amounts, dates, times, spelled sequences, and more. 4. The system's only input is the audio file. Its transcript is stored verbatim with run metadata (endpoint, evaluation date, inference settings). \[02.b\] metric definitions ### What each metric means * `CTEM` · Canonical Token/Entity Match. Correct target entities ÷ target entities. An entity is correct only when the exact canonical value is recoverable; formatting differences are fine, wrong or missing characters are not. * `TSR` · Task Success Rate. Recordings with _all_ target entities correct ÷ recordings. The strictest headline metric: one corrupted digit fails the whole recording. With 300 clips, standard error on TSR is roughly ±2.6pp, so gaps of a few points at the top are noise, not a stable ranking. * `WER` · format-invariant Word Error Rate. Every target entity span accepts either the documented acoustic rendering or the canonical rendering as reference, so `212-555-0100`and “two one two five five five zero one zero zero” score identically. Non-entity words use ordinary word-level edit distance. * `per-entity CTEM` · CTEM restricted to one of the 26 entity types: the matrix above. \[02.c\] protocol ### Raw audio only with no hints or cleanup Every system is evaluated under a raw-audio-only protocol: it receives the audio file and nothing else. Benchmark-specific prompts, target entity lists, domain labels, custom vocabulary, grammar constraints, candidate values, and post-ASR correction are all excluded from the main setting. This measures what the model does on its own, not what a tuned integration could extract from it. Batch systems receive each recording as a complete file at the provider's documented input format. Streaming systems receive headerless PCM paced in realtime (100 ms chunks) over the provider's websocket protocol, with VAD, turn detection, and vendor canonicalization disabled where the API allows. Batch and streaming endpoints of the same vendor are tracked as separate systems because their error profiles differ substantially. \[02.d\] entity verification ### How entity matches are judged For each recording, every target entity is checked against the system transcript by a versioned LLM verifier: `openai_gpt_5_6_sol_group10_sync_v1` (`gpt-5.6-sol`, 10 datapoints per request, reasoning effort `medium`). The verifier marks an entity present only when enough evidence exists to recover the exact canonical value. The same verifier scores every system; its prompt, schema, and settings are frozen into a config digest that ships with every entity-match artifact. A single-judge setup has its own error floor; that is a known limitation, not hidden. \[02.e\] what is not measured ### Out of scope on this board * Multilingual / non-English accuracy (English-only snapshot). * Diarization, speaker ID, timestamp accuracy, punctuation / formatting quality. * Noise robustness at specific SNRs, telephony 8 kHz, far-field microphones. * Domain-specific vocabularies (medical, legal, aviation, heavy proper nouns) beyond the released workplace domains. * Effect of custom vocabulary / keyword boosting is excluded from the main setting by design. * On-device / mobile deployment, hallucination on silence, HIPAA / residency, long-form behavior beyond the released clips. * Raw Word Error Rate and STT latency (time to final segment): not the job of this board (see the sibling Speech-to-Text benchmark for those shortlists). \[02.f\] reproducibility ### Verify any number end-to-end Everything behind this table is released in [openbenchmarks-labs/voice-code-bench ](https://github.com/openbenchmarks-labs/voice-code-bench): audio, transcripts and entity annotations, per-system transcripts plus entity-match decisions, verifier configs, and the scoring CLI. The dataset is also on Hugging Face as [besimple-ai/voice-code-bench ](https://huggingface.co/datasets/besimple-ai/voice-code-bench) (MIT). The release script rescores published artifacts and rewrites `baselines/results.csv`, the table on this page, without provider credentials. \[03\] faq ## Structured Speech-to-Text Benchmark: common questions Scope, independence, and how this board differs from raw WER leaderboards. For workflow picks (booking, coding, form fill), use the sections above the tables. ### What is the Structured Speech-to-Text Benchmark? The Structured Speech-to-Text Benchmark is an independent speech-to-text (ASR) benchmark that measures whether a transcript preserves exact structured values in English workplace speech: emails, phone numbers, CLI flags, file paths, URLs, IDs, and 20 other entity types. It scores 17 systems on 300 human-recorded segments (1,482 audited entities) using Task Success Rate (TSR), Canonical Token/Entity Match (CTEM), and format-invariant WER. Dataset by Besimple AI; baselines run and scored by Openbenchmarks. ### Does a low Word Error Rate mean the transcript is good enough for software? No. WER treats every word equally, so a system can post excellent WER while corrupting the one email, URL, or CLI flag a downstream system needs. On this board AssemblyAI Universal 3.5 Pro has the best WER (3.9%) but recovers every structured value in only 50.3% of recordings, while Deepgram Nova-3 and ElevenLabs Scribe v2 sit at the top of TSR (71.0% vs 68.0%). That decoupling is why this board exists. ### Where should I go for raw WER or the fastest STT model? For raw transcript quality (Word Error Rate) and speech-to-text latency, including unqualified “best STT model” shortlists, use the [Speech-to-Text benchmark](https://openbenchmarks.com/speech-to-text-benchmark-by-coval) . Use this Structured Speech-to-Text Benchmark when you need the best STT for downstream tasks or software that depends on exact structured values. ### Who created the dataset, and is the benchmark independent? The recordings and entity annotations are authored and released by Besimple AI as besimple-ai/voice-code-bench on Hugging Face (MIT). Openbenchmarks independently runs the 17 tracked systems, scores them with a versioned LLM verifier held constant across systems, and publishes every artifact. No vendor pays for inclusion, ranking, or removal. ### What does the Structured Speech-to-Text Benchmark not measure? Multilingual accuracy, diarization, timestamps, punctuation quality, noise/telephony robustness, domain vocabularies, custom vocabulary boosts, on-device deployment, and long-form behavior beyond the released clips. For your domain, run your own production clips. For raw Word Error Rate and STT latency, go to the [Speech-to-Text benchmark](https://openbenchmarks.com/speech-to-text-benchmark-by-coval) . Openbenchmarks open benchmarks [← all benchmarks](https://openbenchmarks.com/) [source ↗](https://github.com/openbenchmarks-labs/voice-code-bench "Public data + code mirror on GitHub") [dataset ↗](https://huggingface.co/datasets/besimple-ai/voice-code-bench "Source dataset by Besimple AI on Hugging Face") © 2026 Openbenchmarks