Inference Benchmark

600 paired rounds

1. 200 meeting-note lookups, 200 ticket-triage tasks, and 200 contract-term extractions frozen before execution. Selection is a deterministic shuffle of stable item IDs tied to the dataset-release hash, so providers cannot receive different samples.
2. Cycle through the three task families one item at a time. For each question, prepare all ten provider requests before releasing them through one shared asynchronous barrier.
3. Rotate provider launch order each round and record each request's monotonic start time.
4. Wait until all ten calls complete, time out, or fail before starting the next question. A fast provider remains idle while the slowest finishes, preserving per-provider concurrency of exactly one.

Measuring the wait the user actually experiences

• Every duration uses a monotonic clock beginning immediately before the request enters the HTTP transport. Connection-pool waiting, network transit, provider queueing, model prefill, and generation are included because the user waits through all of them.
• Time To First Output token (TTFO) ends at the first token-bearing reasoning or visible-answer event. It answers when the model first produces any output, even if that output is reasoning.
• Time To First Visible Answer token (TTFA) ends when accumulated visible answer content reaches its first local-tokenizer token. Reasoning does not stop this clock, making TTFA the better measure when users cannot see reasoning.
• TTFO vs TTFA: Even if reasoning effort is set to low, the model may or may not produce reasoning tokens. So time to first visible answer token varies based on reasoning presence. The distinction is necessary because TTFO measures when the first output token arrived (either reasoning or visible answer), making it a fairer measure of latency across providers.
• E2E latency ends at the final non-empty visible answer delta.

Separate startup delay from answer throughput

• For a complete visible answer containing N local tokens, generation speed is (N − 1) ÷ (last answer-token time − first answer-token time). Removing the first token prevents startup latency from being counted twice.
• Only visible answer tokens count. Reasoning tokens are excluded, as are complete answers with fewer than two tokens or without a positive first-to-last generation window; those requests remain visible in metric coverage.
• P50 describes typical answer throughput. The slow tail is reported as P95 and P99: 95% and 99% of measured responses respectively generated at least that fast.

A fast answer still has to be correct and arrive

• Meeting-note lookup compares the returned answer with the canonical answer and allowed aliases after trimming, collapsing repeated whitespace, and normalizing case.
• Ticket triage requires the exact lowercase category-and-priority pair. Contract extraction requires every schema field, exact typed values, and ISO YYYY-MM-DD dates. Neither task receives partial credit in task success.
• Task success rate is exact-correct, schema-valid answers divided by all 600 submitted requests per provider. A timeout, HTTP failure, malformed JSON, incomplete object, and confidently wrong answer all reduce this rate.
• Failure rate isolates operational failures: timeout, HTTP 429, other HTTP 4xx, HTTP 5xx, and transport error. The categories are mutually exclusive; a malformed or incorrect HTTP-successful answer lowers task success but is not relabeled as a transport failure.
• Latency and generation-speed percentiles use only requests that produced the required visible-answer timing sample. Failed requests are not accounted for in latency calculations, which is why task success and failure rate must be read beside the speed columns.