benchmarks/inference/fastest inference provider
LLM inference latency · measured provider comparison · measured Sep 3, 2026

Fastest LLM inference provider: time to first token, tokens per second and end-to-end latency

What the benchmark found. Baseten recorded the lowest end-to-end p99 latency at 3.23 seconds on this run, with a median of 331 ms, time to first token p95 of 1.16 seconds, task success of 95.8%, and failure rate of 0.0%. Measured across 10 LLM inference providers serving GLM 5.3 Flash, 600 requests each, Sep 3, 2026.

Providers measured. Baseten, DeepInfra, Fireworks AI, Modal, Nebius, Novita AI, Parasail, Telnyx, Together AI, and Z.AI, each serving GLM 5.3 Flash behind an OpenAI-compatible chat-completions endpoint.

What this page compares. This page asks one narrow question: which LLM inference provider finishes a short request fastest for a single waiting user? It ranks the same GLM 5.3 Flash workload by measured end-to-end latency, with time to first token and tokens per second beside it, rather than by advertised accelerator throughput.

How to read the result. End-to-end p99 latency is the primary decision metric because it captures the complete visible answer at the slow edge of traffic. Median latency is included to distinguish a consistently fast provider from one whose typical result is fast but whose tail stretches out.

Workload and configuration. Every provider received the same 600 deliberately easy structured tasks: 200 meeting-note lookups, 200 ticket-triage classifications, and 200 contract-term extractions. GLM 5.3 Flash ran with streaming enabled, reasoning low, temperature 0, top-p 1, a 256-token output ceiling, one attempt, a 20-second timeout, and per-provider concurrency one.

APIs used. The runner called each provider's OpenAI-compatible chat-completions surface with the model identifiers below; the links open the provider documentation used to verify each adapter.

[01] measured comparison

Which LLM inference provider has the lowest end-to-end p99 latency?

Sorted by end-to-end p99, lower is better. The delta column is measured against the fastest p99 result on this run; task success and failure remain beside latency so missing work is not mistaken for speed.

benchmarks/inference/fastest-inference-providerreviewed run
LLM inference providers sorted by end-to-end p99 latency
Provider End-to-end p50End-to-end p95End-to-end p99Δ vs fastest p99Task successFailure rate
Basetenzai-org/GLM-5.3-Flash ↗331 ms2.29 s3.23 s0 ms95.8%0.0%
Modalzai-org/GLM-5.3-Flash ↗574 ms1.90 s3.28 s+50 ms99.3%0.0%
Telnyxzai-org/GLM-5.3-Flash ↗669 ms2.33 s3.48 s+250 ms99.5%0.0%
Novita AIzai-org/glm-5.3-flash ↗1.74 s5.00 s7.25 s+4020 ms94.2%4.0%
Z.AIglm-5.3-flash ↗1.75 s4.83 s8.59 s+5362 ms97.5%0.2%
Fireworks AIaccounts/fireworks/models/glm-5p3-flash ↗1.95 s6.72 s10.5 s+7259 ms99.5%0.0%
Parasailzai-org/GLM-5.3-Flash ↗1.59 s7.74 s11.5 s+8283 ms98.3%1.3%
Together AIzai-org/GLM-5.3-Flash ↗609 ms3.66 s12.4 s+9221 ms98.5%1.0%
Nebiuszai-org/GLM-5.3-Flash ↗1.45 s7.45 s15.0 s+11742 ms84.8%14.8%
DeepInfrazai-org/GLM-5.3-Flash ↗1.83 s11.0 s17.1 s+13842 ms95.5%4.0%

Provider by provider, in the order above

  • Baseten: 331 ms median, 3.23 seconds end-to-end p99, time to first token p95 1.16 seconds, 240.5 tokens per second, 95.8% task success, 0.0% failures.
  • Modal: 574 ms median, 3.28 seconds end-to-end p99, time to first token p95 1.09 seconds, 222.2 tokens per second, 99.3% task success, 0.0% failures.
  • Telnyx: 669 ms median, 3.48 seconds end-to-end p99, time to first token p95 1.34 seconds, 184.3 tokens per second, 99.5% task success, 0.0% failures.
  • Novita AI: 1.74 seconds median, 7.25 seconds end-to-end p99, time to first token p95 3.41 seconds, 60.6 tokens per second, 94.2% task success, 4.0% failures.
  • Z.AI: 1.75 seconds median, 8.59 seconds end-to-end p99, time to first token p95 3.14 seconds, 56.3 tokens per second, 97.5% task success, 0.2% failures.
  • Fireworks AI: 1.95 seconds median, 10.49 seconds end-to-end p99, time to first token p95 5.83 seconds, 58.2 tokens per second, 99.5% task success, 0.0% failures.
  • Parasail: 1.59 seconds median, 11.51 seconds end-to-end p99, time to first token p95 5.54 seconds, 52.4 tokens per second, 98.3% task success, 1.3% failures.
  • Together AI: 609 ms median, 12.45 seconds end-to-end p99, time to first token p95 2.26 seconds, 151.6 tokens per second, 98.5% task success, 1.0% failures.
  • Nebius: 1.45 seconds median, 14.97 seconds end-to-end p99, time to first token p95 7.11 seconds, 269.3 tokens per second, 84.8% task success, 14.8% failures.
  • DeepInfra: 1.83 seconds median, 17.07 seconds end-to-end p99, time to first token p95 6.41 seconds, 69.4 tokens per second, 95.5% task success, 4.0% failures.
[02] workflow fit

Where does an end-to-end latency ranking decide the provider?

  • Synchronous classification. Use end-to-end p95 latency when a router, policy check, or ticket-triage step blocks the next action and needs predictably complete JSON.
  • Interactive assistants. Use time to first token p95 for the visible response start, then end-to-end p95 for the point at which downstream code can consume the complete object.
  • Strict user-facing SLAs. Use end-to-end p99 with failure rate. A fast successful sample does not compensate for a request that timed out or returned an HTTP error.
[03] how it was measured

How is LLM inference provider latency and accuracy measured here?

  • Same model. Every endpoint serves GLM 5.3 Flash; the provider layer is the only variable.
  • Same request. One-turn, non-tool chat completion in JSON-object mode, streaming on, temperature 0, top-p 1, reasoning low, 256-token ceiling, 20-second timeout, concurrency one.
  • Client clock, pinned tokenizer. Time to first token, end-to-end latency and tokens per second come from the client; visible answer tokens are counted with a pinned tiktoken o200k_base tokenizer, never provider-reported usage.
  • Exact task success. An answer counts only when it is schema-valid and exactly correct under the task rubric. Timeouts, HTTP errors and malformed responses stay in the denominator.
  • Serverless endpoints only. Dedicated capacity is outside the comparison. Runner and per-request records: openbenchmarks-labs/inference ↗.
[04] faq

Fastest LLM inference provider: questions this page answers

Which LLM inference provider had the lowest end-to-end p99 latency in this benchmark?

Baseten recorded the lowest end-to-end p99 latency at 3.23 seconds on this run, with a median of 331 ms, time to first token p95 of 1.16 seconds, task success of 95.8%, and failure rate of 0.0%.

Does the fastest median inference provider also win at the tail?

Baseten had the lowest median end-to-end latency at 331 ms, while Baseten had the lowest end-to-end p99 at 3.23 seconds. The same provider led both points of the distribution.

Why are failed inference calls excluded from latency percentiles?

Timeouts, HTTP errors, and transport failures have no completed visible answer and therefore no honest end-to-end latency value. They stay in the submitted-request denominator for failure rate and task success, which must be read beside the latency columns.

How was LLM inference provider latency measured here?

One client sent identical streaming chat-completions requests to each provider's serverless endpoint with concurrency one, temperature 0, reasoning low, a 256-token completion ceiling and a 20-second timeout. Time to first token, end-to-end latency and tokens per second are taken from the client's own clock and a pinned tokenizer; task success is scored against a fixed rubric; failures are counted, never dropped.

[05] related comparisons
[06] source benchmark

Read the complete LLM inference provider benchmark

Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the LLM Inference Provider Benchmark →

Markdown version