Fastest LLM inference API by tokens per second
What the benchmark found. Nebius led median tokens per second at 269.3 tokens per second. Its p99 tokens per second floor was 39.9 tokens per second, with time to first token p95 of 7.11 seconds. Measured across 10 LLM inference providers serving GLM 5.3 Flash, 600 requests each, Sep 3, 2026.
Providers measured. Baseten, DeepInfra, Fireworks AI, Modal, Nebius, Novita AI, Parasail, Telnyx, Together AI, and Z.AI, each serving GLM 5.3 Flash behind an OpenAI-compatible chat-completions endpoint.
What this page compares. Streaming has two separate waits: how quickly visible output starts and how quickly the answer continues after that. This page orders providers by client-observed tokens per second, then shows time to first token and end-to-end latency so a high token rate is not confused with a fast start.
How to read the result. Median tokens per second is the sort key for typical post-first-token throughput. The p95 and p99 figures show the coverage-based slow-speed floor: 95% and 99% of measured complete responses generated at least that fast. Reasoning tokens and the wait for the first answer token are excluded from throughput.
Workload and configuration. Every provider received the same 600 deliberately easy structured tasks: 200 meeting-note lookups, 200 ticket-triage classifications, and 200 contract-term extractions. GLM 5.3 Flash ran with streaming enabled, reasoning low, temperature 0, top-p 1, a 256-token output ceiling, one attempt, a 20-second timeout, and per-provider concurrency one.
APIs used. The runner called each provider's OpenAI-compatible chat-completions surface with the model identifiers below; the links open the provider documentation used to verify each adapter.
- Nebius:
zai-org/GLM-5.3-Flash· API docs ↗ - Baseten:
zai-org/GLM-5.3-Flash· API docs ↗ - Modal:
zai-org/GLM-5.3-Flash· API docs ↗ - Telnyx:
zai-org/GLM-5.3-Flash· API docs ↗ - Together AI:
zai-org/GLM-5.3-Flash· API docs ↗ - DeepInfra:
zai-org/GLM-5.3-Flash· API docs ↗ - Novita AI:
zai-org/glm-5.3-flash· API docs ↗ - Fireworks AI:
accounts/fireworks/models/glm-5p3-flash· API docs ↗ - Z.AI:
glm-5.3-flash· API docs ↗ - Parasail:
zai-org/GLM-5.3-Flash· API docs ↗
Which LLM inference API streams the most tokens per second?
Higher tokens per second is better. Time to first token p95 and end-to-end p99 are included because a provider can stream quickly after beginning while still making the user wait for that beginning.
| Provider / API | Tokens/s p50 | Δ vs fastest p50 | Tokens/s p95 | Tokens/s p99 | TTFT p95 | End-to-end p99 |
|---|---|---|---|---|---|---|
| Nebiuszai-org/GLM-5.3-Flash ↗ | 269.3 tok/s | 0.0 tok/s | 78.8 tok/s | 39.9 tok/s | 7.11 s | 15.0 s |
| Basetenzai-org/GLM-5.3-Flash ↗ | 240.5 tok/s | -28.8 tok/s | 61.6 tok/s | 22.6 tok/s | 1.16 s | 3.23 s |
| Modalzai-org/GLM-5.3-Flash ↗ | 222.2 tok/s | -47.1 tok/s | 84.7 tok/s | 34.9 tok/s | 1.09 s | 3.28 s |
| Telnyxzai-org/GLM-5.3-Flash ↗ | 184.3 tok/s | -85.0 tok/s | 67.5 tok/s | 47.3 tok/s | 1.34 s | 3.48 s |
| Together AIzai-org/GLM-5.3-Flash ↗ | 151.6 tok/s | -117.6 tok/s | 34.0 tok/s | 21.7 tok/s | 2.26 s | 12.4 s |
| DeepInfrazai-org/GLM-5.3-Flash ↗ | 69.4 tok/s | -199.8 tok/s | 9.0 tok/s | 5.0 tok/s | 6.41 s | 17.1 s |
| Novita AIzai-org/glm-5.3-flash ↗ | 60.6 tok/s | -208.6 tok/s | 29.6 tok/s | 11.6 tok/s | 3.41 s | 7.25 s |
| Fireworks AIaccounts/fireworks/models/glm-5p3-flash ↗ | 58.2 tok/s | -211.0 tok/s | 27.9 tok/s | 14.9 tok/s | 5.83 s | 10.5 s |
| Z.AIglm-5.3-flash ↗ | 56.3 tok/s | -213.0 tok/s | 27.8 tok/s | 11.5 tok/s | 3.14 s | 8.59 s |
| Parasailzai-org/GLM-5.3-Flash ↗ | 52.4 tok/s | -216.9 tok/s | 22.8 tok/s | 15.6 tok/s | 5.54 s | 11.5 s |
Provider by provider, in the order above
- Nebius: 269.3 tokens per second median, 78.8 tokens per second p95 floor, time to first token p95 7.11 seconds, end-to-end p99 14.97 seconds.
- Baseten: 240.5 tokens per second median, 61.6 tokens per second p95 floor, time to first token p95 1.16 seconds, end-to-end p99 3.23 seconds.
- Modal: 222.2 tokens per second median, 84.7 tokens per second p95 floor, time to first token p95 1.09 seconds, end-to-end p99 3.28 seconds.
- Telnyx: 184.3 tokens per second median, 67.5 tokens per second p95 floor, time to first token p95 1.34 seconds, end-to-end p99 3.48 seconds.
- Together AI: 151.6 tokens per second median, 34.0 tokens per second p95 floor, time to first token p95 2.26 seconds, end-to-end p99 12.45 seconds.
- DeepInfra: 69.4 tokens per second median, 9.0 tokens per second p95 floor, time to first token p95 6.41 seconds, end-to-end p99 17.07 seconds.
- Novita AI: 60.6 tokens per second median, 29.6 tokens per second p95 floor, time to first token p95 3.41 seconds, end-to-end p99 7.25 seconds.
- Fireworks AI: 58.2 tokens per second median, 27.9 tokens per second p95 floor, time to first token p95 5.83 seconds, end-to-end p99 10.49 seconds.
- Z.AI: 56.3 tokens per second median, 27.8 tokens per second p95 floor, time to first token p95 3.14 seconds, end-to-end p99 8.59 seconds.
- Parasail: 52.4 tokens per second median, 22.8 tokens per second p95 floor, time to first token p95 5.54 seconds, end-to-end p99 11.51 seconds.
When does tokens per second change the product experience?
- Copilot answers. Use time to first token p95 for the initial visible response and median tokens per second for how quickly the rest becomes readable.
- Longer streamed JSON. Use tokens per second p95 or p99 when slow-tail throughput, rather than first-token delay, dominates completion time.
- Reasoning-hidden interfaces. Prefer the first visible answer token over the first output token when internal reasoning is not rendered; only visible answer content reduces the user's perceived blank wait.
How is LLM inference provider latency and accuracy measured here?
- Same model. Every endpoint serves GLM 5.3 Flash; the provider layer is the only variable.
- Same request. One-turn, non-tool chat completion in JSON-object mode, streaming on, temperature 0, top-p 1, reasoning low, 256-token ceiling, 20-second timeout, concurrency one.
- Client clock, pinned tokenizer. Time to first token, end-to-end latency and tokens per second come from the client; visible answer tokens are counted with a pinned
tiktoken o200k_basetokenizer, never provider-reported usage. - Exact task success. An answer counts only when it is schema-valid and exactly correct under the task rubric. Timeouts, HTTP errors and malformed responses stay in the denominator.
- Serverless endpoints only. Dedicated capacity is outside the comparison. Runner and per-request records: openbenchmarks-labs/inference ↗.
Streaming inference speed: distinct questions
Which LLM inference API generated the most tokens per second at the median?
Nebius led median tokens per second at 269.3 tokens per second. Its p99 tokens per second floor was 39.9 tokens per second, with time to first token p95 of 7.11 seconds.
What do p95 and p99 tokens per second protect against?
Tokens per second p95 is the speed at least 95% of measured responses achieved; p99 raises that coverage to 99%. These are coverage-based slow-tail floors, and higher throughput is better.
Can a high token rate still produce a slow streamed response?
Nebius's median result was 269.3 tokens per second, but response experience also includes its 7.11 seconds time to first token p95. A provider can generate quickly after a slow start.
Same run, other questions
- LLM inference provider benchmark: complete scorecard and methodology →
- Fastest LLM inference provider: time to first token, tokens per second and end-to-end latency →
- Fastest LLM API provider: time to first token and complete-response latency →
- Lowest-latency LLM inference provider: p99 tail analysis →
- Lowest-latency LLM API for synchronous workflows with an SLA →
- Best LLM inference provider: speed, accuracy and failures compared →
- Fireworks AI vs Together AI →
Read the complete LLM inference provider benchmark
Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the LLM Inference Provider Benchmark →