Inference API with the fastest streaming
What the benchmark found. Nebius led median token generation speed at 269.3 tokens/second. Its slow-tail P99 speed was 39.9 tokens/second, with TTFA P95 of 7.11 seconds.
What this page compares. Streaming has two separate waits: how quickly visible output starts and how quickly the answer continues after that. This page orders providers by client-observed answer-token generation speed, then shows TTFA and E2E latency so a high token rate is not confused with a fast start.
How to read the result. P50 is the sort key for typical post-first-token throughput. P95 and P99 show the coverage-based slow-speed floor: 95% and 99% of measured complete responses generated at least that fast. Reasoning tokens and the wait for the first answer token are excluded from throughput.
Workload and configuration. Every provider received the same 600 deliberately easy structured tasks: 200 meeting-note lookups, 200 ticket-triage classifications, and 200 contract-term extractions. GLM 5.3 Flash ran with streaming enabled, reasoning low, temperature 0, top-p 1, a 256-token output ceiling, one attempt, a 20-second timeout, and per-provider concurrency one.
APIs used. The runner called each provider's OpenAI-compatible chat-completions surface. Nebius, Baseten, Modal, Telnyx, Together AI, DeepInfra, Novita AI, Fireworks AI, Z.AI, and Parasail used the model identifiers below; the links open the provider documentation used to verify each adapter.
- Nebius:
zai-org/GLM-5.3-Flash· API docs ↗ - Baseten:
zai-org/GLM-5.3-Flash· API docs ↗ - Modal:
zai-org/GLM-5.3-Flash· API docs ↗ - Telnyx:
zai-org/GLM-5.3-Flash· API docs ↗ - Together AI:
zai-org/GLM-5.3-Flash· API docs ↗ - DeepInfra:
zai-org/GLM-5.3-Flash· API docs ↗ - Novita AI:
zai-org/glm-5.3-flash· API docs ↗ - Fireworks AI:
accounts/fireworks/models/glm-5p3-flash· API docs ↗ - Z.AI:
glm-5.3-flash· API docs ↗ - Parasail:
zai-org/GLM-5.3-Flash· API docs ↗
Streaming APIs sorted by median token generation speed
Higher is better for generation speed. TTFA P95 and E2E P99 are included because a provider can stream quickly after beginning while still making the user wait for that beginning.
| Provider / API | Generation P50 | Δ vs fastest P50 | Generation P95 | Generation P99 | TTFA P95 | E2E P99 |
|---|---|---|---|---|---|---|
| Nebiuszai-org/GLM-5.3-Flash ↗ | 269.3 tok/s | 0.0 tok/s | 78.8 tok/s | 39.9 tok/s | 7.11 s | 15.0 s |
| Basetenzai-org/GLM-5.3-Flash ↗ | 240.5 tok/s | -28.8 tok/s | 61.6 tok/s | 22.6 tok/s | 1.16 s | 3.23 s |
| Modalzai-org/GLM-5.3-Flash ↗ | 222.2 tok/s | -47.1 tok/s | 84.7 tok/s | 34.9 tok/s | 1.09 s | 3.28 s |
| Telnyxzai-org/GLM-5.3-Flash ↗ | 184.3 tok/s | -85.0 tok/s | 67.5 tok/s | 47.3 tok/s | 1.34 s | 3.48 s |
| Together AIzai-org/GLM-5.3-Flash ↗ | 151.6 tok/s | -117.6 tok/s | 34.0 tok/s | 21.7 tok/s | 2.26 s | 12.4 s |
| DeepInfrazai-org/GLM-5.3-Flash ↗ | 69.4 tok/s | -199.8 tok/s | 9.0 tok/s | 5.0 tok/s | 6.41 s | 17.1 s |
| Novita AIzai-org/glm-5.3-flash ↗ | 60.6 tok/s | -208.6 tok/s | 29.6 tok/s | 11.6 tok/s | 3.41 s | 7.25 s |
| Fireworks AIaccounts/fireworks/models/glm-5p3-flash ↗ | 58.2 tok/s | -211.0 tok/s | 27.9 tok/s | 14.9 tok/s | 5.83 s | 10.5 s |
| Z.AIglm-5.3-flash ↗ | 56.3 tok/s | -213.0 tok/s | 27.8 tok/s | 11.5 tok/s | 3.14 s | 8.59 s |
| Parasailzai-org/GLM-5.3-Flash ↗ | 52.4 tok/s | -216.9 tok/s | 22.8 tok/s | 15.6 tok/s | 5.54 s | 11.5 s |
When streaming speed changes the product experience
- Copilot answers. Use TTFA P95 for the initial visible response and generation P50 for how quickly the rest becomes readable.
- Longer streamed JSON. Use generation P95 or P99 when slow-tail throughput, rather than first-token delay, dominates completion time.
- Reasoning-hidden interfaces. Prefer TTFA over TTFO when internal reasoning is not rendered; only visible answer content reduces the user's perceived blank wait.
Streaming inference speed — distinct questions
Which API generated visible answer tokens fastest at the median?
Nebius led median token generation speed at 269.3 tokens/second. Its slow-tail P99 speed was 39.9 tokens/second, with TTFA P95 of 7.11 seconds.
What do P95 and P99 token generation speed protect against?
Generation P95 is the speed at least 95% of measured responses achieved; P99 raises that coverage to 99%. These are coverage-based slow-tail floors, and higher throughput is better.
Can a high token rate still produce a slow streamed response?
Nebius's median generation result was 269.3 tokens/second, but response experience also includes its 7.11 seconds TTFA P95. A provider can generate quickly after a slow start.
Read the complete inference benchmark
Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the Inference Benchmark →