Best inference provider for chat applications
What the benchmark found. Nebius led median token generation speed at 269.3 tokens/second. Its slow-tail P99 speed was 39.9 tokens/second, with TTFA P95 of 7.11 seconds. The measured output is short and structured, so this is evidence about provider streaming behavior rather than a long-conversation benchmark.
What this page compares. A chat application exposes the generation path directly: users notice both the blank wait before text appears and the pace at which the answer continues. This page focuses on visible-answer token generation speed across the pooled benchmark rather than presenting the main board's complete metric matrix.
How to read the result. The table is ordered by median post-first-token generation speed. Read generation P95 and P99 as slow-speed floors, then check TTFA P95 because a fast stream can still begin late. This benchmark uses short JSON answers, so validate the result again on your expected conversational response lengths.
Workload and configuration. Every provider received the same 600 deliberately easy structured tasks: 200 meeting-note lookups, 200 ticket-triage classifications, and 200 contract-term extractions. GLM 5.3 Flash ran with streaming enabled, reasoning low, temperature 0, top-p 1, a 256-token output ceiling, one attempt, a 20-second timeout, and per-provider concurrency one.
APIs used. The runner called each provider's OpenAI-compatible chat-completions surface. Nebius, Baseten, Modal, Telnyx, Together AI, DeepInfra, Novita AI, Fireworks AI, Z.AI, and Parasail used the model identifiers below; the links open the provider documentation used to verify each adapter.
- Nebius:
zai-org/GLM-5.3-Flash· API docs ↗ - Baseten:
zai-org/GLM-5.3-Flash· API docs ↗ - Modal:
zai-org/GLM-5.3-Flash· API docs ↗ - Telnyx:
zai-org/GLM-5.3-Flash· API docs ↗ - Together AI:
zai-org/GLM-5.3-Flash· API docs ↗ - DeepInfra:
zai-org/GLM-5.3-Flash· API docs ↗ - Novita AI:
zai-org/glm-5.3-flash· API docs ↗ - Fireworks AI:
accounts/fireworks/models/glm-5p3-flash· API docs ↗ - Z.AI:
glm-5.3-flash· API docs ↗ - Parasail:
zai-org/GLM-5.3-Flash· API docs ↗
Inference providers sorted by chat-generation speed
Higher token generation speed is better. The delta shows the median speed gap from the fastest stream, while TTFA and E2E keep response startup and completion visible as separate waits.
| Provider / API | Generation P50 | Δ vs fastest P50 | Generation P95 | Generation P99 | TTFA P95 | E2E P99 |
|---|---|---|---|---|---|---|
| Nebiuszai-org/GLM-5.3-Flash ↗ | 269.3 tok/s | 0.0 tok/s | 78.8 tok/s | 39.9 tok/s | 7.11 s | 15.0 s |
| Basetenzai-org/GLM-5.3-Flash ↗ | 240.5 tok/s | -28.8 tok/s | 61.6 tok/s | 22.6 tok/s | 1.16 s | 3.23 s |
| Modalzai-org/GLM-5.3-Flash ↗ | 222.2 tok/s | -47.1 tok/s | 84.7 tok/s | 34.9 tok/s | 1.09 s | 3.28 s |
| Telnyxzai-org/GLM-5.3-Flash ↗ | 184.3 tok/s | -85.0 tok/s | 67.5 tok/s | 47.3 tok/s | 1.34 s | 3.48 s |
| Together AIzai-org/GLM-5.3-Flash ↗ | 151.6 tok/s | -117.6 tok/s | 34.0 tok/s | 21.7 tok/s | 2.26 s | 12.4 s |
| DeepInfrazai-org/GLM-5.3-Flash ↗ | 69.4 tok/s | -199.8 tok/s | 9.0 tok/s | 5.0 tok/s | 6.41 s | 17.1 s |
| Novita AIzai-org/glm-5.3-flash ↗ | 60.6 tok/s | -208.6 tok/s | 29.6 tok/s | 11.6 tok/s | 3.41 s | 7.25 s |
| Fireworks AIaccounts/fireworks/models/glm-5p3-flash ↗ | 58.2 tok/s | -211.0 tok/s | 27.9 tok/s | 14.9 tok/s | 5.83 s | 10.5 s |
| Z.AIglm-5.3-flash ↗ | 56.3 tok/s | -213.0 tok/s | 27.8 tok/s | 11.5 tok/s | 3.14 s | 8.59 s |
| Parasailzai-org/GLM-5.3-Flash ↗ | 52.4 tok/s | -216.9 tok/s | 22.8 tok/s | 15.6 tok/s | 5.54 s | 11.5 s |
Chat experiences this streaming view can inform
- Embedded product chat. Use TTFA P95 for the initial blank state and generation P50 for the reading pace after the first visible token.
- Operator-assist chat. Use generation P95 when an agent must scan a streamed suggestion under time pressure and slow continuation is noticeable.
- Structured conversational actions. Keep E2E and task success beside throughput when the chat response becomes JSON consumed by application code.
Chat inference speed — application-specific questions
Which measured provider streamed chat output fastest after the first token?
Nebius led median token generation speed at 269.3 tokens/second. Its slow-tail P99 speed was 39.9 tokens/second, with TTFA P95 of 7.11 seconds. The measured output is short and structured, so this is evidence about provider streaming behavior rather than a long-conversation benchmark.
How should a chat application interpret generation P95 and P99?
Generation P95 is the speed at least 95% of measured responses achieved; P99 raises that coverage to 99%. These are coverage-based slow-tail floors, and higher throughput is better.
Why can the fastest token generator still feel slow in a chat UI?
Nebius's median generation result was 269.3 tokens/second, but response experience also includes its 7.11 seconds TTFA P95. A provider can generate quickly after a slow start.
Read the complete inference benchmark
Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the Inference Benchmark →