Best LLM inference provider for chat applications
What the benchmark found. Nebius led median tokens per second at 269.3 tokens per second. Its p99 tokens per second floor was 39.9 tokens per second, with time to first token p95 of 7.11 seconds. The measured output is short and structured, so this is evidence about provider streaming behavior rather than a long-conversation benchmark. Measured across 10 LLM inference providers serving GLM 5.3 Flash, 600 requests each, Sep 3, 2026.
Providers measured. Baseten, DeepInfra, Fireworks AI, Modal, Nebius, Novita AI, Parasail, Telnyx, Together AI, and Z.AI, each serving GLM 5.3 Flash behind an OpenAI-compatible chat-completions endpoint.
What this page compares. A chat application exposes the generation path directly: users notice both the blank wait before text appears and the pace at which the answer continues. This page focuses on visible-answer tokens per second across the pooled benchmark rather than presenting the main board's complete metric matrix.
How to read the result. The table is ordered by median post-first-token tokens per second. Read the p95 and p99 figures as slow-speed floors, then check time to first token p95 because a fast stream can still begin late. This benchmark uses short JSON answers, so validate the result again on your expected conversational response lengths.
Workload and configuration. Every provider received the same 600 deliberately easy structured tasks: 200 meeting-note lookups, 200 ticket-triage classifications, and 200 contract-term extractions. GLM 5.3 Flash ran with streaming enabled, reasoning low, temperature 0, top-p 1, a 256-token output ceiling, one attempt, a 20-second timeout, and per-provider concurrency one.
APIs used. The runner called each provider's OpenAI-compatible chat-completions surface with the model identifiers below; the links open the provider documentation used to verify each adapter.
- Nebius:
zai-org/GLM-5.3-Flash· API docs ↗ - Baseten:
zai-org/GLM-5.3-Flash· API docs ↗ - Modal:
zai-org/GLM-5.3-Flash· API docs ↗ - Telnyx:
zai-org/GLM-5.3-Flash· API docs ↗ - Together AI:
zai-org/GLM-5.3-Flash· API docs ↗ - DeepInfra:
zai-org/GLM-5.3-Flash· API docs ↗ - Novita AI:
zai-org/glm-5.3-flash· API docs ↗ - Fireworks AI:
accounts/fireworks/models/glm-5p3-flash· API docs ↗ - Z.AI:
glm-5.3-flash· API docs ↗ - Parasail:
zai-org/GLM-5.3-Flash· API docs ↗
Which LLM inference provider streams chat output fastest?
Higher tokens per second is better. The delta shows the median speed gap from the fastest stream, while time to first token and end-to-end latency keep response startup and completion visible as separate waits.
| Provider / API | Tokens/s p50 | Δ vs fastest p50 | Tokens/s p95 | Tokens/s p99 | TTFT p95 | End-to-end p99 |
|---|---|---|---|---|---|---|
| Nebiuszai-org/GLM-5.3-Flash ↗ | 269.3 tok/s | 0.0 tok/s | 78.8 tok/s | 39.9 tok/s | 7.11 s | 15.0 s |
| Basetenzai-org/GLM-5.3-Flash ↗ | 240.5 tok/s | -28.8 tok/s | 61.6 tok/s | 22.6 tok/s | 1.16 s | 3.23 s |
| Modalzai-org/GLM-5.3-Flash ↗ | 222.2 tok/s | -47.1 tok/s | 84.7 tok/s | 34.9 tok/s | 1.09 s | 3.28 s |
| Telnyxzai-org/GLM-5.3-Flash ↗ | 184.3 tok/s | -85.0 tok/s | 67.5 tok/s | 47.3 tok/s | 1.34 s | 3.48 s |
| Together AIzai-org/GLM-5.3-Flash ↗ | 151.6 tok/s | -117.6 tok/s | 34.0 tok/s | 21.7 tok/s | 2.26 s | 12.4 s |
| DeepInfrazai-org/GLM-5.3-Flash ↗ | 69.4 tok/s | -199.8 tok/s | 9.0 tok/s | 5.0 tok/s | 6.41 s | 17.1 s |
| Novita AIzai-org/glm-5.3-flash ↗ | 60.6 tok/s | -208.6 tok/s | 29.6 tok/s | 11.6 tok/s | 3.41 s | 7.25 s |
| Fireworks AIaccounts/fireworks/models/glm-5p3-flash ↗ | 58.2 tok/s | -211.0 tok/s | 27.9 tok/s | 14.9 tok/s | 5.83 s | 10.5 s |
| Z.AIglm-5.3-flash ↗ | 56.3 tok/s | -213.0 tok/s | 27.8 tok/s | 11.5 tok/s | 3.14 s | 8.59 s |
| Parasailzai-org/GLM-5.3-Flash ↗ | 52.4 tok/s | -216.9 tok/s | 22.8 tok/s | 15.6 tok/s | 5.54 s | 11.5 s |
Provider by provider, in the order above
- Nebius: 269.3 tokens per second median, 78.8 tokens per second p95 floor, time to first token p95 7.11 seconds, end-to-end p99 14.97 seconds.
- Baseten: 240.5 tokens per second median, 61.6 tokens per second p95 floor, time to first token p95 1.16 seconds, end-to-end p99 3.23 seconds.
- Modal: 222.2 tokens per second median, 84.7 tokens per second p95 floor, time to first token p95 1.09 seconds, end-to-end p99 3.28 seconds.
- Telnyx: 184.3 tokens per second median, 67.5 tokens per second p95 floor, time to first token p95 1.34 seconds, end-to-end p99 3.48 seconds.
- Together AI: 151.6 tokens per second median, 34.0 tokens per second p95 floor, time to first token p95 2.26 seconds, end-to-end p99 12.45 seconds.
- DeepInfra: 69.4 tokens per second median, 9.0 tokens per second p95 floor, time to first token p95 6.41 seconds, end-to-end p99 17.07 seconds.
- Novita AI: 60.6 tokens per second median, 29.6 tokens per second p95 floor, time to first token p95 3.41 seconds, end-to-end p99 7.25 seconds.
- Fireworks AI: 58.2 tokens per second median, 27.9 tokens per second p95 floor, time to first token p95 5.83 seconds, end-to-end p99 10.49 seconds.
- Z.AI: 56.3 tokens per second median, 27.8 tokens per second p95 floor, time to first token p95 3.14 seconds, end-to-end p99 8.59 seconds.
- Parasail: 52.4 tokens per second median, 22.8 tokens per second p95 floor, time to first token p95 5.54 seconds, end-to-end p99 11.51 seconds.
Which chat experiences can this streaming view inform?
- Embedded product chat. Use time to first token p95 for the initial blank state and median tokens per second for the reading pace after the first visible token.
- Operator-assist chat. Use tokens per second p95 when an agent must scan a streamed suggestion under time pressure and slow continuation is noticeable.
- Structured conversational actions. Keep end-to-end latency and task success beside throughput when the chat response becomes JSON consumed by application code.
How is LLM inference provider latency and accuracy measured here?
- Same model. Every endpoint serves GLM 5.3 Flash; the provider layer is the only variable.
- Same request. One-turn, non-tool chat completion in JSON-object mode, streaming on, temperature 0, top-p 1, reasoning low, 256-token ceiling, 20-second timeout, concurrency one.
- Client clock, pinned tokenizer. Time to first token, end-to-end latency and tokens per second come from the client; visible answer tokens are counted with a pinned
tiktoken o200k_basetokenizer, never provider-reported usage. - Exact task success. An answer counts only when it is schema-valid and exactly correct under the task rubric. Timeouts, HTTP errors and malformed responses stay in the denominator.
- Serverless endpoints only. Dedicated capacity is outside the comparison. Runner and per-request records: openbenchmarks-labs/inference ↗.
Chat inference speed: application-specific questions
Which measured provider streamed chat output fastest after the first token?
Nebius led median tokens per second at 269.3 tokens per second. Its p99 tokens per second floor was 39.9 tokens per second, with time to first token p95 of 7.11 seconds. The measured output is short and structured, so this is evidence about provider streaming behavior rather than a long-conversation benchmark.
How should a chat application interpret tokens per second p95 and p99?
Tokens per second p95 is the speed at least 95% of measured responses achieved; p99 raises that coverage to 99%. These are coverage-based slow-tail floors, and higher throughput is better.
Why can the fastest token generator still feel slow in a chat UI?
Nebius's median result was 269.3 tokens per second, but response experience also includes its 7.11 seconds time to first token p95. A provider can generate quickly after a slow start.
Same run, other questions
- LLM inference provider benchmark: complete scorecard and methodology →
- Best LLM inference provider for AI agents: tool calling latency and accuracy →
- Best LLM inference provider for tool-calling agents →
- Fastest LLM inference provider for AI agents →
- Most accurate LLM inference provider for AI agents →
- Best LLM inference API for chatbots and copilots: time to first token and tokens per second →
- Best LLM inference provider: speed, accuracy and failures compared →
Read the complete LLM inference provider benchmark
Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the LLM Inference Provider Benchmark →