Best inference provider for chatbots and copilots
What the benchmark found. Modal had the lowest TTFA P95 at 1.09 seconds. Its median visible-answer generation speed was 222.2 tokens/second, and its generation P95 floor was 84.7 tokens/second. The benchmark output is short and structured; teams should replay representative conversational lengths before production selection.
What this page compares. Chatbots and copilots have two user-visible streaming phases: time to the first answer token and the speed of visible continuation. This page puts TTFA and answer-token generation in the same table without pretending unlike units can be averaged into one universal score.
How to read the result. Rows are ordered by TTFA P95 to prioritize the blank wait experienced by most users, with median and P95 generation speed shown beside it. Choose a latency budget for both phases and check E2E P95 plus failure rate before treating a fast starter as the best complete experience.
Workload and configuration. Every provider received the same 600 deliberately easy structured tasks: 200 meeting-note lookups, 200 ticket-triage classifications, and 200 contract-term extractions. GLM 5.3 Flash ran with streaming enabled, reasoning low, temperature 0, top-p 1, a 256-token output ceiling, one attempt, a 20-second timeout, and per-provider concurrency one.
APIs used. The runner called each provider's OpenAI-compatible chat-completions surface. Modal, Baseten, Telnyx, Together AI, Z.AI, Novita AI, Parasail, Fireworks AI, DeepInfra, and Nebius used the model identifiers below; the links open the provider documentation used to verify each adapter.
- Modal:
zai-org/GLM-5.3-Flash· API docs ↗ - Baseten:
zai-org/GLM-5.3-Flash· API docs ↗ - Telnyx:
zai-org/GLM-5.3-Flash· API docs ↗ - Together AI:
zai-org/GLM-5.3-Flash· API docs ↗ - Z.AI:
glm-5.3-flash· API docs ↗ - Novita AI:
zai-org/glm-5.3-flash· API docs ↗ - Parasail:
zai-org/GLM-5.3-Flash· API docs ↗ - Fireworks AI:
accounts/fireworks/models/glm-5p3-flash· API docs ↗ - DeepInfra:
zai-org/GLM-5.3-Flash· API docs ↗ - Nebius:
zai-org/GLM-5.3-Flash· API docs ↗
Chatbot and copilot APIs compared on start and streaming pace
This projection adds two measured deltas absent from the main table: TTFA P95 versus the fastest starter and median generation speed versus the fastest continuation. Lower TTFA and higher generation speed are better.
| Provider / API | TTFA P95 | Δ TTFA vs fastest | Generation P50 | Δ generation vs fastest | Generation P95 | E2E P95 | Failure rate |
|---|---|---|---|---|---|---|---|
| Modalzai-org/GLM-5.3-Flash ↗ | 1.09 s | 0 ms | 222.2 tok/s | -47.1 tok/s | 84.7 tok/s | 1.90 s | 0.0% |
| Basetenzai-org/GLM-5.3-Flash ↗ | 1.16 s | +72 ms | 240.5 tok/s | -28.8 tok/s | 61.6 tok/s | 2.29 s | 0.0% |
| Telnyxzai-org/GLM-5.3-Flash ↗ | 1.34 s | +251 ms | 184.3 tok/s | -85.0 tok/s | 67.5 tok/s | 2.33 s | 0.0% |
| Together AIzai-org/GLM-5.3-Flash ↗ | 2.26 s | +1171 ms | 151.6 tok/s | -117.6 tok/s | 34.0 tok/s | 3.66 s | 1.0% |
| Z.AIglm-5.3-flash ↗ | 3.14 s | +2045 ms | 56.3 tok/s | -213.0 tok/s | 27.8 tok/s | 4.83 s | 0.2% |
| Novita AIzai-org/glm-5.3-flash ↗ | 3.41 s | +2313 ms | 60.6 tok/s | -208.6 tok/s | 29.6 tok/s | 5.00 s | 4.0% |
| Parasailzai-org/GLM-5.3-Flash ↗ | 5.54 s | +4447 ms | 52.4 tok/s | -216.9 tok/s | 22.8 tok/s | 7.74 s | 1.3% |
| Fireworks AIaccounts/fireworks/models/glm-5p3-flash ↗ | 5.83 s | +4737 ms | 58.2 tok/s | -211.0 tok/s | 27.9 tok/s | 6.72 s | 0.0% |
| DeepInfrazai-org/GLM-5.3-Flash ↗ | 6.41 s | +5317 ms | 69.4 tok/s | -199.8 tok/s | 9.0 tok/s | 11.0 s | 4.0% |
| Nebiuszai-org/GLM-5.3-Flash ↗ | 7.11 s | +6015 ms | 269.3 tok/s | 0.0 tok/s | 78.8 tok/s | 7.45 s | 14.8% |
Interactive products where both streaming phases matter
- Writing copilots. Budget TTFA for the first visible suggestion and generation speed for how quickly the remainder becomes editable.
- Support chatbots. Track response start separately from complete-response latency when users see the answer stream before routing metadata is ready.
- In-product assistants. Use slow-tail TTFA and generation floors when a delayed or halting stream is more damaging than median behavior.
Chatbot and copilot streaming decisions
Which provider started visible chatbot answers fastest at TTFA P95?
Modal had the lowest TTFA P95 at 1.09 seconds. Its median visible-answer generation speed was 222.2 tokens/second, and its generation P95 floor was 84.7 tokens/second. The benchmark output is short and structured; teams should replay representative conversational lengths before production selection.
Did the fastest chatbot starter also generate tokens fastest?
Modal had the lowest TTFA P95 at 1.09 seconds, while Nebius had the highest median generation speed at 269.3 tokens/second. These may be different providers because startup delay and continuation speed measure different phases.
How should a copilot team combine TTFA and generation speed without a synthetic score?
Use TTFA P95 to budget the blank wait before visible text, generation P50 for the typical continuation, and generation P95 for the slow-speed floor. Do not combine them into an unvalidated synthetic score.
Read the complete inference benchmark
Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the Inference Benchmark →