Best LLM inference API for chatbots and copilots: time to first token and tokens per second

Which LLM inference API starts answers fastest and streams fastest for chatbots?

This projection adds two measured deltas absent from the main table: time to first token p95 versus the fastest starter and median tokens per second versus the fastest continuation. Lower time to first token and higher tokens per second are better.

Provider by provider, in the order above

• Modal: 222.2 tokens per second median, 84.7 tokens per second p95 floor, time to first token p95 1.09 seconds, end-to-end p99 3.28 seconds.
• Baseten: 240.5 tokens per second median, 61.6 tokens per second p95 floor, time to first token p95 1.16 seconds, end-to-end p99 3.23 seconds.
• Telnyx: 184.3 tokens per second median, 67.5 tokens per second p95 floor, time to first token p95 1.34 seconds, end-to-end p99 3.48 seconds.
• Together AI: 151.6 tokens per second median, 34.0 tokens per second p95 floor, time to first token p95 2.26 seconds, end-to-end p99 12.45 seconds.
• Z.AI: 56.3 tokens per second median, 27.8 tokens per second p95 floor, time to first token p95 3.14 seconds, end-to-end p99 8.59 seconds.
• Novita AI: 60.6 tokens per second median, 29.6 tokens per second p95 floor, time to first token p95 3.41 seconds, end-to-end p99 7.25 seconds.
• Parasail: 52.4 tokens per second median, 22.8 tokens per second p95 floor, time to first token p95 5.54 seconds, end-to-end p99 11.51 seconds.
• Fireworks AI: 58.2 tokens per second median, 27.9 tokens per second p95 floor, time to first token p95 5.83 seconds, end-to-end p99 10.49 seconds.
• DeepInfra: 69.4 tokens per second median, 9.0 tokens per second p95 floor, time to first token p95 6.41 seconds, end-to-end p99 17.07 seconds.
• Nebius: 269.3 tokens per second median, 78.8 tokens per second p95 floor, time to first token p95 7.11 seconds, end-to-end p99 14.97 seconds.

Which interactive products depend on both streaming phases?

• Writing copilots. Budget time to first token for the first visible suggestion and tokens per second for how quickly the remainder becomes editable.
• Support chatbots. Track response start separately from complete-response latency when users see the answer stream before routing metadata is ready.
• In-product assistants. Use slow-tail time to first token and tokens per second floors when a delayed or halting stream is more damaging than median behavior.

How is LLM inference provider latency and accuracy measured here?

• Same model. Every endpoint serves GLM 5.3 Flash; the provider layer is the only variable.
• Same request. One-turn, non-tool chat completion in JSON-object mode, streaming on, temperature 0, top-p 1, reasoning low, 256-token ceiling, 20-second timeout, concurrency one.
• Client clock, pinned tokenizer. Time to first token, end-to-end latency and tokens per second come from the client; visible answer tokens are counted with a pinned tiktoken o200k_base tokenizer, never provider-reported usage.
• Exact task success. An answer counts only when it is schema-valid and exactly correct under the task rubric. Timeouts, HTTP errors and malformed responses stay in the denominator.
• Serverless endpoints only. Dedicated capacity is outside the comparison. Runner and per-request records: openbenchmarks-labs/inference ↗.

Chatbot and copilot streaming decisions

Same run, other questions

• LLM inference provider benchmark: complete scorecard and methodology →
• Best LLM inference provider for AI agents: tool calling latency and accuracy →
• Best LLM inference provider for tool-calling agents →
• Fastest LLM inference provider for AI agents →
• Most accurate LLM inference provider for AI agents →
• Best LLM inference provider for chat applications →
• Best LLM inference provider: speed, accuracy and failures compared →

Read the complete LLM inference provider benchmark

Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the LLM Inference Provider Benchmark →