Lowest-latency LLM inference provider: p99 tail analysis

Which LLM inference provider has the lowest p99 latency, and whose tail expands least?

Sorted by absolute end-to-end p99. Delta versus the fastest provider and p99-minus-p50 spread are derived from the same request-level aggregate, adding information absent from the main table.

Provider by provider, in the order above

• Baseten: 331 ms median, 3.23 seconds end-to-end p99, time to first token p95 1.16 seconds, 240.5 tokens per second, 95.8% task success, 0.0% failures.
• Modal: 574 ms median, 3.28 seconds end-to-end p99, time to first token p95 1.09 seconds, 222.2 tokens per second, 99.3% task success, 0.0% failures.
• Telnyx: 669 ms median, 3.48 seconds end-to-end p99, time to first token p95 1.34 seconds, 184.3 tokens per second, 99.5% task success, 0.0% failures.
• Novita AI: 1.74 seconds median, 7.25 seconds end-to-end p99, time to first token p95 3.41 seconds, 60.6 tokens per second, 94.2% task success, 4.0% failures.
• Z.AI: 1.75 seconds median, 8.59 seconds end-to-end p99, time to first token p95 3.14 seconds, 56.3 tokens per second, 97.5% task success, 0.2% failures.
• Fireworks AI: 1.95 seconds median, 10.49 seconds end-to-end p99, time to first token p95 5.83 seconds, 58.2 tokens per second, 99.5% task success, 0.0% failures.
• Parasail: 1.59 seconds median, 11.51 seconds end-to-end p99, time to first token p95 5.54 seconds, 52.4 tokens per second, 98.3% task success, 1.3% failures.
• Together AI: 609 ms median, 12.45 seconds end-to-end p99, time to first token p95 2.26 seconds, 151.6 tokens per second, 98.5% task success, 1.0% failures.
• Nebius: 1.45 seconds median, 14.97 seconds end-to-end p99, time to first token p95 7.11 seconds, 269.3 tokens per second, 84.8% task success, 14.8% failures.
• DeepInfra: 1.83 seconds median, 17.07 seconds end-to-end p99, time to first token p95 6.41 seconds, 69.4 tokens per second, 95.5% task success, 4.0% failures.

Where does latency distribution shape matter?

• Interactive SLA budgeting. Use absolute p99 to reserve the inference portion of a user-facing response budget.
• Multi-step agent chains. Use tail expansion to identify providers whose rare delays may compound across several sequential model calls.
• Predictable queue processing. Use p95 and p99 with failure rate when downstream worker capacity depends on completion-time variance.

How is LLM inference provider latency and accuracy measured here?

• Same model. Every endpoint serves GLM 5.3 Flash; the provider layer is the only variable.
• Same request. One-turn, non-tool chat completion in JSON-object mode, streaming on, temperature 0, top-p 1, reasoning low, 256-token ceiling, 20-second timeout, concurrency one.
• Client clock, pinned tokenizer. Time to first token, end-to-end latency and tokens per second come from the client; visible answer tokens are counted with a pinned tiktoken o200k_base tokenizer, never provider-reported usage.
• Exact task success. An answer counts only when it is schema-valid and exactly correct under the task rubric. Timeouts, HTTP errors and malformed responses stay in the denominator.
• Serverless endpoints only. Dedicated capacity is outside the comparison. Runner and per-request records: openbenchmarks-labs/inference ↗.

Lowest LLM latency: tail-focused questions

Same run, other questions

• LLM inference provider benchmark: complete scorecard and methodology →
• Fastest LLM inference provider: time to first token, tokens per second and end-to-end latency →
• Fastest LLM API provider: time to first token and complete-response latency →
• Fastest LLM inference API by tokens per second →
• Lowest-latency LLM API for synchronous workflows with an SLA →
• Best LLM inference provider: speed, accuracy and failures compared →
• Fireworks AI vs Together AI →

Read the complete LLM inference provider benchmark

Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the LLM Inference Provider Benchmark →