Lowest-latency LLM API for synchronous workflows with an SLA

Which LLM API consumes the least latency budget in a blocking call?

End-to-end p50, p95, and p99 show the complete-answer distribution; delta versus the lowest p99 quantifies tail budget consumed relative to the run leader. Task success and failures prevent incomplete calls from disappearing behind latency-only reporting.

Provider by provider, in the order above

• Baseten: 331 ms median, 3.23 seconds end-to-end p99, time to first token p95 1.16 seconds, 240.5 tokens per second, 95.8% task success, 0.0% failures.
• Modal: 574 ms median, 3.28 seconds end-to-end p99, time to first token p95 1.09 seconds, 222.2 tokens per second, 99.3% task success, 0.0% failures.
• Telnyx: 669 ms median, 3.48 seconds end-to-end p99, time to first token p95 1.34 seconds, 184.3 tokens per second, 99.5% task success, 0.0% failures.
• Novita AI: 1.74 seconds median, 7.25 seconds end-to-end p99, time to first token p95 3.41 seconds, 60.6 tokens per second, 94.2% task success, 4.0% failures.
• Z.AI: 1.75 seconds median, 8.59 seconds end-to-end p99, time to first token p95 3.14 seconds, 56.3 tokens per second, 97.5% task success, 0.2% failures.
• Fireworks AI: 1.95 seconds median, 10.49 seconds end-to-end p99, time to first token p95 5.83 seconds, 58.2 tokens per second, 99.5% task success, 0.0% failures.
• Parasail: 1.59 seconds median, 11.51 seconds end-to-end p99, time to first token p95 5.54 seconds, 52.4 tokens per second, 98.3% task success, 1.3% failures.
• Together AI: 609 ms median, 12.45 seconds end-to-end p99, time to first token p95 2.26 seconds, 151.6 tokens per second, 98.5% task success, 1.0% failures.
• Nebius: 1.45 seconds median, 14.97 seconds end-to-end p99, time to first token p95 7.11 seconds, 269.3 tokens per second, 84.8% task success, 14.8% failures.
• DeepInfra: 1.83 seconds median, 17.07 seconds end-to-end p99, time to first token p95 6.41 seconds, 69.4 tokens per second, 95.5% task success, 4.0% failures.

Which synchronous paths treat the full response as the boundary?

• Request-time enrichment. A web request waits for a typed model result before rendering or committing a transaction.
• Policy and routing checks. The application cannot choose a branch until classification JSON is complete and schema-valid.
• Chained API orchestration. Each blocking inference call consumes part of a larger end-to-end SLA and can compound tail delay.

How is LLM inference provider latency and accuracy measured here?

• Same model. Every endpoint serves GLM 5.3 Flash; the provider layer is the only variable.
• Same request. One-turn, non-tool chat completion in JSON-object mode, streaming on, temperature 0, top-p 1, reasoning low, 256-token ceiling, 20-second timeout, concurrency one.
• Client clock, pinned tokenizer. Time to first token, end-to-end latency and tokens per second come from the client; visible answer tokens are counted with a pinned tiktoken o200k_base tokenizer, never provider-reported usage.
• Exact task success. An answer counts only when it is schema-valid and exactly correct under the task rubric. Timeouts, HTTP errors and malformed responses stay in the denominator.
• Serverless endpoints only. Dedicated capacity is outside the comparison. Runner and per-request records: openbenchmarks-labs/inference ↗.

Inference SLAs for blocking application paths

Same run, other questions

• LLM inference provider benchmark: complete scorecard and methodology →
• Fastest LLM inference provider: time to first token, tokens per second and end-to-end latency →
• Fastest LLM API provider: time to first token and complete-response latency →
• Lowest-latency LLM inference provider: p99 tail analysis →
• Fastest LLM inference API by tokens per second →
• Best LLM inference provider: speed, accuracy and failures compared →
• Fireworks AI vs Together AI →

Read the complete LLM inference provider benchmark

Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the LLM Inference Provider Benchmark →