benchmarks/inference/lowest latency llm inference provider
LLM inference latency · tail-shape analysis

LLM inference provider with the lowest latency

What the benchmark found. Baseten had the lowest absolute E2E P99 at 3.23 seconds. Modal had the smallest P99-minus-P50 tail expansion at 2.70 seconds; those are different latency questions.

What this page compares. A single latency number hides whether a provider is predictably fast or merely fast most of the time. This page focuses on distribution shape: E2E P99, the delta from the run leader, and the spread between median and P99.

How to read the result. A smaller P99-minus-P50 spread indicates less tail expansion on this workload, but it is not a substitute for absolute latency. Read spread with P99 and failure rate; a narrow distribution can still be consistently slow.

Workload and configuration. Every provider received the same 600 deliberately easy structured tasks: 200 meeting-note lookups, 200 ticket-triage classifications, and 200 contract-term extractions. GLM 5.3 Flash ran with streaming enabled, reasoning low, temperature 0, top-p 1, a 256-token output ceiling, one attempt, a 20-second timeout, and per-provider concurrency one.

APIs used. The runner called each provider's OpenAI-compatible chat-completions surface. Baseten, Modal, Telnyx, Novita AI, Z.AI, Fireworks AI, Parasail, Together AI, Nebius, and DeepInfra used the model identifiers below; the links open the provider documentation used to verify each adapter.

[01] measured comparison

Lowest-latency providers with tail expansion exposed

Sorted by absolute E2E P99. Delta versus the fastest provider and P99-minus-P50 spread are derived from the same request-level aggregate, adding information absent from the main table.

benchmarks/inference/lowest-latency-llm-inference-providerreviewed run
LLM inference latency tail shape and delta from fastest provider
ProviderE2E P50E2E P95E2E P99Δ vs fastestP99 − P50Failure rate
Basetenzai-org/GLM-5.3-Flash ↗331 ms2.29 s3.23 s0 ms2.90 s0.0%
Modalzai-org/GLM-5.3-Flash ↗574 ms1.90 s3.28 s+50 ms2.70 s0.0%
Telnyxzai-org/GLM-5.3-Flash ↗669 ms2.33 s3.48 s+250 ms2.81 s0.0%
Novita AIzai-org/glm-5.3-flash ↗1.74 s5.00 s7.25 s+4020 ms5.51 s4.0%
Z.AIglm-5.3-flash ↗1.75 s4.83 s8.59 s+5362 ms6.84 s0.2%
Fireworks AIaccounts/fireworks/models/glm-5p3-flash ↗1.95 s6.72 s10.5 s+7259 ms8.53 s0.0%
Parasailzai-org/GLM-5.3-Flash ↗1.59 s7.74 s11.5 s+8283 ms9.92 s1.3%
Together AIzai-org/GLM-5.3-Flash ↗609 ms3.66 s12.4 s+9221 ms11.8 s1.0%
Nebiuszai-org/GLM-5.3-Flash ↗1.45 s7.45 s15.0 s+11742 ms13.5 s14.8%
DeepInfrazai-org/GLM-5.3-Flash ↗1.83 s11.0 s17.1 s+13842 ms15.2 s4.0%
[02] workflow fit

Where latency distribution shape matters

  • Interactive SLA budgeting. Use absolute P99 to reserve the inference portion of a user-facing response budget.
  • Multi-step agent chains. Use tail expansion to identify providers whose rare delays may compound across several sequential model calls.
  • Predictable queue processing. Use P95 and P99 with failure rate when downstream worker capacity depends on completion-time variance.
[03] faq

Lowest LLM latency — tail-focused questions

Which LLM inference provider had the smallest P99 wait?

Baseten had the lowest absolute E2E P99 at 3.23 seconds. Modal had the smallest P99-minus-P50 tail expansion at 2.70 seconds; those are different latency questions.

What does P99-minus-P50 reveal about an inference API?

Modal had the smallest measured P99-minus-P50 spread at 2.70 seconds. That describes tail expansion, not absolute speed.

Can the most consistent provider still be slower overall?

Yes. A provider can have a narrow P99-minus-P50 spread while both values remain slower than another provider's. Absolute P99 answers the wait question; spread describes how much the tail expands beyond the median.

[04] source benchmark

Read the complete inference benchmark

Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the Inference Benchmark →