benchmarks/inference/fastest inference provider for small tasks
small-task inference latency · measured p99 · measured Sep 3, 2026

Fastest LLM inference provider for small tasks

What the benchmark found. Baseten recorded the lowest end-to-end p99 latency at 3.23 seconds on this run, with a median of 331 ms, time to first token p95 of 1.16 seconds, task success of 95.8%, and failure rate of 0.0%. The ranking covers concurrency-one microtasks with a maximum 256-token completion, not batch throughput. Measured across 10 LLM inference providers serving GLM 5.3 Flash, 600 requests each, Sep 3, 2026.

Providers measured. Baseten, DeepInfra, Fireworks AI, Modal, Nebius, Novita AI, Parasail, Telnyx, Together AI, and Z.AI, each serving GLM 5.3 Flash behind an OpenAI-compatible chat-completions endpoint.

What this page compares. This page ranks providers on the complete client-observed wait for short lookup, classification, and extraction requests. The output is bounded to 256 tokens, so the result targets microtasks rather than essays, code generation, or long-context synthesis.

How to read the result. Use end-to-end p99 when the small task blocks another application step. Compare p50 to understand the typical wait, then inspect task success and failure rate so a provider is not rewarded for responses that did not complete correctly.

Workload and configuration. Every provider received the same 600 deliberately easy structured tasks: 200 meeting-note lookups, 200 ticket-triage classifications, and 200 contract-term extractions. GLM 5.3 Flash ran with streaming enabled, reasoning low, temperature 0, top-p 1, a 256-token output ceiling, one attempt, a 20-second timeout, and per-provider concurrency one.

APIs used. The runner called each provider's OpenAI-compatible chat-completions surface with the model identifiers below; the links open the provider documentation used to verify each adapter.

[01] measured comparison

Which LLM inference provider returns small tasks fastest at p99?

Lower is better. Delta versus the fastest p99 result makes the tail-latency gap explicit, while task success and failures retain the denominator context for all 600 submitted requests.

benchmarks/inference/fastest-inference-provider-for-small-tasksreviewed run
LLM inference providers sorted by end-to-end p99 latency
Provider End-to-end p50End-to-end p95End-to-end p99Δ vs fastest p99Task successFailure rate
Basetenzai-org/GLM-5.3-Flash ↗331 ms2.29 s3.23 s0 ms95.8%0.0%
Modalzai-org/GLM-5.3-Flash ↗574 ms1.90 s3.28 s+50 ms99.3%0.0%
Telnyxzai-org/GLM-5.3-Flash ↗669 ms2.33 s3.48 s+250 ms99.5%0.0%
Novita AIzai-org/glm-5.3-flash ↗1.74 s5.00 s7.25 s+4020 ms94.2%4.0%
Z.AIglm-5.3-flash ↗1.75 s4.83 s8.59 s+5362 ms97.5%0.2%
Fireworks AIaccounts/fireworks/models/glm-5p3-flash ↗1.95 s6.72 s10.5 s+7259 ms99.5%0.0%
Parasailzai-org/GLM-5.3-Flash ↗1.59 s7.74 s11.5 s+8283 ms98.3%1.3%
Together AIzai-org/GLM-5.3-Flash ↗609 ms3.66 s12.4 s+9221 ms98.5%1.0%
Nebiuszai-org/GLM-5.3-Flash ↗1.45 s7.45 s15.0 s+11742 ms84.8%14.8%
DeepInfrazai-org/GLM-5.3-Flash ↗1.83 s11.0 s17.1 s+13842 ms95.5%4.0%

Provider by provider, in the order above

  • Baseten: 331 ms median, 3.23 seconds end-to-end p99, time to first token p95 1.16 seconds, 240.5 tokens per second, 95.8% task success, 0.0% failures.
  • Modal: 574 ms median, 3.28 seconds end-to-end p99, time to first token p95 1.09 seconds, 222.2 tokens per second, 99.3% task success, 0.0% failures.
  • Telnyx: 669 ms median, 3.48 seconds end-to-end p99, time to first token p95 1.34 seconds, 184.3 tokens per second, 99.5% task success, 0.0% failures.
  • Novita AI: 1.74 seconds median, 7.25 seconds end-to-end p99, time to first token p95 3.41 seconds, 60.6 tokens per second, 94.2% task success, 4.0% failures.
  • Z.AI: 1.75 seconds median, 8.59 seconds end-to-end p99, time to first token p95 3.14 seconds, 56.3 tokens per second, 97.5% task success, 0.2% failures.
  • Fireworks AI: 1.95 seconds median, 10.49 seconds end-to-end p99, time to first token p95 5.83 seconds, 58.2 tokens per second, 99.5% task success, 0.0% failures.
  • Parasail: 1.59 seconds median, 11.51 seconds end-to-end p99, time to first token p95 5.54 seconds, 52.4 tokens per second, 98.3% task success, 1.3% failures.
  • Together AI: 609 ms median, 12.45 seconds end-to-end p99, time to first token p95 2.26 seconds, 151.6 tokens per second, 98.5% task success, 1.0% failures.
  • Nebius: 1.45 seconds median, 14.97 seconds end-to-end p99, time to first token p95 7.11 seconds, 269.3 tokens per second, 84.8% task success, 14.8% failures.
  • DeepInfra: 1.83 seconds median, 17.07 seconds end-to-end p99, time to first token p95 6.41 seconds, 69.4 tokens per second, 95.5% task success, 4.0% failures.
[02] workflow fit

Which short workflows expose tail latency?

  • Form autofill. A user waits for a small extracted object before reviewing or submitting a record.
  • Request routing. A classifier must finish before the application selects a queue, policy, or next tool.
  • Inline factual lookup. A compact answer is rendered inside an existing workflow rather than as a long generated response.
[03] how it was measured

How is LLM inference provider latency and accuracy measured here?

  • Same model. Every endpoint serves GLM 5.3 Flash; the provider layer is the only variable.
  • Same request. One-turn, non-tool chat completion in JSON-object mode, streaming on, temperature 0, top-p 1, reasoning low, 256-token ceiling, 20-second timeout, concurrency one.
  • Client clock, pinned tokenizer. Time to first token, end-to-end latency and tokens per second come from the client; visible answer tokens are counted with a pinned tiktoken o200k_base tokenizer, never provider-reported usage.
  • Exact task success. An answer counts only when it is schema-valid and exactly correct under the task rubric. Timeouts, HTTP errors and malformed responses stay in the denominator.
  • Serverless endpoints only. Dedicated capacity is outside the comparison. Runner and per-request records: openbenchmarks-labs/inference ↗.
[04] faq

Fast inference for small tasks

Which provider returned small structured tasks fastest at end-to-end p99?

Baseten recorded the lowest end-to-end p99 latency at 3.23 seconds on this run, with a median of 331 ms, time to first token p95 of 1.16 seconds, task success of 95.8%, and failure rate of 0.0%. The ranking covers concurrency-one microtasks with a maximum 256-token completion, not batch throughput.

Can the fastest typical microtask provider lose at the slow tail?

Baseten had the lowest median end-to-end latency at 331 ms, while Baseten had the lowest end-to-end p99 at 3.23 seconds. The same provider led both points of the distribution.

Why are timed-out small-task requests reported outside latency percentiles?

Timeouts, HTTP errors, and transport failures have no completed visible answer and therefore no honest end-to-end latency value. They stay in the submitted-request denominator for failure rate and task success, which must be read beside the latency columns.

[05] related comparisons
[06] source benchmark

Read the complete LLM inference provider benchmark

Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the LLM Inference Provider Benchmark →

Markdown version