benchmarks/inference/fastest inference provider for classification
classification latency · ticket routing · measured tail · measured Sep 3, 2026

Fastest LLM API for classification and ticket routing

What the benchmark found. Baseten recorded the lowest end-to-end p99 latency at 1.20 seconds on this run, with a median of 274 ms, time to first token p95 of 538 ms, task success of 98.0%, and failure rate of 0.0%. The ordering is computed from the ticket-triage family alone, with 200 submitted classifications per provider. Measured across 10 LLM inference providers serving GLM 5.3 Flash, 200 requests each, Sep 3, 2026.

Providers measured. Baseten, DeepInfra, Fireworks AI, Modal, Nebius, Novita AI, Parasail, Telnyx, Together AI, and Z.AI, each serving GLM 5.3 Flash behind an OpenAI-compatible chat-completions endpoint.

What this page compares. A synchronous classifier blocks the next application branch until the complete category and priority object arrives. This page uses only the 200 ticket-triage requests per provider and orders them by complete-response end-to-end p99 rather than pooled benchmark latency.

How to read the result. Use end-to-end p95 for the latency most requests must fit and p99 for the rarer waits that can breach a routing SLA. Read both beside exact task success and failure rate: a fast wrong label or failed request does not complete the routing job.

Workload and configuration. Every provider received the same 600 deliberately easy structured tasks: 200 meeting-note lookups, 200 ticket-triage classifications, and 200 contract-term extractions. GLM 5.3 Flash ran with streaming enabled, reasoning low, temperature 0, top-p 1, a 256-token output ceiling, one attempt, a 20-second timeout, and per-provider concurrency one.

Task-specific table. The table below uses only the published ticket-triage aggregate rather than the three-family pooled summary. Each provider therefore contributes 200 submitted requests to this view.

APIs used. The runner called each provider's OpenAI-compatible chat-completions surface with the model identifiers below; the links open the provider documentation used to verify each adapter.

[01] measured comparison

Which LLM API returns a classification fastest at p99?

Lower latency is better. The delta is calculated from the fastest ticket-triage p99, while p50 and p95 show whether the same provider also leads away from the extreme tail.

benchmarks/inference/fastest-inference-provider-for-classificationreviewed run
LLM inference providers sorted by end-to-end p99 latency
Provider End-to-end p50End-to-end p95End-to-end p99Δ vs fastest p99Task successFailure rate
Basetenzai-org/GLM-5.3-Flash ↗274 ms749 ms1.20 s0 ms98.0%0.0%
Modalzai-org/GLM-5.3-Flash ↗493 ms1.21 s2.76 s+1555 ms98.0%0.0%
Telnyxzai-org/GLM-5.3-Flash ↗645 ms2.24 s3.78 s+2582 ms98.5%0.0%
Novita AIzai-org/glm-5.3-flash ↗1.63 s2.77 s4.11 s+2905 ms95.0%3.5%
Z.AIglm-5.3-flash ↗1.62 s3.32 s5.75 s+4551 ms96.5%0.5%
Together AIzai-org/GLM-5.3-Flash ↗417 ms1.48 s6.83 s+5629 ms98.5%0.0%
Fireworks AIaccounts/fireworks/models/glm-5p3-flash ↗1.07 s5.71 s7.70 s+6498 ms98.5%0.0%
Parasailzai-org/GLM-5.3-Flash ↗1.48 s6.08 s13.1 s+11924 ms98.5%0.5%
Nebiuszai-org/GLM-5.3-Flash ↗1.02 s6.19 s15.0 s+13789 ms84.5%14.5%
DeepInfrazai-org/GLM-5.3-Flash ↗1.23 s8.43 s17.1 s+15870 ms96.5%2.0%

Provider by provider, in the order above

  • Baseten: 274 ms median, 1.20 seconds end-to-end p99, time to first token p95 538 ms, 313.0 tokens per second, 98.0% task success, 0.0% failures.
  • Modal: 493 ms median, 2.76 seconds end-to-end p99, time to first token p95 1.06 seconds, 270.3 tokens per second, 98.0% task success, 0.0% failures.
  • Telnyx: 645 ms median, 3.78 seconds end-to-end p99, time to first token p95 2.21 seconds, 211.9 tokens per second, 98.5% task success, 0.0% failures.
  • Novita AI: 1.63 seconds median, 4.11 seconds end-to-end p99, time to first token p95 2.64 seconds, 65.4 tokens per second, 95.0% task success, 3.5% failures.
  • Z.AI: 1.62 seconds median, 5.75 seconds end-to-end p99, time to first token p95 2.65 seconds, 57.2 tokens per second, 96.5% task success, 0.5% failures.
  • Together AI: 417 ms median, 6.83 seconds end-to-end p99, time to first token p95 1.31 seconds, 178.6 tokens per second, 98.5% task success, 0.0% failures.
  • Fireworks AI: 1.07 seconds median, 7.70 seconds end-to-end p99, time to first token p95 5.62 seconds, 63.3 tokens per second, 98.5% task success, 0.0% failures.
  • Parasail: 1.48 seconds median, 13.12 seconds end-to-end p99, time to first token p95 5.67 seconds, 48.8 tokens per second, 98.5% task success, 0.5% failures.
  • Nebius: 1.02 seconds median, 14.99 seconds end-to-end p99, time to first token p95 6.10 seconds, 323.3 tokens per second, 84.5% task success, 14.5% failures.
  • DeepInfra: 1.23 seconds median, 17.07 seconds end-to-end p99, time to first token p95 6.23 seconds, 89.5 tokens per second, 96.5% task success, 2.0% failures.
[02] workflow fit

Where is classification latency blocking?

  • Pre-chat intent detection. The classifier must choose the bot flow or human queue before the interaction can continue.
  • Incident severity routing. A priority enum controls paging or escalation, so completion latency and exactness must be evaluated together.
  • Tool dispatch. A bounded class selects the next API call; end-to-end latency, not first-token latency, determines when dispatch can occur.
[03] how it was measured

How is LLM inference provider latency and accuracy measured here?

  • Same model. Every endpoint serves GLM 5.3 Flash; the provider layer is the only variable.
  • Same request. One-turn, non-tool chat completion in JSON-object mode, streaming on, temperature 0, top-p 1, reasoning low, 256-token ceiling, 20-second timeout, concurrency one.
  • Client clock, pinned tokenizer. Time to first token, end-to-end latency and tokens per second come from the client; visible answer tokens are counted with a pinned tiktoken o200k_base tokenizer, never provider-reported usage.
  • Exact task success. An answer counts only when it is schema-valid and exactly correct under the task rubric. Timeouts, HTTP errors and malformed responses stay in the denominator.
  • Serverless endpoints only. Dedicated capacity is outside the comparison. Runner and per-request records: openbenchmarks-labs/inference ↗.
[04] faq

Fast classification APIs: ticket-family evidence

Which provider had the lowest ticket-classification end-to-end p99?

Baseten recorded the lowest end-to-end p99 latency at 1.20 seconds on this run, with a median of 274 ms, time to first token p95 of 538 ms, task success of 98.0%, and failure rate of 0.0%. The ordering is computed from the ticket-triage family alone, with 200 submitted classifications per provider.

Did one provider lead both median and tail ticket-routing latency?

Baseten had the lowest median end-to-end latency at 274 ms, while Baseten had the lowest end-to-end p99 at 1.20 seconds. The same provider led both points of the distribution.

How do timed-out classification calls affect this comparison?

Timeouts, HTTP errors, and transport failures have no completed visible answer and therefore no honest end-to-end latency value. They stay in the submitted-request denominator for failure rate and task success, which must be read beside the latency columns.

[05] related comparisons
[06] source benchmark

Read the complete LLM inference provider benchmark

Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the LLM Inference Provider Benchmark →

Markdown version