benchmarks/inference/fastest inference provider for classification
classification latency · ticket routing · measured tail

Fastest inference provider for classification and intent routing

What the benchmark found. Baseten recorded the lowest E2E P99 at 1.20 seconds on this run, with E2E P50 of 274 ms, task success of 98.0%, and failure rate of 0.0%. The ordering is computed from the ticket-triage family alone, with 200 submitted classifications per provider.

What this page compares. A synchronous classifier blocks the next application branch until the complete category and priority object arrives. This page uses only the 200 ticket-triage requests per provider and orders them by complete-response E2E P99 rather than pooled benchmark latency.

How to read the result. Use E2E P95 for the latency most requests must fit and P99 for the rarer waits that can breach a routing SLA. Read both beside exact task success and failure rate: a fast wrong label or failed request does not complete the routing job.

Workload and configuration. Every provider received the same 600 deliberately easy structured tasks: 200 meeting-note lookups, 200 ticket-triage classifications, and 200 contract-term extractions. GLM 5.3 Flash ran with streaming enabled, reasoning low, temperature 0, top-p 1, a 256-token output ceiling, one attempt, a 20-second timeout, and per-provider concurrency one.

Task-specific table. The table below uses only the published ticket-triage aggregate rather than the three-family pooled summary. Each provider therefore contributes 200 submitted requests to this view.

APIs used. The runner called each provider's OpenAI-compatible chat-completions surface. Baseten, Modal, Telnyx, Novita AI, Z.AI, Together AI, Fireworks AI, Parasail, Nebius, and DeepInfra used the model identifiers below; the links open the provider documentation used to verify each adapter.

[01] measured comparison

Classification providers sorted by ticket E2E P99

Lower latency is better. The delta is calculated from the fastest ticket-triage P99, while P50 and P95 show whether the same provider also leads away from the extreme tail.

benchmarks/inference/fastest-inference-provider-for-classificationreviewed run
Inference providers sorted by E2E P99 latency
Provider E2E P50E2E P95E2E P99Δ vs fastest P99Task successFailure rate
Basetenzai-org/GLM-5.3-Flash ↗274 ms749 ms1.20 s0 ms98.0%0.0%
Modalzai-org/GLM-5.3-Flash ↗493 ms1.21 s2.76 s+1555 ms98.0%0.0%
Telnyxzai-org/GLM-5.3-Flash ↗645 ms2.24 s3.78 s+2582 ms98.5%0.0%
Novita AIzai-org/glm-5.3-flash ↗1.63 s2.77 s4.11 s+2905 ms95.0%3.5%
Z.AIglm-5.3-flash ↗1.62 s3.32 s5.75 s+4551 ms96.5%0.5%
Together AIzai-org/GLM-5.3-Flash ↗417 ms1.48 s6.83 s+5629 ms98.5%0.0%
Fireworks AIaccounts/fireworks/models/glm-5p3-flash ↗1.07 s5.71 s7.70 s+6498 ms98.5%0.0%
Parasailzai-org/GLM-5.3-Flash ↗1.48 s6.08 s13.1 s+11924 ms98.5%0.5%
Nebiuszai-org/GLM-5.3-Flash ↗1.02 s6.19 s15.0 s+13789 ms84.5%14.5%
DeepInfrazai-org/GLM-5.3-Flash ↗1.23 s8.43 s17.1 s+15870 ms96.5%2.0%
[02] workflow fit

Routing paths where classification latency is blocking

  • Pre-chat intent detection. The classifier must choose the bot flow or human queue before the interaction can continue.
  • Incident severity routing. A priority enum controls paging or escalation, so completion latency and exactness must be evaluated together.
  • Tool dispatch. A bounded class selects the next API call; E2E latency, not first-token latency, determines when dispatch can occur.
[03] faq

Fast classification APIs — ticket-family evidence

Which provider had the lowest ticket-classification E2E P99?

Baseten recorded the lowest E2E P99 at 1.20 seconds on this run, with E2E P50 of 274 ms, task success of 98.0%, and failure rate of 0.0%. The ordering is computed from the ticket-triage family alone, with 200 submitted classifications per provider.

Did one provider lead both median and tail ticket-routing latency?

Baseten had the lowest E2E P50 at 274 ms, while Baseten had the lowest E2E P99 at 1.20 seconds. The same provider led both points of the distribution.

How do timed-out classification calls affect this comparison?

Timeouts, HTTP errors, and transport failures have no completed visible answer and therefore no honest E2E latency value. They stay in the submitted-request denominator for failure rate and task success, which must be read beside the latency columns.

[04] source benchmark

Read the complete inference benchmark

Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the Inference Benchmark →