Fastest inference provider for small tasks
What the benchmark found. Baseten recorded the lowest E2E P99 at 3.23 seconds on this run, with E2E P50 of 331 ms, task success of 95.8%, and failure rate of 0.0%. The ranking covers concurrency-one microtasks with a maximum 256-token completion, not batch throughput.
What this page compares. This page ranks providers on the complete client-observed wait for short lookup, classification, and extraction requests. The output is bounded to 256 tokens, so the result targets microtasks rather than essays, code generation, or long-context synthesis.
How to read the result. Use E2E P99 when the small task blocks another application step. Compare P50 to understand the typical wait, then inspect task success and failure rate so a provider is not rewarded for responses that did not complete correctly.
Workload and configuration. Every provider received the same 600 deliberately easy structured tasks: 200 meeting-note lookups, 200 ticket-triage classifications, and 200 contract-term extractions. GLM 5.3 Flash ran with streaming enabled, reasoning low, temperature 0, top-p 1, a 256-token output ceiling, one attempt, a 20-second timeout, and per-provider concurrency one.
APIs used. The runner called each provider's OpenAI-compatible chat-completions surface. Baseten, Modal, Telnyx, Novita AI, Z.AI, Fireworks AI, Parasail, Together AI, Nebius, and DeepInfra used the model identifiers below; the links open the provider documentation used to verify each adapter.
- Baseten:
zai-org/GLM-5.3-Flash· API docs ↗ - Modal:
zai-org/GLM-5.3-Flash· API docs ↗ - Telnyx:
zai-org/GLM-5.3-Flash· API docs ↗ - Novita AI:
zai-org/glm-5.3-flash· API docs ↗ - Z.AI:
glm-5.3-flash· API docs ↗ - Fireworks AI:
accounts/fireworks/models/glm-5p3-flash· API docs ↗ - Parasail:
zai-org/GLM-5.3-Flash· API docs ↗ - Together AI:
zai-org/GLM-5.3-Flash· API docs ↗ - Nebius:
zai-org/GLM-5.3-Flash· API docs ↗ - DeepInfra:
zai-org/GLM-5.3-Flash· API docs ↗
Inference providers sorted by small-task E2E P99
Lower is better. Delta versus the fastest P99 result makes the tail-latency gap explicit, while task success and failures retain the denominator context for all 600 submitted requests.
| Provider | E2E P50 | E2E P95 | E2E P99 | Δ vs fastest P99 | Task success | Failure rate |
|---|---|---|---|---|---|---|
| Basetenzai-org/GLM-5.3-Flash ↗ | 331 ms | 2.29 s | 3.23 s | 0 ms | 95.8% | 0.0% |
| Modalzai-org/GLM-5.3-Flash ↗ | 574 ms | 1.90 s | 3.28 s | +50 ms | 99.3% | 0.0% |
| Telnyxzai-org/GLM-5.3-Flash ↗ | 669 ms | 2.33 s | 3.48 s | +250 ms | 99.5% | 0.0% |
| Novita AIzai-org/glm-5.3-flash ↗ | 1.74 s | 5.00 s | 7.25 s | +4020 ms | 94.2% | 4.0% |
| Z.AIglm-5.3-flash ↗ | 1.75 s | 4.83 s | 8.59 s | +5362 ms | 97.5% | 0.2% |
| Fireworks AIaccounts/fireworks/models/glm-5p3-flash ↗ | 1.95 s | 6.72 s | 10.5 s | +7259 ms | 99.5% | 0.0% |
| Parasailzai-org/GLM-5.3-Flash ↗ | 1.59 s | 7.74 s | 11.5 s | +8283 ms | 98.3% | 1.3% |
| Together AIzai-org/GLM-5.3-Flash ↗ | 609 ms | 3.66 s | 12.4 s | +9221 ms | 98.5% | 1.0% |
| Nebiuszai-org/GLM-5.3-Flash ↗ | 1.45 s | 7.45 s | 15.0 s | +11742 ms | 84.8% | 14.8% |
| DeepInfrazai-org/GLM-5.3-Flash ↗ | 1.83 s | 11.0 s | 17.1 s | +13842 ms | 95.5% | 4.0% |
Short workflows where tail latency is visible
- Form autofill. A user waits for a small extracted object before reviewing or submitting a record.
- Request routing. A classifier must finish before the application selects a queue, policy, or next tool.
- Inline factual lookup. A compact answer is rendered inside an existing workflow rather than as a long generated response.
Fast inference for small tasks
Which provider returned small structured tasks fastest at E2E P99?
Baseten recorded the lowest E2E P99 at 3.23 seconds on this run, with E2E P50 of 331 ms, task success of 95.8%, and failure rate of 0.0%. The ranking covers concurrency-one microtasks with a maximum 256-token completion, not batch throughput.
Can the fastest typical microtask provider lose at the slow tail?
Baseten had the lowest E2E P50 at 331 ms, while Baseten had the lowest E2E P99 at 3.23 seconds. The same provider led both points of the distribution.
Why are timed-out small-task requests reported outside latency percentiles?
Timeouts, HTTP errors, and transport failures have no completed visible answer and therefore no honest E2E latency value. They stay in the submitted-request denominator for failure rate and task success, which must be read beside the latency columns.
Read the complete inference benchmark
Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the Inference Benchmark →