Fastest inference provider for my AI agent
What the benchmark found. Baseten recorded the lowest E2E P99 at 3.23 seconds on this run, with E2E P50 of 331 ms, task success of 95.8%, and failure rate of 0.0%. The ordering reflects complete answers for sequential agent steps, not maximum batch throughput.
What this page compares. This page treats the fastest provider as the one that returns the complete answer an agent can act on, not merely the one that opens a stream first. Providers are ordered by E2E P99 across the same short lookup, classification, and extraction calls.
How to read the result. Use E2E P99 when an agent step sits on the critical path, because sequential calls compound slow-tail delays. TTFA matters only when the agent exposes partial text to a person before it has a complete result.
Workload and configuration. Every provider received the same 600 deliberately easy structured tasks: 200 meeting-note lookups, 200 ticket-triage classifications, and 200 contract-term extractions. GLM 5.3 Flash ran with streaming enabled, reasoning low, temperature 0, top-p 1, a 256-token output ceiling, one attempt, a 20-second timeout, and per-provider concurrency one.
APIs used. The runner called each provider's OpenAI-compatible chat-completions surface. Baseten, Modal, Telnyx, Novita AI, Z.AI, Fireworks AI, Parasail, Together AI, Nebius, and DeepInfra used the model identifiers below; the links open the provider documentation used to verify each adapter.
- Baseten:
zai-org/GLM-5.3-Flash· API docs ↗ - Modal:
zai-org/GLM-5.3-Flash· API docs ↗ - Telnyx:
zai-org/GLM-5.3-Flash· API docs ↗ - Novita AI:
zai-org/glm-5.3-flash· API docs ↗ - Z.AI:
glm-5.3-flash· API docs ↗ - Fireworks AI:
accounts/fireworks/models/glm-5p3-flash· API docs ↗ - Parasail:
zai-org/GLM-5.3-Flash· API docs ↗ - Together AI:
zai-org/GLM-5.3-Flash· API docs ↗ - Nebius:
zai-org/GLM-5.3-Flash· API docs ↗ - DeepInfra:
zai-org/GLM-5.3-Flash· API docs ↗
Inference providers sorted by AI-agent completion latency
Lower E2E P99 is better. Delta from the fastest provider shows the extra tail delay introduced at one agent step; task success and failures remain visible because an unusable response cannot advance the agent.
| Provider | E2E P50 | E2E P95 | E2E P99 | Δ vs fastest P99 | Task success | Failure rate |
|---|---|---|---|---|---|---|
| Basetenzai-org/GLM-5.3-Flash ↗ | 331 ms | 2.29 s | 3.23 s | 0 ms | 95.8% | 0.0% |
| Modalzai-org/GLM-5.3-Flash ↗ | 574 ms | 1.90 s | 3.28 s | +50 ms | 99.3% | 0.0% |
| Telnyxzai-org/GLM-5.3-Flash ↗ | 669 ms | 2.33 s | 3.48 s | +250 ms | 99.5% | 0.0% |
| Novita AIzai-org/glm-5.3-flash ↗ | 1.74 s | 5.00 s | 7.25 s | +4020 ms | 94.2% | 4.0% |
| Z.AIglm-5.3-flash ↗ | 1.75 s | 4.83 s | 8.59 s | +5362 ms | 97.5% | 0.2% |
| Fireworks AIaccounts/fireworks/models/glm-5p3-flash ↗ | 1.95 s | 6.72 s | 10.5 s | +7259 ms | 99.5% | 0.0% |
| Parasailzai-org/GLM-5.3-Flash ↗ | 1.59 s | 7.74 s | 11.5 s | +8283 ms | 98.3% | 1.3% |
| Together AIzai-org/GLM-5.3-Flash ↗ | 609 ms | 3.66 s | 12.4 s | +9221 ms | 98.5% | 1.0% |
| Nebiuszai-org/GLM-5.3-Flash ↗ | 1.45 s | 7.45 s | 15.0 s | +11742 ms | 84.8% | 14.8% |
| DeepInfrazai-org/GLM-5.3-Flash ↗ | 1.83 s | 11.0 s | 17.1 s | +13842 ms | 95.5% | 4.0% |
Agent paths where inference latency compounds
- Sequential tool chains. Use E2E P99 when the output of one inference call determines the arguments for the next tool call.
- Interactive research agents. Use TTFA when narration is visible, but use E2E latency for the structured result that unlocks another search or fetch.
- Real-time voice or support agents. Use P95 and P99 with failure rate when a short classification blocks a user-visible response.
Fast AI-agent inference questions
Which provider completed short AI-agent inference calls fastest at P99?
Baseten recorded the lowest E2E P99 at 3.23 seconds on this run, with E2E P50 of 331 ms, task success of 95.8%, and failure rate of 0.0%. The ordering reflects complete answers for sequential agent steps, not maximum batch throughput.
Can the median AI-agent latency leader differ from the P99 leader?
Baseten had the lowest E2E P50 at 331 ms, while Baseten had the lowest E2E P99 at 3.23 seconds. The same provider led both points of the distribution.
Why do failed agent inference calls stay outside latency percentiles?
Timeouts, HTTP errors, and transport failures have no completed visible answer and therefore no honest E2E latency value. They stay in the submitted-request denominator for failure rate and task success, which must be read beside the latency columns.
Read the complete inference benchmark
Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the Inference Benchmark →