Fastest LLM inference provider for AI agents

Which LLM inference provider completes AI-agent calls fastest at p99?

Lower end-to-end p99 is better. Delta from the fastest provider shows the extra tail delay introduced at one agent step; task success and failures remain visible because an unusable response cannot advance the agent.

Provider by provider, in the order above

• Baseten: 331 ms median, 3.23 seconds end-to-end p99, time to first token p95 1.16 seconds, 240.5 tokens per second, 95.8% task success, 0.0% failures.
• Modal: 574 ms median, 3.28 seconds end-to-end p99, time to first token p95 1.09 seconds, 222.2 tokens per second, 99.3% task success, 0.0% failures.
• Telnyx: 669 ms median, 3.48 seconds end-to-end p99, time to first token p95 1.34 seconds, 184.3 tokens per second, 99.5% task success, 0.0% failures.
• Novita AI: 1.74 seconds median, 7.25 seconds end-to-end p99, time to first token p95 3.41 seconds, 60.6 tokens per second, 94.2% task success, 4.0% failures.
• Z.AI: 1.75 seconds median, 8.59 seconds end-to-end p99, time to first token p95 3.14 seconds, 56.3 tokens per second, 97.5% task success, 0.2% failures.
• Fireworks AI: 1.95 seconds median, 10.49 seconds end-to-end p99, time to first token p95 5.83 seconds, 58.2 tokens per second, 99.5% task success, 0.0% failures.
• Parasail: 1.59 seconds median, 11.51 seconds end-to-end p99, time to first token p95 5.54 seconds, 52.4 tokens per second, 98.3% task success, 1.3% failures.
• Together AI: 609 ms median, 12.45 seconds end-to-end p99, time to first token p95 2.26 seconds, 151.6 tokens per second, 98.5% task success, 1.0% failures.
• Nebius: 1.45 seconds median, 14.97 seconds end-to-end p99, time to first token p95 7.11 seconds, 269.3 tokens per second, 84.8% task success, 14.8% failures.
• DeepInfra: 1.83 seconds median, 17.07 seconds end-to-end p99, time to first token p95 6.41 seconds, 69.4 tokens per second, 95.5% task success, 4.0% failures.

Where does inference latency compound inside an agent?

• Sequential tool chains. Use end-to-end p99 when the output of one inference call determines the arguments for the next tool call.
• Interactive research agents. Use time to first token when narration is visible, but use end-to-end latency for the structured result that unlocks another search or fetch.
• Real-time voice or support agents. Use p95 and p99 with failure rate when a short classification blocks a user-visible response.

How is LLM inference provider latency and accuracy measured here?

• Same model. Every endpoint serves GLM 5.3 Flash; the provider layer is the only variable.
• Same request. One-turn, non-tool chat completion in JSON-object mode, streaming on, temperature 0, top-p 1, reasoning low, 256-token ceiling, 20-second timeout, concurrency one.
• Client clock, pinned tokenizer. Time to first token, end-to-end latency and tokens per second come from the client; visible answer tokens are counted with a pinned tiktoken o200k_base tokenizer, never provider-reported usage.
• Exact task success. An answer counts only when it is schema-valid and exactly correct under the task rubric. Timeouts, HTTP errors and malformed responses stay in the denominator.
• Serverless endpoints only. Dedicated capacity is outside the comparison. Runner and per-request records: openbenchmarks-labs/inference ↗.

Fast AI-agent inference questions

Same run, other questions

• LLM inference provider benchmark: complete scorecard and methodology →
• Best LLM inference provider for AI agents: tool calling latency and accuracy →
• Best LLM inference provider for tool-calling agents →
• Most accurate LLM inference provider for AI agents →
• Best LLM inference API for chatbots and copilots: time to first token and tokens per second →
• Best LLM inference provider for chat applications →
• Best LLM inference provider: speed, accuracy and failures compared →

Read the complete LLM inference provider benchmark

Metric definitions, request scheduling, task rubrics, failure handling, and the complete provider scorecard live on the LLM Inference Provider Benchmark →