Web Search for Coding Agents
Why we built this. We evaluate web search in a coding-agent workflow using realistic product tickets. We test search APIs on hard enterprise SaaS documentation. The agent receives a small existing httpxclient and must patch it against the vendor's current official API documentation. The ticket describes the desired capability in natural language, but deliberately does not reveal the exact implementation details.
Sample task. A sales team needs to price a quote in ServiceNow Sales CRM. The file the agent gets already makes a live ServiceNow request, but it is the default help-desk call: read the incident table. The gold path is one endpoint.
ticket
Price this quote using the current Sales CRM pricing engine. The starter still reads the incident table, which is the help-desk API for listing support tickets — that cannot compute a price. Look up the official docs and switch the call.
what the docs actually require
POST /api/sn_csm_pricing/v1/pricingengine/computePrice
The inherited code would run today: GET /api/now/table/incident. That returns help-desk tickets. It does not price a quote. The path above lives in Sales CRM pricing docs and is not written anywhere in the ticket.
@@ price.pydef run() -> dict:- response = httpx.get(- f"{BASE}/api/now/table/incident",+ response = httpx.post(+ # source: https://www.servicenow.com/docs/.../sales_crm_pricing-POST-compute-price.html+ f"{BASE}/api/sn_csm_pricing/v1/pricingengine/computePrice", headers={ "Authorization": f"Bearer {TOKEN}", "Accept": "application/json", },
Red is the inherited client. Green is the change after the agent finds the official docs. The # source:comment has to be a docs URL that actually showed up in that run's search or fetch results and not a URL hallucinated by the model.
How it is measured. The model receives only the ticket and the starter code. The search or fetch provider is varied while the model (gpt-5.6-sol), task, budgets (5 search, 32 turns), and runner remain fixed. A submission passes only if the submitted file compiles, all ground-truth tokens are present, each has an associated # source: URL, and that URL actually appeared in the search or fetch results from that run. Both boards are published: search & fetch, then search-only.
Vendors benchmarked. Exa deep, Exa auto, Parallel advanced, Firecrawl, Parallel basic, Tavily advanced, Tavily basic, Linkup standard, Exa fast, Parallel fast, Exa instant, Parallel turbo, Tavily fast, Linkup fast, Brave. Every task is evaluated under two tool configurations: search only and search & fetch. When fetch is enabled, web_fetch goes through the same vendor under test as web_search— a search & fetch pair from one vendor, not search from one and fetch from another.
How to pick a provider.Start from whether fetch is the vendor's or yours. There is no single best search API. It is a tradeoff between grounded task completion, latency, and tokens.
- Search & fetch. The agent searches and fetches through the same vendor. Read the search & fetch board.
- Search only, bring your own fetch. The agent only calls
web_search. Use this if fetch is your own scrape, browser, or another vendor. Read the search-only board. - Highest grounded completion. Read
Task completionon the board that matches your tool setup. - Fastest loop. Read
Median task timeandAvg search time. - Leanest context. Read
Median task tokens.
Benchmark Data
Ranked by task completion, then median task time.
| Rank | Vendor | Endpoint & configuration | Task completion | Median task time | Avg search time | Median task tokens |
|---|---|---|---|---|---|---|
| 1 | Exa deep | POST /search type=deepPOST /contents | 83.0 ± 1.4 | 38s | 4.10s | 23,930 |
| 2 | Exa auto | POST /search type=autoPOST /contents | 82.0 ± 1.4 | 23s | 1.23s | 26,500 |
| 3 | Parallel advanced | POST /v1/search mode=advancedPOST /v1/extract | 79.0 ± 2.8 | 41s | 3.19s | 28,647 |
| 4 | Firecrawl | POST /v2/searchPOST /v2/scrape | 75.5 ± 0.7 | 36s | 3.03s | 17,628 |
| 5 | Parallel basic | POST /v1/search mode=basicPOST /v1/extract | 72.5 ± 2.1 | 37s | 1.73s | 40,452 |
| 6 | Tavily advanced | POST /search search_depth=advancedPOST /extract extract_depth=advanced | 60.5 ± 0.7 | 46s | 3.52s | 24,710 |
| 7 | Tavily basic | POST /search search_depth=basicPOST /extract extract_depth=basic | 58.0 ± 0.0 | 34s | 1.59s | 28,643 |
| 8 | Linkup standard | POST /v1/search depth=standardPOST /v1/fetch mode=standard | 49.5 ± 6.4 | 48s | 2.01s | 60,367 |
Task completion is mean ± SD of 2 runs; n = 100 tasks. Median task time, avg search time, and median task tokens are pooled across those same 2 runs.
| Rank | Vendor | Endpoint & configuration | Task completion | Median task time | Avg search time | Median task tokens |
|---|---|---|---|---|---|---|
| 1 | Firecrawl | POST /v2/search | 70.5 ± 2.1 | 28s | 3.02s | 7,456 |
| 2 | Exa fast | POST /search type=fast | 66.5 ± 2.1 | 21s | 665ms | 22,632 |
| 3 | Parallel fast | POST /v1/search mode=fast | 61.0 ± 1.4 | 22s | 956ms | 15,103 |
| 4 | Exa instant | POST /search type=instant | 60.5 ± 3.5 | 21s | 462ms | 22,615 |
| 5 | Parallel turbo | POST /v1/search mode=turbo | 54.5 ± 0.7 | 20s | 368ms | 18,029 |
| 6 | Tavily fast | POST /search search_depth=fast | 47.5 ± 3.5 | 21s | 288ms | 23,698 |
| 7 | Linkup fast | POST /v1/search depth=fast | 43.0 ± 1.4 | 27s | 1.37s | 23,943 |
| 8 | Brave Search | POST /res/v1/llm/context | 42.0 ± 1.4 | 23s | 550ms | 21,175 |
Task completion is mean ± SD of 2 runs; n = 100 tasks. Median task time, avg search time, and median task tokens are pooled across those same 2 runs.
[02] methodology and metric definitions+
Real User Workflow in this Benchmark
We evaluate web search in a coding-agent workflow using realistic product tickets.
The agent receives a small existing httpx client and must patch it against the vendor's current official API documentation. The ticket describes the desired capability in natural language, but deliberately does not reveal the exact implementation details.
The ground-truth answer requires discovering an opaque detail from official documentation, such as:
- an API path
- a required header
- a function or import name
- another vendor-specific implementation identifier
Target documentation is enterprise SaaS documentation like Workday and SAP.
The benchmark is intentionally designed so that transforming ticket language into an API-looking string should not be enough. The agent should have to search.
The no-search condition is a necessity. If the model can solve a task by guessing the API or documentation structure, that task may still measure coding ability, but it no longer provides a clean measurement of web-search quality.
How the benchmark is built
Tasks are built from currently live official vendor documentation. Finding those pages is done by a script, not by the coding agent. The search path used here is a neutral third party that is not being evaluated in this benchmark.
- Discover. We start from a known list of official docs pages. A script then runs ordinary web searches for more pages on the same enterprise SaaS documentation sites. Anything not on those official sites is dropped. No LLM is in this step.
- Fetch. A second script downloads the full text of those pages into a local folder. Still no LLM. The Authoring Agent only sees these saved files later.
- Generate. An Authoring Agent reads only those stored pages and proposes product tickets: opaque gold token from the page, capability English that does not name it, and a starter that calls a real but wrong API. Each ticket has a minimum of three very similar configurations; disambiguation is necessary to find the right one.
- Verify. Automatic keep-gate rejects a ticket if it does not follow dataset guidelines such as naming the ground truth keywords in the ticket.
- Manual review. A human reads each accepted draft before it is promoted.
Before a task enters the dataset, it must pass an explicit no-search baseline. If the model can recover the ground-truth implementation from the ticket and starter alone, the task is excluded or rewritten.
The public 60-ticket set is on Hugging Face as openbenchmarks/OB-Code-Websearch. The boards on this page are scored on a held-out private set that is not distributed so that vendors and models cannot train and fit to the Benchmark. Use the public rows to inspect the format; scores on those rows are not comparable to the boards. Harness, scoring and vendor runners are in openbenchmarks-labs/web-search-for-coding-agents.
The model receives only the ticket and the starter code. It is run under one of two tool configurations:
[web_search]
[web_search, web_fetch]Current limits:
Model: gpt-5.6-sol
Max turns: 32
Search budget: 5
Fetch budget: 5
Repeats: 2Each vendor is run twice on the same locked ticket set, on both the search-only board and the search & fetch board, to account for model variability. The published task-completion score is the mean of those two runs, shown with the sample standard deviation. Median task time, avg search time, and median task tokens are a single pooled number from the same two runs, not ± SD.
What each metric means
Task completion· mean share of tickets that pass, across two independent runs of the same 100-task set. A submission passes only if the submitted file compiles, all ground-truth tokens are present, each has an associated# source:URL, and that URL actually appeared in the search or fetch results from that run. A sufficiently informative search snippet is enough. The table reports mean ± SD of the two board pass rates.Median task time· median end-to-end wall-clock time for the agent to finish the task, from the first turn until submission, pooled across both repeats.Avg search time· mean latency of eachweb_searchtool call, pooled across both repeats.Median task tokens· median LLM prompt plus completion tokens per ticket, pooled across both repeats.
Why don't we use popular docs like Stripe?
Benchmark is Designed Against Model's Reward Hacking in User Workflow
A key dataset-design finding was that vendors with highly regular, developer-friendly documentation can be poor fits for this benchmark. In earlier Stripe tasks, the model often inferred the documentation slug or URL structure directly from the ticket and reached the correct implementation without performing meaningful retrieval.
For example, when documentation followed a predictable pattern such as /changelog/dahlia/{kebab-case-ticket-title}, the model could construct the likely documentation location from the ticket itself. That tests the model's ability to exploit documentation naming conventions, not the quality of the search provider. Those tasks were removed from the active benchmark.
Keep-gate: if a capable model can reliably solve the task without searching, the task does not belong in the primary search benchmark.