Web Search for Coding Agents Benchmark

Real User Workflow in this Benchmark

We evaluate web search in a coding-agent workflow using realistic product tickets.

The agent receives a small existing httpx client and must patch it against the vendor's current official API documentation. The ticket describes the desired capability in natural language, but deliberately does not reveal the exact implementation details.

The ground-truth answer requires discovering an opaque detail from official documentation, such as:

• an API path
• a required header
• a function or import name
• another vendor-specific implementation identifier

Target documentation is enterprise SaaS documentation like Workday and SAP.

The benchmark is intentionally designed so that transforming ticket language into an API-looking string should not be enough. The agent should have to search.

The no-search condition is a necessity. If the model can solve a task by guessing the API or documentation structure, that task may still measure coding ability, but it no longer provides a clean measurement of web-search quality.

How the benchmark is built

Tasks are built from currently live official vendor documentation. Finding those pages is done by a script, not by the coding agent. The search path used here is a neutral third party that is not being evaluated in this benchmark.

1. Discover. We start from a known list of official docs pages. A script then runs ordinary web searches for more pages on the same enterprise SaaS documentation sites. Anything not on those official sites is dropped. No LLM is in this step.
2. Fetch. A second script downloads the full text of those pages into a local folder. Still no LLM. The Authoring Agent only sees these saved files later.
3. Generate. An Authoring Agent reads only those stored pages and proposes product tickets: opaque gold token from the page, capability English that does not name it, and a starter that calls a real but wrong API. Each ticket has a minimum of three very similar configurations; disambiguation is necessary to find the right one.
4. Verify. Automatic keep-gate rejects a ticket if it does not follow dataset guidelines such as naming the ground truth keywords in the ticket.
5. Manual review. A human reads each accepted draft before it is promoted.

Before a task enters the dataset, it must pass an explicit no-search baseline. If the model can recover the ground-truth implementation from the ticket and starter alone, the task is excluded or rewritten.

The public 30-ticket set is on Hugging Face as openbenchmarks/OB-Code-Websearch. The boards on this page are scored on a held-out private set that is not distributed so that vendors and models cannot train and fit to the Benchmark. Use the public rows to inspect the format; scores on those rows are not comparable to the boards. Harness, scoring and vendor runners are in openbenchmarks-labs/web-search-for-coding-agents.

The model receives only the ticket and the starter code. It is run under one of two tool configurations:

[web_search]
[web_search, web_fetch]

Current limits:

Model: gpt-5.6-sol
Max turns: 32
Search budget: 5
Fetch budget: 5
Repeats: 3

Each vendor is run three times on the same locked ticket set, on both the search-only board and the search & fetch board, to account for model variability. The published task-completion score is the mean of those three runs, shown with the sample standard deviation. Median task time, median task cost, avg search time, and median task tokens are a single pooled number from the same three runs, not ± SD.

What each metric means

• Task completion · mean share of tickets that pass, across three independent runs of the same 100-task set. A submission passes only if the submitted file compiles, all ground-truth tokens are present, each has an associated # source: URL, and that URL actually appeared in the search or fetch results from that run. A sufficiently informative search snippet is enough. The table reports mean ± SD of the three board pass rates.
• Median task time · median end-to-end wall-clock time for the agent to finish the task, from the first turn until submission, pooled across all three repeats.
• Avg search time · mean latency of each web_search tool call, pooled across all three repeats.
• Median task tokens · median LLM prompt plus completion tokens per ticket, pooled across all three repeats.
• API list price· PAYG dollar rate of one search call, and of one fetch call on the search & fetch board.
• Median task cost · median LLM dollar cost per ticket (Braintrust estimated_cost() on the eval trace) plus median search/fetch API spend (call counts × list price), pooled across all three repeats.