Company News Search Benchmark

How the benchmark is built

The task is fixed and the product category varies. Every endpoint on this board answers the same 300 company-news questions, so the only thing that changes between rows is which API was called. The query, the model that reads the results and extracts an answer (gpt-5.6-terra, medium), and the judge (claude-opus-5 via Amazon Bedrock) are held constant.

1. Collect official wires and newsrooms. Fix ground truth by human labelling before calling any vendors. Set of 300 company-news questions.
2. Send every endpoint the same natural-language question, one request, max 10 results. The query is the user question, unchanged. Nothing is rewritten per vendor, and no vendor gets a news-specific filter the others do not.
3. Persist the raw HTTP envelope, normalized hits (url, title, snippet), and latency for every call.
4. An LLM in the harness (gpt-5.6-terra, medium) reads title + excerpt only and extracts an answer. A separate post-hoc judge from an independent model family (claude-opus-5 via Amazon Bedrock) scores accuracy and AR@K against the human labelled ground truth. It never fetches the live page. No vendor APIs are recalled for extract or score.
5. Score accuracy, AR@1, AR@5, snippet tokens, latency, and list-price cost. List providers alphabetically within each band.

A 100-question public set, including gold answers, is on Hugging Face as openbenchmarks/OB-News-Websearch. Source URLs are omitted from that dump. Use it to inspect the format and run a local harness; scores on those rows are not comparable to this board, which uses the locked 300-question set. Harness, scoring and vendor runners are in openbenchmarks-labs/company-news.

Two mechanisms, one question set

General web search APIs and dedicated news indexes reach company news by different mechanisms: one searches the open web, the other ingests wires and newsrooms directly. They are banded and ranked inside each band rather than cross-ranked, because a single ordering would assert the two are drop-in substitutes, which is a procurement claim this benchmark does not test.

The comparison across bands is still sound: both bands answer the identical 300 questions under the identical protocol above. The gap between the best web search API and the best news index is a measured difference in result quality, not an artefact of the two being asked different things.

Out of scope for this benchmark

This is an atomic company-news fact lookup benchmark. The following are not measured yet.

1. Query rewriting and multi-hop searches. The user query is sent as-is to every API and is a single fact-lookup query. Multi-hop search is measured on the multi-turn company search benchmark.
2. Varying search depth. Every endpoint returns the top 10 results. We measure retrieval quality at that cutoff; we do not run higher-k searches or tune depth.
3. Streaming, webhooks and freshness latency. A news index may push an item minutes after publication. This board measures pull-time lookup quality, not time-to-index.

What each metric means

• Accuracy · gpt-5.6-terra (medium) writes an answer from the returned titles and snippets. The judge scores that written answer against human labelled ground truth. Reported alongside recall, latency, and cost.
• AR@K · answer recall at K. Whether any of the top K vendor snippets already contained the human labelled ground truth. No extract. The judge scores the snippets themselves.
• AR@1 · the first snippet already contained the human labelled ground truth.
• AR@5 · whether any one of the top 5 snippets already contained the full human labelled ground truth. AR@5 can be higher than accuracy: accuracy scores the written extract, which reads all 10 snippets. If a top-5 snippet already has the full fact but later hits in the top 10 conflict, the extract can return empty or say the fact was not found. AR@5 is then 1 and accuracy is 0.
• Latency · speed. Mean wall time of the search request, not the extract or score calls.
• Snippet tokens · mean approximate tokens in the returned results per question (~4 characters / token).
• Total $ · search list price for the run. Judge tokens are not in this column.
• Cost per 1,000 queries · published PAYG list price of the endpoint we called, linked to the vendor pricing page. Per-request search APIs are $ / 1,000 queries. Autobound and Datahyena bill credits, so that cell is $ / 1,000 credits. Volume discounts and signup credits are excluded.
• $ / 1k correct · $ / 1k queries divided by extracted-answer accuracy. A supporting cost metric. Same cost-to-quality figure as the cheapest page.
• Endpoint & configuration · the HTTP path we called, then the vendor setting that tells two rows of the same vendor apart. Single-endpoint vendors show their scope when the call has one. The vendor column uses the company name on every arm.