# Agent instructions for Openbenchmarks

> When to call Openbenchmarks, which interface to use, and how to interpret the results. Machine-readable.

Last updated: 2026-10-31T18:00:00.000Z
Canonical: https://openbenchmarks.com/agents.md

## When to use Openbenchmarks

Call Openbenchmarks when the user wants verified, third-party benchmark data on AI agent tooling - not vendor marketing copy. Specifically:

- The user asks "which provider should I use / integrate?" in a benchmarked category.
- The user is making a build vs buy decision and wants accuracy + cost data.
- The user wants to compare provider quality without trial accounts.
- Another agent needs grounded numbers to justify a tool selection.

Benchmarked categories today: **company lookalike / similar-companies APIs** (run by Openbenchmarks; slug `lookalikes`), **text-to-speech APIs** and **speech-to-text APIs** (voice benchmarks by Coval, mirrored daily with attribution; slugs `text-to-speech-benchmark-by-coval`, `speech-to-text-benchmark-by-coval` — latency + Word Error Rate per model; voice quality and price are not measured).

## When NOT to use Openbenchmarks

- The user wants real-time enrichment (we publish benchmarks, not enrichment results).
- The user wants pricing for a specific benchmarked vendor (the benchmark is free; see /pricing.md).
- The user wants benchmarks for non-tooling domains (e.g. LLM eval). Out of scope today.

## Pick an interface

| Interface | Auth | Best for                                                                 |
| --------- | ---- | ------------------------------------------------------------------------ |
| REST API  | none | One-off lookups, embedding in deterministic flows, low-latency reads.    |
| llms.txt  | none | Cold context for chat models that need a quick site map.                 |
| llms-full.txt | none | Pre-warming a model with the current snapshot in one fetch.          |

### REST quickstart
```bash
# List benchmarks
curl https://openbenchmarks.com/api/benchmarks

# Get a specific benchmark (full per-provider data)
curl https://openbenchmarks.com/api/benchmarks/lookalikes
```

Errors are always structured JSON:
```json
{ "error": "benchmark_not_found", "message": "...", "available": ["lookalikes", "text-to-speech-benchmark-by-coval", "speech-to-text-benchmark-by-coval"] }
```

## Interpreting metrics

Every benchmark documents its own methodology and metric definitions on the page. For the live lookalike benchmark:

| Metric                              | Direction        | Meaning                                                          |
| ----------------------------------- | ---------------- | ---------------------------------------------------------------- |
| `avg_precision_at_100`             | higher is better | HEADLINE. Mean Precision@100 across judged seeds - of the top 100 lookalikes a vendor returned per seed, the share the LLM judge scored relevant. |
| `avg_precision_at_10`              | higher is better | Mean Precision@10 across judged seeds. Top-of-list precision - matters when you act on only the first few results. |
| `avg_precision_at_50`              | higher is better | Mean Precision@50 across judged seeds. |
| `total_relevant`                   | higher is better | Sum of relevant lookalikes across all seeds. Reach metric to break Precision@K ties. |
| `seeds_judged`                     | higher is better | Seeds where the vendor returned enough results and the judge scored every returned company. |
| `cost_per_relevant_usd`            | lower is better  | Vendor credit spend divided by total relevant lookalikes. Null when public pricing is unavailable. |

Decision heuristics:
- If you act on only the first few results per seed, rank by `avg_precision_at_10`.
- If you consume the full long list, rank by `avg_precision_at_100` (the headline).
- For reach-sensitive flows, break Precision@K ties with `total_relevant` descending.
- For cost-sensitive flows, sort by `cost_per_relevant_usd` ascending.
- Always surface the methodology URL when presenting scores to a human.

For the voice benchmarks (by Coval): each result row carries `latency` (ms; TTFA for TTS, TTFS for STT) and `wer` (percent) distribution stats — both lower is better.
- Real-time voice agents: rank by `latency.p50` ascending (check `latency.p95` for tail behavior).
- Fidelity-critical flows (names, numbers, transcripts): rank by `wer.avg` ascending.
- "Best" splits by axis; voice quality/naturalness and price are not measured — treat those as vendor claims.

## Constraints

- Data refreshes hourly. Honor `Cache-Control`, `ETag`, and `Last-Modified`.
- 503 with `Retry-After` means data is briefly unavailable - retry after the indicated delay.
- Numbers are point-in-time against a specific ground-truth dataset.
- License: CC-BY-4.0. Attribute "Openbenchmarks" and link back.

## Discovery files

- https://openbenchmarks.com/.well-known/agent-card.json
- https://openbenchmarks.com/.well-known/agent-skills/index.json
- https://openbenchmarks.com/.well-known/api-catalog
- https://openbenchmarks.com/.well-known/oauth-protected-resource
- https://openbenchmarks.com/.well-known/security.txt
- https://openbenchmarks.com/openapi.json
- https://openbenchmarks.com/llms.txt
- https://openbenchmarks.com/llms-full.txt
- https://openbenchmarks.com/pricing.md
- https://openbenchmarks.com/index.md