For most of software history, software was a tool operated by humans.
So the product could be described by its features. What does it let you do?
Now software is becoming the thing that does the work.
You give it a task, and the product is whether it gets that task done.
Once software owns the task, the important question changes.
How well can it complete my task?
That is what an eval measures.
A benchmark evaluates these capabilities across many vendors, on the same task and under the same conditions, so their ability to complete that task can be compared directly.
What we do
Openbenchmarks is an independent evals and benchmarking company.
Our benchmarks are domain-specific and task-specific. Each one measures real work that people in that domain actually do, not a synthetic stand-in for it.
For each domain, we work with people who know the task deeply, build the ground truth with them, and design the benchmark around how that work is actually done.
Then we test every API, tool, and agent that claims to do it.
How the data is split
Three ways, the same training, validation and test structure used in machine learning. The founders came up through Kaggle-style competition, where a leaderboard only means anything because the private set is not visible to anyone.
| Set | Who sees it | What it is for |
|---|---|---|
| Public | everyone | reproducing the method |
| Validation | customers | product development |
| Private | nobody | the published benchmark |
A provider can optimise perfectly against everything it can see and still gets measured on questions it has never seen.
How a benchmark is produced
| Step | What happens |
|---|---|
| Ground truth | Built by hand from first-party sources. The correct answers exist before any provider runs, and every question is verifiable after the fact. |
| Provider runs | The same input goes to every provider through its public API, on documented credentials and the cheapest publicly listed plan. Where a model or a judge is involved, it is held constant. The only thing that changes is the provider. |
| Scoring | Metrics fit the task instead of one scorecard forced onto everything. Extracted-answer accuracy for lookup, task completion for coding agents, F1 against a reviewed set for multi-hop, precision@K for lookalikes. Cost and latency sit alongside and never get blended into the score. |
| Publication | Aggregate scores go on the board and into the public JSON API. Raw inputs and PII never leave secure storage. When a number changes, the changelog says why. |
Open source
The eval harness is public. The prompts, the judge, the scoring code, the runners that call each provider.
Anyone can open a pull request. Add a vendor we missed. Propose a metric we should be reporting. Change how something is scored and argue for it in the open.
If you think a number on one of these boards is wrong, the code that produced it is at github.com/openbenchmarks-labs. The public datasets are on Hugging Face, and every score is in the public API.
How we make money
Nobody pays to be included, ranked or removed. If a provider belongs in a category we benchmark, it goes on the board whether it wants to be there or not.
We sell evals as a service. Customers get the public set, the validation set, and evals on both. That is what a provider uses to see where it is losing and improve their products. It buys nothing on the leaderboard, because the leaderboard runs on a private dataset that no customer can reach and that is regularly refreshed.
Corrections and contact
If you are a benchmarked provider and you think a number is wrong, email founders@openbenchmarks.com with the provider, dataset slice, run timestamp, and your evidence. We re-run and re-publish with the methodology updated.