---
title: "About | Openbenchmarks"
description: "Openbenchmarks is an independent benchmarking company. How our benchmarks are designed, split, run and published, and how we stay independent."
url: https://openbenchmarks.com/about
---

[Openbenchmarks ](https://openbenchmarks.com/) 

[benchmarks](https://openbenchmarks.com/lookalikes) 

# About Openbenchmarks

Openbenchmarks is an independent benchmarking company. We develop open benchmarks that measure how well APIs, tools and agents complete specific tasks. For each one we publish the methodology and a public dataset, and open-source the evaluation harness so anyone can contribute to it.

[Backed by Y Combinator](https://www.ycombinator.com/companies/openbenchmarks) 

## Evals as ground truth

Evals are taking the central position at every company. They touch every team, from product and engineering to go-to-market. Openbenchmarks measures products against the workflows and tasks their users actually run, and helps companies use evals and benchmarks in their go-to-market.

Their importance grows as code generation speeds up. Companies can now ship more than they can evaluate, so the constraint is knowing whether what shipped is useful to the customer it was built for.

Openbenchmarks sits between what customers need and what products do. We turn the work customers need done into tasks that can be measured, and evaluate every provider against them independently. The results are published for the people choosing a product and for the agents researching on their behalf.

## Agents are the next buyers

A growing share of software is chosen by agents. They search for tools, read benchmarks and comparisons, and shortlist options on a buyer's behalf, and increasingly they make the decision themselves.

When they do, they give particular weight to independent, verifiable sources, and our own measurements show agents retrieving and citing our benchmarks. A benchmark is the evaluation an agent would otherwise have to run itself. We run it in advance and publish the result where agents can find it.

## Independence

Nobody pays to be included, ranked or removed. A provider that belongs in a category we benchmark is evaluated whether or not it is a customer.

Some benchmarks start with us, and some start with a partner pointing to a domain its customers care about. We work with as many vendors in that category as we can, and make sure the tasks reflect what their customers actually do. We design the tasks ourselves, every task applies to every provider, and partners can rank below other providers on the benchmarks they inform.

We sell evals as a service, and many partners are also customers. Any vendor can license the validation set and evaluations on it, and use them for product insights and to improve its product on the tasks the benchmark measures.

Published rankings come only from the held-out set, which is run just before publication. Its questions are never published, its answer keys never leave Openbenchmarks, and evaluation runs are not identifiable to providers. No one, partner or not, sees a ranking before it goes live, and results are published as measured.

## What we benchmark

Each benchmark measures a real task in a given domain. The setup is designed so that any difference in score comes from the provider. We run each product the way its vendor would recommend a customer use it, so every vendor is measured with its best foot forward. Any vendor, partner or not, can give us its recommended configuration, and the harness follows each API's own settings instead of forcing a generic call.

Every benchmark is measured on three axes, each reported on its own:

| Axis | What it measures |
| --- | --- |
| Correctness | Whether the provider completes the task correctly, defined for each domain: the exact figure for a finance question, the agreed clause for a redlined contract |
| Cost | What it costs to complete the task at the provider's listed price |
| Latency | How long the provider takes to return a result |

We design benchmarks that measure where each product is genuinely different, so its customers can make a confident choice about which product to pick.

## Dataset splits

| Split | Access | Purpose |
| --- | --- | --- |
| Public | Open, on Hugging Face | Reproducing the method |
| Validation | Licensed to customers | Product development |
| Held-out | Not released | All published results |

Published scores are computed on the held-out set only. A provider can optimise against everything it can see and is still measured on questions it has not seen.

## Open source

The evaluation harness is public, including the prompts, the judge configuration, the scoring code and the runner for each provider, at [github.com/openbenchmarks-labs](https://github.com/openbenchmarks-labs). Public splits are on [Hugging Face](https://huggingface.co/openbenchmarks), and results for several benchmarks are also available from the [public API](https://openbenchmarks.com/api/benchmarks).

Any provider can be evaluated on a benchmark by opening a pull request that adds it to the harness, and we evaluate every provider submitted. Pull requests that propose a metric or change how a task is scored are welcome too.

## Corrections

If you represent a benchmarked provider and believe a result is wrong, write to [founders@openbenchmarks.com](mailto:founders@openbenchmarks.com) with the provider, the dataset split, the run timestamp and your evidence. We re-run the evaluation and publish the result with the methodology updated.

Openbenchmarks 

[← home](https://openbenchmarks.com/) [terms](https://openbenchmarks.com/legal/terms) [privacy](https://openbenchmarks.com/legal/privacy) © 2026 Openbenchmarks