Document Processing Benchmark

9 distinct cases make up the benchmark

Every task type is built from tracked changes in the source file, so every answer key is derived from the markup rather than annotated by hand. These distinct types test different capabilities: a deletion inside a number fails differently from a deleted clause.

• Numeral strike: a strike inside a numeral. Digits struck mid-number leave no word boundary to recover.
• Word-internal strike: a strike inside a word, which collapses into a different real word rather than into nonsense.
• Struck clause: a clause struck with nothing put back.
• Old and new side by side: superseded and replacement wording adjacent in the text, where the question is which one binds.
• Struck definition term: a term struck from a definition, which changes every downstream clause that relies on it.
• Struck table cell: a whole cell struck, where the loss is structural rather than textual.
• Mixed edits on a page: insertions and a deletion together, so the parser has to keep two kinds of markup apart.
• Proposal between rounds: what happened to a proposal across two drafts, which needs both documents read consistently.
• Stale cross-reference: renumbering that leaves a reference pointing at the wrong clause.

Source data and its transformation

Built from RedlineBench (CC-BY-4.0), a set of simulated contract negotiations marked up by trained attorneys representing each side. The tracked changes are genuine Word tracked changes and render identically to any other redline. The parties are fictional and the deals simulated; what is real is the markup, and the benchmark tests the rendering-and-parsing path, which is unaffected by whether the underlying deal was live.

1. Strip comments. The attorneys' margin comments are their own reasoning, and several state the answer outright. A parser that captures annotations would hand the agent the answer, so we would be scoring annotation capture instead of strike fidelity. Comment parts, relationships, content-type overrides and in-document anchors are all removed, and a check proves no comment text survives anywhere in the file.
2. Render. LibreOffice converts each stripped file to PDF. Tracked changes come through struck and coloured, the way Word draws them.
3. Read the changes. Every edited clause is sorted by the shape of the edit, taken straight from the source file. Types that span a page or two drafts also record where each clause landed once printed.
4. Write the questions. A model writes one question per edit from the resolved facts. It never sees any vendor's output, and it is asked to discard edits that do not change what the contract requires.
5. Check. Automatic checks reject any question whose right and wrong answers turn out to be the same, or that asks about the text as text rather than about what the contract requires.

The render to PDF tests the parser

In the Word file a deletion is structured: the text is explicitly labelled as removed. Print that file to PDF and the label is gone. All that remains is a line drawn across characters that are otherwise ordinary text. Whether the distinction survives is now entirely down to the parser.

How each parser was called

Every arm was given the same thing, the contract PDF, and asked for markdown. None was handed the source Word file, which would let it read the deletion out of OOXML instead of recovering it from the page. Where a vendor ships a first-party SDK it is used rather than raw HTTP, so the arm exercises the path that vendor's own documentation puts customers on. The exact interface and settings for each arm are in the endpoint column of the board, and each parser name there links to the vendor documentation for the call.

Tracked-change handling is the one setting that is not uniform, because it cannot be: Reducto, Extend and Pulse expose it as a flag, Datalab as a separate endpoint, and LlamaParse applies it without asking. Each arm runs it the way its vendor documents. The Datalab convert arm is the exception and is on the board deliberately: it is the same vendor with no tracked-change support at all, which is what the gap between the two Datalab rows is measuring.

The frontier arms have no such flag to set, because for a general model the prompt is the flag. They receive the identical PDF and one shared instruction that asks for a faithful markdown rendering.

That prompt in full, identical for every frontier arm:

Convert this document to clean, well-structured markdown. Reproduce the page faithfully, preserving headings, paragraphs, lists, tables and inline formatting. Output only the document content, with no commentary, preamble or code fences.

What it asks for is what a document parser sells: the page, in markdown, faithfully. Whether faithfully includes the deletions is the thing under test.

How an answer is marked

Every answer is bucketed by a model judge, gpt-6-sol. It sees the question, the correct, stale and fused reference answers, and the agent's answer. It never sees the contract and is never told which parser produced the text. The answers to one question are judged together, twenty to a call, each bucketed on its own, so “the section labelled 2.2” against an expected “Section 2.2” is never marked down for phrasing. A fifth label, ambiguous, is reserved for an answer that asserts two positions at once and is counted with other.

Version 1 scored by whole-token matching against the three reference strings and sent only the residue to a judge. That match is still recorded on every row as an audit of the judge. The judge prompt was revised on 29 September after 13 of 20 sampled disagreements between judge and token match turned out to be answers marked wrong for naming a live replacement clause the reference did not mention.

Nine replicates make one score

Parser output varies between calls. A probe step parsed a few contracts repeatedly per vendor and confirmed it, so a single parse is a sample, not a measurement. Every contract is parsed three times by each vendor and every question is asked three times against each parse. An arm is scored on nine replicates and 13,500 graded answers, and accuracy is the share of all nine judged correct.

The ± on the board is the sample standard deviation of accuracy across the nine, parser noise and reader noise together. Two arms within each other's spread are tied, not ranked. The agent answering every question is gpt-6-luna in every row, with a cache breakpoint between the contract and the question; version 1 used gpt-5.6-sol.

Upper and lower bounds for this task

Baseline and Best case are not products. Both are generated from the source file rather than read off the page, and both answer the identical questions through the identical agent, so the only thing that differs between them is how much of the markup survived to reach it.

• Best case: struck text kept and labelled ~~like this~~. What a parse that lost nothing would enable. Carried as oracle in the published run files.
• Baseline: the same text with every character unmarked. What happens when the strike is dropped entirely. Carried as blind in the published run files.

Both are rendered from the source file, so they are parsed once and asked three times. Together they set the scale. Gap closed is (arm − baseline) / (best case − baseline), which rescales accuracy onto the range a parser can actually occupy.

Why isn't the best case 100%?

Because a perfect parse does not make a perfect reader. That row hands the agent every deletion clearly labelled and it still gets 81 of 1,500 questions wrong. About half of those are simply wrong in a way the markup does not explain, and 23 where it answered from text that was visibly marked as struck. That residual is the agent's own error rate rather than any parser's, and it is the same agent in every row.

This is exactly why the board reports gap closed next to accuracy. The reachable range on these questions is 25.2% to 94.6%, not 0 to 100, so scoring 93.1% sits far closer to the best case than the raw number suggests. Reading accuracy against 100% would credit every parser for the questions no parser can lose, and blame it for the ones no parser can win.

Cost details

Each document is parsed three times per arm, and every parse is billed. The money columns on the board are per pass, one parse of the corpus and one answer per question, averaged over the threeparses, so parser $ per page and per correct answer mean the same thing here as on the version 1 board. Questions against an already-parsed document are free to the vendor. The contract sits at the front of the agent's prompt, identical across all questions on that parse, so only the first call pays to write the cache and the rest read it. Where the API reports usage it is taken from there (Datalab), where it bills in credits from credits consumed (Extend, Pulse), otherwise from the list price per page, and from token list prices for the frontier models. Reader and judge cost are kept separate from vendor cost and neither is in the parse column.

The parse cost column is the measured cost per 1,000 pages. Contracts here differ a lot in length, and vendors price by the page, so a per-document figure would mostly measure how long the contracts are.

Agent cost is excluded from the parse column and is not uniform across arms: a verbose parser costs more to read, independently of whether it read correctly. That cost is on the board as Total tokens / 1k correct: the input and output tokens the answering agent used across every question, divided by correct answers. Parser $ / 1k correct does the same for the parse bill, so a parser that is cheap per page but leads to wrong answers shows what it really costs.

Why three large document processing platforms could not be benchmarked

Amazon Textract, Azure Document Intelligence and Google Document AI are the document services most enterprises already pay for, so their absence needs an answer. It is not that they score badly. It is that none of them has anywhere to put the result: a strike is a property of a character, and in all three response schemas a character has no such property. A benchmark score would only restate what each vendor's own API reference already says, so we cite that instead. The schema field and its reference are on the board in [01] evidence.

Azure was run end to end before being ruled out, because a schema gap and a real output are different kinds of evidence. Asked for markdown, it returned the contract in full and marked none of the deletions.

best case  at least ~~ten three~~ five (5~~310~~) business daysazure      at least ten three five (5310) business days