Document Processing Benchmark

9 distinct cases make up the benchmark

Every task type is built from tracked changes in the source file, so every answer key is derived from the markup rather than annotated by hand. These distinct types test different capabilities: a deletion inside a number fails differently from a deleted clause.

• Numeral strike: a strike inside a numeral. Digits struck mid-number leave no word boundary to recover.
• Word-internal strike: a strike inside a word, which collapses into a different real word rather than into nonsense.
• Struck clause: a clause struck with nothing put back.
• Old and new side by side: superseded and replacement wording adjacent in the text, where the question is which one binds.
• Struck definition term: a term struck from a definition, which changes every downstream clause that relies on it.
• Struck table cell: a whole cell struck, where the loss is structural rather than textual.
• Mixed edits on a page: insertions and a deletion together, so the parser has to keep two kinds of markup apart.
• Proposal between rounds: what happened to a proposal across two drafts, which needs both documents read consistently.
• Stale cross-reference: renumbering that leaves a reference pointing at the wrong clause.

Source data and its transformation

Built from RedlineBench (CC-BY-4.0), a set of simulated contract negotiations marked up by trained attorneys representing each side. The tracked changes are genuine Word tracked changes and render identically to any other redline. The parties are fictional and the deals simulated; what is real is the markup, and the benchmark tests the rendering-and-parsing path, which is unaffected by whether the underlying deal was live.

1. Strip comments. The attorneys' margin comments are their own reasoning, and several state the answer outright. A parser that captures annotations would hand the agent the answer, so we would be scoring annotation capture instead of strike fidelity. Comment parts, relationships, content-type overrides and in-document anchors are all removed, and a check proves no comment text survives anywhere in the file.
2. Render. LibreOffice converts each stripped file to PDF. Tracked changes come through struck and coloured, the way Word draws them.
3. Read the changes. Every edited clause is sorted by the shape of the edit, taken straight from the source file. Types that span a page or two drafts also record where each clause landed once printed.
4. Write the questions. A model writes one question per edit from the resolved facts. It never sees any vendor's output, and it is asked to discard edits that do not change what the contract requires.
5. Check. Automatic checks reject any question whose right and wrong answers turn out to be the same, or that asks about the text as text rather than about what the contract requires.

The render to PDF tests the parser

In the Word file a deletion is structured: the text is explicitly labelled as removed. Print that file to PDF and the label is gone. All that remains is a line drawn across characters that are otherwise ordinary text. Whether the distinction survives is now entirely down to the parser.

How each parser was called

Every arm was given the same thing, the contract PDF, and asked for markdown. None was handed the source Word file, which would let it read the deletion out of OOXML instead of recovering it from the page. Where a vendor ships a first-party SDK it is used rather than raw HTTP, so the arm exercises the path that vendor's own documentation puts customers on. The exact interface and settings for each arm are in the endpoint column of the board, and each parser name there links to the vendor documentation for the call.

Tracked-change handling is the one setting that is not uniform, because it cannot be: Reducto, Extend and Pulse expose it as a flag, Datalab as a separate endpoint, and LlamaParse applies it without asking. Each arm runs it the way its vendor documents. The Datalab convert arm is the exception and is on the board deliberately: it is the same vendor with no tracked-change support at all, which is what the gap between the two Datalab rows is measuring.

The frontier arms have no such flag to set, because for a general model the prompt is the flag. They receive the identical PDF and one shared instruction that asks for a faithful markdown rendering.

That prompt in full, identical for every frontier arm:

Convert this document to clean, well-structured markdown. Reproduce the page faithfully, preserving headings, paragraphs, lists, tables and inline formatting. Output only the document content, with no commentary, preamble or code fences.

What it asks for is what a document parser sells: the page, in markdown, faithfully. Whether faithfully includes the deletions is the thing under test.

How an answer is marked

Most responses sort themselves, because every item carries the wrong answers as well as the right one. Where wording gets in the way (“the section labelled 2.2” against an expected “Section 2.2”), a model judge decides, so a parser is never marked down for the phrasing of an answer that was right. The judge is not told which parser produced the answer.

Upper and lower bounds for this task

Baseline and Best case are not products. Both are generated from the source file rather than read off the page, and both answer the identical questions through the identical agent, so the only thing that differs between them is how much of the markup survived to reach it.

• Best case: struck text kept and labelled ~~like this~~. What a parse that lost nothing would enable. Carried as oracle in the published run files.
• Baseline: the same text with every character unmarked. What happens when the strike is dropped entirely. Carried as blind in the published run files.

Together they set the scale. Gap closed is (arm − baseline) / (best case − baseline), which rescales accuracy onto the range a parser can actually occupy.

Why isn't the best case 100%?

Because a perfect parse does not make a perfect reader. That row hands the agent every deletion clearly labelled and it still gets 121 of 1,500 questions wrong. About half of those are simply wrong in a way the markup does not explain, and 45 where it answered from text that was visibly marked as struck. That residual is the agent's own error rate rather than any parser's, and it is the same agent in every row.

This is exactly why the board reports gap closed next to accuracy. The reachable range on these questions is 28.8% to 91.9%, not 0 to 100, so scoring 89.1% sits far closer to the best case than the raw number suggests. Reading accuracy against 100% would credit every parser for the questions no parser can lose, and blame it for the ones no parser can win.

Cost details

Each document is parsed once per arm and reused for every question against it. The contract sits at the front of the prompt, identical across all questions on that document, so only the first call pays to write the cache and the rest read it.

The parse cost column is the measured cost per 1,000 pages. Contracts here differ a lot in length, and vendors price by the page, so a per-document figure would mostly measure how long the contracts are.

Agent cost is excluded from the parse column and is not uniform across arms: a verbose parser costs more to read, independently of whether it read correctly. That cost is on the board as Total tokens / 1k correct: the input and output tokens the answering agent used across every question, divided by correct answers. Parser $ / 1k correct does the same for the parse bill, so a parser that is cheap per page but leads to wrong answers shows what it really costs.

Why three large document processing platforms could not be benchmarked

Amazon Textract, Azure Document Intelligence and Google Document AI are the document services most enterprises already pay for, so their absence needs an answer. It is not that they score badly. It is that none of them has anywhere to put the result: a strike is a property of a character, and in all three response schemas a character has no such property. A benchmark score would only restate what each vendor's own API reference already says, so we cite that instead. The schema field and its reference are on the board in [01] evidence.

Azure was run end to end before being ruled out, because a schema gap and a real output are different kinds of evidence. Asked for markdown, it returned the contract in full and marked none of the deletions.

best case  at least ~~ten three~~ five (5~~310~~) business daysazure      at least ten three five (5310) business days