ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

\providecommand \llamaindexbranded ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Boyang Zhang Adrian Lyjak Eli Stewart Zhaoqi Li Simon Suo {boyang, adrian, eli, zhaoqi, simon}@runllama.ai

1 Introduction

Benchmark Corpus Schemas Domains

Real docs

Long records

Scans / handwriting

Grounding Cost

Fixed KIE

SROIE [ 19 ] 1,000 fixed 1 ✓ \checkmark ∘ \circ ∘ \circ

DocILE [ 39 ] 6,680 fixed 1 ✓ \checkmark ∘ \circ ✓ \checkmark

RealKIE [ 42 ] 1,867 fixed 5 ✓ \checkmark ∘ \circ ∘ \circ

Schema guided

Contextual EB [ 12 ] 35 5 5 ✓ \checkmark ∘ \circ

Extend LongArray [ 9 ] 45 3 3 ✓ \checkmark

Micro1 LongExtract-50 [ 26 ] 50 per-doc 7 ✓ \checkmark ✓ \checkmark

VAREX [ 4 ] 1,798 per-doc 1

ExtractBench (ours) 370 67 8 ✓ \checkmark ✓ \checkmark ✓ \checkmark ✓ \checkmark ✓ \checkmark

✓ \checkmark = covered and scored ∘ \circ = partial or incidental coverage; blank means absent.

Table 1 : Comparison of representative fixed-ontology document-IE benchmarks (upper block) and modern schema-guided extraction benchmarks (lower block). ExtractBench is the only benchmark that jointly evaluates long-record completeness, real scans and handwriting, word- and page-level grounding, and measured cost. The full capability matrix appears in Table ˜ 7 ( Section ˜ A.4 ).

Until recently, extracting structured data from business documents was performed by humans: knowledge workers read financial filings, insurance claims, purchase orders, and government forms, then keyed the relevant values into a system of record for downstream workflows. This work is usually highly manual and repetitive, and any mistakes can be costly [ 18 , 5 ] . With the recent development of large language models and autonomous agents, we see fast-growing demand from enterprises to deploy agents to complete extraction-focused document workflows historically performed by humans.

Schema definition is at the center of the extraction workflow. A schema defines one extraction task, shared across all documents of the same type. For example, one invoice schema covers invoices from every vendor regardless of how different each invoice may look. Given that enterprises write a new schema for almost every new workflow, a system cannot be tuned to just one fixed template. We define the extraction task as schema-guided extraction (defined precisely in Section ˜ 2.1 ): given a document and a user-defined schema as input, the agent faithfully follows the schema to produce the correct output along with source evidence as grounding metadata.

In real-world use cases faced by enterprises, there are many sources of challenges and failure cases in an extraction workflow, with common ones including missing rows in long lists, selecting the wrong occurrence of a sparse fact, overfilling dense forms, and confusing similar dates, identifiers, or amounts. There are also particular challenges in accurately understanding the structure of the document, which we call perception challenges : scan or handwriting noise, hierarchical headers, cross-page continuation, and large or irregular tables. Length creates a separate problem: a system can read local values correctly but still truncate a long schedule.

Visual grounding and traceability are another critical element for making agent-powered extraction effective and reliable at production scale for enterprises. Given that there are always inevitable failures — such as when an agent fails in reconciling fund holdings because a long schedule is truncated and rows are missing from the output — it requires a human in the loop to use visual grounding signals to quickly identify and correct the issues. Additionally, the cost of extraction per page also matters at production volume [ 23 ] . In high-volume, document-intensive enterprise workflows, a cost difference of one cent per page may determine if an AI initiative is financially viable.

Although there have been attempts from multiple existing benchmarks to tackle this challenge, they all have critical limitations ( Table ˜ 1 ; Section ˜ 4 and Section ˜ A.4 give the detailed comparison). Classic information extraction filled fixed, hand-built templates with per-task systems [ 16 ] . Fixed KIE benchmarks, such as SROIE [ 19 ] and DocILE [ 39 ] , do not handle user-specified schemas. More recent schema-guided benchmarks [ 12 , 4 ] cover only a narrow dimension of the problem. The three closest benchmarks each cover one slice of these requirements: schema-conformant JSON against enterprise-scale schemas [ 12 ] , row completeness on synthetic rendered arrays [ 9 ] , and long statistical reports and filings [ 26 ] . None of the three measures cost, scores grounding, or contains a scanned or handwritten document. For example, Contextual AI’s ExtractBench [ 12 ] 1 1 1 The unrelated academic benchmark by Contextual AI shares the ExtractBench name [ 12 ] ; Section 4 details how the two differ in scope. does not cover any handwritten documents, nor does it take visual grounding or cost into consideration.

To bridge the gap, we introduce ExtractBench, a comprehensive benchmark for schema-guided enterprise document extraction that carries broad task coverage, evaluates traceability, and measures cost. ExtractBench contains 370 documents ( 4,869 pages) across 8 business domains and 67 document types. Each document type has one schema shared across its documents. Each document is tagged by task challenge, perception challenge, table structure, domain, and length ( Section ˜ 2.2 ). The benchmark is composed of real born-digital documents, synthetic long lists based on real layouts, and real regulatory and tax forms with schemas authored from blank templates. To establish high-quality ground truth at scale, we design a scalable pipeline: independent-system proposals are adjudicated for real documents, values are set before rendering for synthetic lists, and humans verify both values and grounding on scanned forms ( Section ˜ 2.3 ). We evaluate accuracy with order-insensitive value F1 over the values in the extracted JSON. To evaluate visual grounding ability, we also score whether a correct value points to its source for fields with human-verified boxes, so reviewers can audit the answer without searching the document ( Section ˜ 2.4 ).

We evaluate 14 frontier methods spanning commercial VLMs, open-source extraction, coding agents, and specialized APIs. We noticed significant performance variance in out-of-the-box frontier models across different challenge dimensions. For example, Gemini 3.5 Flash accuracy dropped significantly from 87.9% on short documents to 27.9% on long ones ( Section ˜ 3 ). LlamaExtract Agentic Plus shows much more consistent performance, with 96.6% on short and 94.4% on long documents. It also outperforms Codex GPT-5.5 (95.6% versus 93.6%) at a much lower cost (8.1 ¢/page versus 27.8 ¢/page). Additionally, commercial VLMs and coding agents do not return word-level boxes, so workflows that require source evidence need specialized extraction APIs ( Section ˜ 3.4 ).

Our contributions include:

• A challenge-tagged benchmark with broad coverage. 370 documents and 4,869 pages span 8 business domains and 67 document types, with tags for task challenge, perception challenge, table structure, domain, and length that support per-challenge analysis.

• A scalable pipeline for schema and ground-truth curation. To produce high-quality, well-specified schema–ground-truth pairs without labeling every field by hand, we combine frontier-model ensembles for real documents, programmatic generation for synthetic long lists, and human verification for scanned forms.

• A broad evaluation of frontier extraction methods. We compare 14 systems spanning commercial VLMs, OSS pipelines, coding agents, and specialized APIs, reporting accuracy, grounding, cost, and the quality–cost tradeoff.

2 ExtractBench

This section defines schema-guided extraction precisely, then describes how ExtractBench applies it to build a challenge-tagged corpus.

2.1 Task Definition

Given a document and a schema, a system returns structured data with evidence ( LABEL:fig:hero ):

f : ( document , schema ) ⟼ ( structured data , evidence ) . f:,(\text{document},\ \text{schema})\longmapsto(\text{structured data},\ \text{evidence}).

Input.

The input is a full document, born-digital or scanned, and a schema written by the user. The user specifies the extraction task via a schema: it lists the fields to extract, and each field has a name, a type, and a natural-language description of what belongs in it. It is expressed as a JSON Schema , the industry-standard way to specify structured output, and may contain scalar fields, nested objects, arrays of records, nullable fields, and value constraints. A schema defines one extraction task and guides all documents of the same type, even though the documents can vary significantly in structure, layout, and styling. For example, insurance claims from different providers vary greatly in length and layout. A document usually holds more than the schema asks for — and sometimes less: any field the document leaves unanswered must come back as null.

Output.

The output is a schema-valid JSON object, with the source page and a bounding box for each value as evidence . It must return correct, exhaustive values (including repeated records), correctly use null for absent information, and ground each extracted value. Section ˜ 2.4 makes these expectations precise.

2.2 Taxonomy and Coverage

Figure 1 : ExtractBench coverage across the five tag axes. Each bar is the share of the 370 documents carrying the tag, with its document count shown beside the bar. The task panel reports the three task challenges; a document tagged with several of one challenge’s sub-tags counts once. Tags may overlap across panels; Table ˜ 4 ( Section ˜ A.1 ) defines every tag and sub-tag.

Existing benchmarks for document extraction often report one aggregate score over a narrow set of document types. An aggregate score does not show whether a system missed a third of a list or got one label wrong, and it cannot distinguish a hard extraction task from a bad scan. ExtractBench instead tags each document along five independent axes: task challenge (what makes extraction hard), perception challenge (how the page was captured), table structure, length, and business domain. Because the axes are independent, a low score can be traced to its actual cause. Figure ˜ 1 shows how the corpus distributes over the five axes. The paragraphs below briefly explain each axis; Table ˜ 4 in Section ˜ A.1 gives additional details and representative document types for every tag.

Task challenges.

A task challenge defines the nature of the extraction task and what makes it difficult.

• T1: long-list completeness. Recover every record of a repeated structure that can span many pages. Typical failures are truncation, duplicated or merged rows, hallucinated records, and values attached to the wrong record.

• T2: needle-in-haystack. Find a small number of requested facts in a long document. T2 has few target records but many plausible mentions, only one of which is canonical; failures are missed targets, wrong occurrences, and unnormalized paraphrases. It is also the only task challenge that asks for far less than the document holds: a median of just 1.6 fields per page ( Section ˜ A.1.1 ).

• T3: dense documents. Fill many fields from a document dense with labels, blanks, checkboxes, handwriting, and scan artifacts. The characteristic failure is over-extraction, inventing a value for a field that is actually blank, compounded by missed checkboxes and mislabeled fields. A dense document also repeats identifiers, dates, and amounts of the same format, so a plausible value can end up in the wrong field. Dense forms are the most common case (T3.a); receipts, invoices, and regulatory filings belong here too. T3.e marks schemas with more than 150 leaf fields. It includes multi-page tax bundles whose schemas are large even when individual pages are not especially dense.

Perception challenges.

A perception challenge records how the page was captured. The tags are rotated or image-only capture (P1), scanned page images (P2), and handwriting (P3) ( Table ˜ 4 ). They are assigned independently of the task challenge, so the same extraction task can appear under more than one perception challenge.

Table structure.

Tables earn a dedicated axis for two reasons. First, most of the values enterprises extract live in tables, from holdings schedules to invoice line items. Second, tables fail in a way no other page element does: a complex table can be read correctly value by value and still be assembled into the wrong structure, a failure that the task and perception axes cannot isolate. The structure tags mark the layouts where this happens: merged or hierarchical headers (S1), a header that does not sit above its data (S2), a table that continues across pages (S3), a table beyond a thousand rows (S4), and a table packed inside a single cell (S5). Each layout has its own failure: a merged header attaches values to the wrong column, a pivoted header transposes the record, a cross-page table loses its continuation, a very large table stops early, and a packed cell comes back as one string instead of its fields.

Document length.

Documents fall into three length buckets: short (L1, up to 10 pages), medium (L2, 11 to 50), and long (L3, more than 50). Length gets its own axis for the same reason the other axes are separate: the same task challenge can appear at any length, and length adds a failure of its own, since a system can read every value on a page correctly and still stop before the end of a long schedule.

Business domains.

An enterprise extraction system needs to work across domains, for two reasons: teams want one system rather than a separate tool per document type, and businesses often do not control what arrives and must process whatever their customers, vendors, and regulators send. ExtractBench therefore spans 8 domains and 67 document types ( Table ˜ 4 ): finance and fund holdings (D1), energy-sector regulatory forms (D2), government procurement and customs (D3), auto valuation (D4), supply-chain and other transactional documents (D5), healthcare remittance (D6), legal and bankruptcy filings (D7), and real-estate closing disclosures (D8). Prior benchmarks for structured extraction usually cover fewer domains and a handful of real document types ( Table ˜ 6 ).

Figure 2 : How ground truth is constructed, one strip per source type. A card’s border marks who runs the step (blue: the extraction-model pool; violet: coding-agent pipeline code; amber: a human); the filled green card is the resulting ground truth, and orange dashes mark repair and disagreement loops.

2.3 Schema and Ground-Truth Construction

To properly evaluate a schema-guided extraction system, the extraction task itself needs to be well specified: each schema must be coherent with the documents it applies to, and every field must have a clear expected value. If a schema is poorly aligned with its document family, or its instructions leave the extraction goal ambiguous, errors can no longer be attributed and a low score may reflect a defect in the benchmark rather than in the system being tested.

Creating well-defined schemas and ground truth that corresponds to them is labor-intensive, particularly when documents are long and the data is dense. Checking every field by hand is prohibitive in time and cost at this scale, and using a single extractor’s output as ground truth would repeat its mistakes and bias the results toward that extractor. This motivates a scalable pipeline that produces high-quality schema and ground-truth pairs without fully manual annotation.

To this end, we combine three sources of documents, each annotated by the method that fits it: frontier-model ensembles for real documents, programmatic generation for synthetic long lists, and human labelers for scanned forms ( Figure ˜ 2 ). Real documents supply the layouts, scan noise, and domain range we want to test, but drawing a box on every one of their fields is prohibitively slow. Synthetic long lists cover documents too large to label by hand: thousands of similar records are slow to annotate and easy to misread. Scanned forms are real documents that need a person to decide each value, because handwriting is unclear and a mark can sit between two fields. Documents from these sources are also re-captured as degraded scans, which needs no new annotation: the values do not change, so the clean document’s ground truth carries over. We use this methodology to build ExtractBench. Section ˜ A.3 gives the full procedures.

Schemas.

A document type is a family of documents that carry the same kind of information — SEC 13F filings, utility bills, mortgage closing disclosures — however much their layouts differ. In ExtractBench, each document type has exactly one schema, shared by all of its documents. The schema takes the form a user writes in production: field names, types, and a natural-language description for each field ( Section ˜ 2.1 ). How each schema is authored depends on its source and is described with each pipeline below. Every field is written to have a deterministic expected value in the document, so the ground truth is the same no matter which system is being scored.

Real documents.

The schema is drafted from sample documents, then several extraction systems from different model and pipeline families run against this candidate schema ( Figure ˜ 2 , top strip): no single extractor is reliable enough, and same-family systems share mistakes. A value on which every system agrees, including null for absent fields, becomes candidate ground truth. Disagreements are classified by cause: if more than one reading of the field is defensible, the schema is at fault, and we tighten its description with aliases, format requirements, location hints, and do-not-confuse guidance until reruns converge; if only one reading is defensible, it is a model failure, and a reviewer settles the contested cells against the page.

Synthetic long lists.

We build each synthetic document backwards, data first and document second, so no human labeling is needed. From a real filing (a fund schedule, holdings register, or creditor matrix), we produce the records, parsed verbatim or generated in its style, and rendering code that a coding agent writes after studying the layout’s fonts, columns, and page chrome ( Figure ˜ 2 , middle strip); the family keeps the real filing’s schema. That code renders the records into a PDF closely matching the real one, with page breaks placed by measurement. Every value is known before the PDF exists, and its page and word-level box are read back from the render, so the ground truth stays exact however long the list grows. Mechanical checks catch style mismatches and clipping, and an extraction-system pool audits the finished document, its disagreements exposing rendering-code bugs that are fixed before the family ships.

Figure 3 : Overall unified value F1 versus mean document-level cost per page, pooled over every scored document. Marker shape and color follow the system grouping of Section ˜ 3.1 . The four open-weight pipelines have no vendor API price and are omitted. Per-length results appear in Section ˜ D.1 .

Specialized APIs

Coding Agents

OSS

Commercial VLM

Dimension

LE Agentic Plus

LE Agentic

LE Cost-Eff.

Datalab A+B

Extend Max

Reducto Deep

Codex GPT-5.5

CC Opus 4.8

Gemma4 26B

Qwen3.6 35B-A3B

NuExtract3

Lift 9B

GPT-5.4 Nano

Gemini 3.5 Flash

Overall

95.6

89.5

86.8

64.5

86.3

90.4

93.6

87.1

66.2

87.3

47.9

77.3

74.9

79.8

Document Length

L1 Short ( ≤ \leq 10 pp)

96.6

92.0

90.8

62.8

92.0

94.2

95.7

90.1

80.5

93.1

54.4

87.2

77.4

87.9

L2 Medium (11–50 pp)

93.3

85.4

80.1

73.8

78.8

80.5

91.2

79.2

40.5

84.8

39.3

62.6

76.4

69.8

L3 Long ( > > 50 pp)

94.4

78.6

69.2

40.5

51.3

92.0

78.9

88.1

12.2

26.8

8.9

25.3

35.8

27.9

Task Challenge

T1 Long-list completeness

96.1

85.9

81.8

80.2

87.2

94.8

91.7

93.6

51.1

79.0

31.8

68.6

72.2

78.8

T2 Needle-in-haystack

93.6

88.3

82.3

73.9

90.3

92.5

91.7

89.1

63.0

85.3

25.2

78.0

74.0

87.9

T3 Dense documents

95.5

92.1

90.5

54.4

85.7

87.5

95.4

82.4

76.8

93.1

58.7

82.9

76.4

80.5

Perception Challenge

P1 Rotated / image-only

95.9

88.2

85.0

78.9

89.0

93.9

81.0

91.2

66.5

86.8

28.9

80.7

64.7

88.6

P2 Scanned

93.9

89.8

87.8

47.6

80.9

81.1

93.4

74.2

69.1

92.0

62.6

76.0

67.4

71.1

P3 Handwriting

93.8

90.9

87.6

47.2

93.8

92.3

93.6

74.7

73.9

92.3

75.8

86.0

67.7

74.2

Table Structure

S1 Merged headers

94.5

78.5

79.4

82.8

91.0

94.2

95.0

94.0

44.6

81.8

33.3

69.9

68.9

80.2

S2 Pivoted / header not at top

95.0

86.1

87.2

84.4

91.4

95.3

94.9

94.3

63.0

89.6

20.9

76.4

77.7

88.4

S3 Cross-page table

95.8

84.3

79.0

78.5

85.1

94.4

89.4

92.5

40.5

73.8

37.6

64.5

72.3

73.6

S4 Enormous table

95.9

78.1

67.8

32.7

24.8

95.3

78.9

87.8

0.0

1.3

3.7

1.2

7.2

1.5

S5 Table within a cell

97.2

87.3

78.2

71.7

75.9

95.4

86.8

93.9

37.1

56.6

50.1

51.3

67.3

53.4

Business Domain

D1 Finance

96.3

91.3

87.2

62.1

79.2

85.1

96.2

84.7

59.5

85.9

48.4

71.8

77.5

75.4

D2 Energy

95.0

90.7

88.5

49.0

94.5

93.7

94.1

82.9

78.7

92.4

76.6

87.9

73.0

78.9

D3 Government

93.4

83.4

83.0

77.9

86.5

92.5

92.7

91.0

58.1

83.7

18.0

71.5

74.3

81.4

D4 Automotive

97.9

95.0

95.2

85.0

91.9

97.3

95.2

98.0

83.9

96.6

14.8

85.9

79.3

98.0

D5 Supply Chain

97.9

93.2

92.0

82.5

96.8

96.1

95.9

99.0

87.4

95.4

31.6

93.5

83.7

98.2

D6 Healthcare

92.6

74.1

70.7

77.9

93.3

90.3

82.8

95.1

39.4

69.0

34.3

61.8

55.2

74.5

D7 Legal

96.7

81.5

69.1

66.6

56.1

92.8

62.0

74.3

17.9

58.8

56.0

42.2

51.7

58.3

D8 Real Estate

94.0

93.7

93.0

73.8

93.2

95.9

93.8

93.4

90.8

93.6

37.8

89.8

85.8

94.7

Table 2 : Unified value F1 (%) by dimension and system. Models are columns, grouped by system type, and dimensions are rows. Within each row, bold and underlined mark the highest and second-highest scores. Red shading marks drops of more than 5/15/25 points from each system’s overall score (darker means larger). Overall aggregates each system’s evaluated documents. Table ˜ 14 ( Section ˜ D.2 ) reports every sub-tag.

Word-level grounding F1 Page-level grounding F1

System Overall Short Medium Long Overall Short Medium Long

LE Agentic Plus 46.4 43.7 54.0 54.7 84.9 89.7 72.2 87.1

LE Agentic 44.1 42.3 50.5 45.7 66.1 69.7 56.6 67.6

LE Cost-Eff. 40.4 40.2 42.3 36.7 64.2 68.9 53.7 56.5

Datalab A+B 2.0 2.7 0.2 0.0 48.5 56.9 38.6 0.0

Extend Max 25.1 33.9 0.2 0.0 48.9 61.7 27.7 0.0

Reducto Deep 43.3 42.8 45.6 41.1 71.7 72.6 70.4 67.3

All other systems 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Table 3: Grounding score (%). Word-level grounding F1 requires a correct value and word-level box at IoU 0.5; page-level grounding F1 requires a correct value and the source page. The final row spans all systems not named above. Bold marks each column’s best value, underlined the second best. LE Agentic and LE Cost-Effective return word-level boxes only when the caller enables the granular bounding-box option; both are run with it on.

Scanned forms.

Scanned forms are the one source where a person checks every field ( Figure ˜ 2 , bottom strip). The schema is authored against the blank form template and frozen before any document is labeled. An ensemble of up to five systems votes on every schema leaf; contested votes go to an adjudication agent that must inspect the page before ruling. A designated pipeline proposes a box per field, and a human annotator accepts, edits, nulls, or redraws each one. This yields 169 human-verified documents, with 84% of verified fields carrying a human-placed box; the rest are mostly blank fields, with nothing on the page to box.

The three pipelines back their ground truth differently: real-document values are confirmed by agreement across independent systems, synthetic values and boxes are exact by construction, and form values and boxes are checked by a person. This determines which metrics each document supports: values are scored everywhere, box-level grounding only where the boxes are verified ( Section ˜ 2.4 ).

2.4 Metrics

ExtractBench measures two things. Value accuracy asks whether a system returned the right values, and is scored with the unified value F1 on every document. Grounding asks whether the system can show where each value came from, and is scored only on documents whose box ground truth is verified ( Section ˜ 2.3 ).

Value accuracy.

The unified value F1 scores whether the extracted values match the expected output, under one definition for scalar fields and arrays of records. Each output is flattened into cells, one per scalar field and per aligned record subfield, and a cell is correct when it matches its expected counterpart after normalization. Precision, recall, and F1 are computed over these cells per document, and slices report unweighted document means ( Appendix ˜ B gives the exact scoring rules).

• Array alignment. A repeated structure is compared as an unordered set of records: records are paired by the Hungarian algorithm to minimize mismatched cells, following how prior extraction benchmarks align line items and arrays [ 39 ] . Unmatched expected records lower recall; extra predictions lower precision.

• Normalization. Values are normalized before comparison: dates to ISO format, strings by collapsing whitespace; everything else requires exact equality, with no numeric tolerance and no LLM judge. The few exceptions are listed in Section ˜ B.1 .

• Missing values. An omitted key scores as an explicit null, so every scalar field counts toward both precision and recall, and a correct null on a blank field is credited ( Table ˜ 9 lists every case). Only repeated records move precision and recall apart, so a gap between them points to dropped or extra records rather than wrong values ( Section ˜ D.1 tabulates both per system and length slice).

Grounding.

For fields with a verified ground-truth box, ExtractBench also reports word-level grounding precision, recall, and F1. A field counts as grounded only when its value is correct and its predicted box overlaps an accepted box for that field, at a fixed IoU threshold of 0.5: a well-placed box around a wrong value earns no credit. Page-level grounding F1 asks the weaker version of the same question, requiring only the correct source page rather than a box, which many systems satisfy even when they return no boxes at all. Section ˜ B.4 gives the details.

3 Experiments

3.1 Setup

We evaluate 14 extraction systems 2 2 2 Models and prices reflect those available as of July 1, 2026. across three high-level approaches:

• VLMs treat extraction as direct multimodal generation: they receive the document and schema and generate structured output in a single model call. We evaluate GPT-5.4 Nano [ 29 ] and Google Gemini 3.5 Flash [ 13 ] , called through constrained structured-output APIs; and Lift 9B [ 7 ] , NuExtract3 [ 27 ] , Qwen3.6 35B-A3B [ 34 ] , and Gemma4 26B [ 14 ] , which are self-hosted.

• Coding agents extract through an iterative tool-use loop: they can inspect the document, write and run parsing code, validate results, and revise the final output. We evaluate Claude Code Opus 4.8 [ 2 ] and Codex GPT-5.5 [ 30 ] , which receive the document and schema with filesystem and tool access (tool configuration in Section ˜ C.1 ).

• Specialized APIs provide a managed document workflow that handles preprocessing, parsing, and schema-guided extraction, sometimes with source grounding. We evaluate Reducto Deep Extract [ 36 ] , Extend Max Context [ 10 ] , Datalab Accurate Parse + Balanced Extract [ 6 , 8 ] , and three LlamaExtract tiers [ 24 ] (Cost-Effective, Agentic, and Agentic Plus).

All systems receive the same document–schema pairs and are evaluated without benchmark-specific tuning; all runs took place in June–July 2026. We report unweighted document-level means for value F1, word- and page-level grounding F1, and cost, using the metrics defined in Section ˜ 2.4 . Per-page costs apply published provider rates to actual token or credit consumption ( Section ˜ C.2 ). Because the four self-hosted VLMs have no directly comparable API price, we omit them from cost comparisons.

3.2 Quality–Cost Frontier

Enterprise extraction workloads often span millions of pages, making per-page cost differences substantial. At one million pages, each cent per page adds $10,000. Figure ˜ 3 compares overall value F1 with measured per-page cost.

The evaluated system families occupy distinct regions of this tradeoff. The VLMs with reported costs occupy the low-cost region ( ≤ \leq 1.0 ¢/page), but neither exceeds 80% F1. Coding agents reach 87.1% and 93.6% F1, but cost 16.2 and 27.8 ¢/page. Specialized APIs span a much wider range. Within this group, LlamaExtract traces the quality–cost frontier: Cost-Effective reaches 86.8% F1 at 1.0 ¢/page, Agentic reaches 89.5% at 3.1 ¢/page, and Agentic Plus reaches 95.6% at 8.1 ¢/page. Agentic Plus outperforms both coding agents while costing no more than half as much. These results show why extraction quality and cost must be evaluated jointly: greater spending does not necessarily produce greater accuracy. A broader comparison of commercial VLMs is provided in Section ˜ D.4 .

3.3 Results Across Dimensions

Overall F1 makes it easy to compare systems, but a single aggregate score cannot show which document characteristics drive their successes and failures. To expose these failure modes, we use ExtractBench’s challenge tags to break down performance across five axes: document length, task challenge, perception challenge, table structure, and business domain ( Table ˜ 2 ).

Document length.

Most systems perform well on short documents, with more than half scoring above 90%, but the spread widens as documents grow longer. On long documents the commercial VLMs fall below 40%, while Claude Code Opus 4.8 (88.1%) and Reducto Deep Extract (92.0%) remain close to their short-document scores. LlamaExtract Agentic Plus is the strongest across all three lengths and never drops below 90% (96.6/93.3/94.4). The long-document failure is concentrated in recall: entire records are dropped rather than misread ( Section ˜ D.1 reports precision and recall separately). We attribute this to context limits: most systems cannot work through a long document in a single pass, and those without a strategy for iterating over it stop early, truncating the remaining records.

Task challenge.

Long-list completeness (T1) and needle-in-haystack (T2) mirror the document-length results: LlamaExtract Agentic Plus and Reducto Deep Extract are the top two systems on both challenges. Dense documents (T3) reorder the ranking. Reducto Deep Extract drops to 87.5%, Claude Code Opus 4.8 to 82.4%, and Datalab Accurate Parse + Balanced Extract to 54.4%. The T3 scores combine several sources of difficulty: form layout, document classification, reviewer annotations, and schema size. LlamaExtract Agentic Plus (95.5%) and Codex GPT-5.5 (95.4%) lead on this challenge. T3.e is the 35-document subset with schemas above 150 leaf fields. Section ˜ D.2 reports its scores and describes which pipelines rejected these documents. Section ˜ D.3 reports pipeline-level success rates and failure causes. Qualitative examples for T1–T3 are provided in Appendix ˜ E .

Perception challenge.

The perception axis exposes system-specific blind spots. Codex GPT-5.5 handles rotated or image-only capture (P1) poorly, dropping to 81.0% from 93.4%–93.6% on the other perception challenges. Reducto Deep Extract shows the complementary weakness: it stays above 90% on rotated or image-only capture and on handwriting (P3), but falls to 81.1% on scanned pages (P2). Qwen3.6 35B-A3B is stronger on scanned pages and handwriting than the other VLMs, scoring above 92% on both. LlamaExtract Agentic Plus remains the strongest system across all three perception challenges.

Table structure.

Enormous tables (S4, beyond a thousand rows) produce the sharpest separation in Table ˜ 2 . Most systems stop early and return only a small fraction of the records: every VLM scores below 10% on this slice, and Datalab Accurate Parse + Balanced Extract (32.7%) and Extend Max Context (24.8%) also fall sharply. By contrast, LlamaExtract Agentic Plus (95.9%), Reducto Deep Extract (95.3%), and Claude Code Opus 4.8 (87.8%) are the top three systems. Cross-page tables (S3) pose a milder version of the same failure, where the difficulty is carrying the table structure across page breaks. Pivoted layouts (S2) are the least discriminative of the structure slices, because most leading systems handle them well.

Business domains.

Domain difficulty largely reflects the mix of task challenges inside each domain ( Table ˜ 2 ). Finance (D1) and government (D3) carry the long-list (T1) and needle-in-haystack (T2) tasks; energy (D2) is dominated by scanned dense forms (T3); and legal (D7) and healthcare (D6) hold long record lists, including creditor matrices, sanctions lists, and clinical event logs. A domain’s score is therefore mostly a reweighting of its task-challenge results, and we read the domain axis as a check on coverage rather than as an independent source of difficulty.

3.4 The Grounding Gap

Grounding makes extraction auditable by letting a reviewer trace each predicted value back to its source. ExtractBench measures this capability explicitly, whereas existing schema-guided extraction benchmarks overlook it ( Table ˜ 1 ). We consider a field grounded only when both the extracted value and its citation are correct, at the page- or word-level. We highlight the grounding gap in Table ˜ 3 .

• VLMs and coding agents do not return evidence by default; they therefore score zero at both grounding levels. Users who need auditable outputs must add a separate evidence-localization component or use an extraction API with grounding built in.

• Granularity challenge. Locating the exact word is much more challenging than finding the correct page. LlamaExtract Agentic Plus achieves 84.9% page-level grounding F1 but only 46.4% word-level F1. Datalab shows a larger gap, at 48.5% versus 2.0%. Page evidence narrows the search, but still leaves reviewers to locate the value among similar candidates.

• Robustness to length. Extend Max Context falls from 61.7% page-level grounding F1 on short documents to 0.0% on long documents, and Datalab follows the same pattern. Reducto Deep Extract is more stable, declining from 72.6% to 67.3%, while LlamaExtract Agentic Plus remains the strongest system overall.

Even the best overall word-level grounding F1 is only 46.4%. Systems are increasingly capable of extracting values and often identifying their source pages, but reliably connecting each value to its exact supporting evidence remains an open problem.

4 Related Work

4.1 Benchmarks for Document Extraction

Document extraction benchmarks cover two main settings: fixed-ontology extraction, where fields are defined in advance, and schema-guided extraction , where the user supplies a schema defining the fields and output structure at inference time.

Fixed-ontology document IE.

Fixed-ontology benchmarks ask how reliably a system can recover a known set of fields as the documents become more challenging. They have progressively expanded document diversity from forms and receipts to multilingual layouts and enterprise domains [ 21 , 19 , 33 , 44 , 42 ] . Other work increases structural and contextual complexity through line items, tables, long documents, and unfamiliar templates [ 43 , 39 , 40 , 17 , 20 ] . Some benchmarks additionally annotate spatial positions or study localization [ 39 , 41 ] . Across these settings, however, the target fields remain fixed by the benchmark: they test robustness within a known ontology rather than whether a system can follow a new user-supplied schema.

Schema-guided extraction benchmarks.

Recent document extraction benchmarks have focused on the schema-guided setting, where users specify the extraction task at inference time without retraining the system [ 12 , 38 , 22 ] . Work in this setting has progressively increased task scale and complexity, testing more complex schemas, longer documents, and larger outputs. ContextualAI’s ExtractBench [ 12 ] stresses schema complexity, with schemas containing up to 369 fields, but evaluates only five shared schemas; LongExtractBench-50 [ 26 ] and VAREX [ 4 ] use a different schema for every document, so they do not test whether one extraction task transfers across diverse document appearances. LongArray-Extract [ 9 ] and LongExtractBench-50 [ 26 ] instead stress completeness over long documents and repeated records, though their public test sets contain only dozens of documents. Several benchmarks [ 4 , 22 , 9 ] use synthetic construction to scale these evaluations, but their generated documents do not capture the visual variability and perception challenges found in real enterprise data. This fragmented coverage makes it difficult to compare system families comprehensively or diagnose why they fail.

To our knowledge, ExtractBench provides the broadest combined coverage of these dimensions, spanning real document families and targeted synthetic stress tests across 8 business domains ( Table ˜ 1 ). ExtractBench is designed around production requirements at scale, jointly measuring value accuracy, source grounding, and per-page cost to capture whether outputs are correct, traceable, and economical to produce. We also stratify the dataset with challenge tags to ensure coverage across task and perception difficulties and diagnose where different system families fail.

4.2 Methods for Document Extraction

Modern schema-guided extraction systems fall into three broad families: general-purpose vision-language models that generate outputs directly, coding agents that inspect documents iteratively with tools, and specialized extraction systems designed around document processing workflows. These approaches make different tradeoffs in completeness, visual robustness, grounding, and cost.

Vision-language models.

General-purpose vision-language models are multimodal reasoners that accept text and images and generate flexible outputs. Document extraction can therefore be reformulated as multimodal generation: document pages are rendered as images, the schema is expressed as text instructions, and the model returns extracted values either as prompted text or as schema-compliant output enforced through a structured-output API. Systems in this family include closed general-purpose models [ 29 , 13 ] , general-purpose open-weight models [ 34 , 14 ] , and models tuned specifically for extraction [ 7 , 27 ] . All three groups accept new schemas without task-specific retraining. This direct, one-pass workflow is simple and efficient, but can miss values in long repeated structures, and the systems evaluated here do not return source evidence.

Coding agents.

Coding agents such as Codex [ 28 , 30 ] and Claude Code [ 1 , 2 ] approach extraction through an iterative tool-use loop: given the document and schema as files, they can inspect pages, write parsing code, run checks, and revise the final JSON. This loop is more flexible than one-pass generation and can help with long documents or repeated records, where the agent can revisit the document rather than rely on a single model response. The same flexibility creates cost and reliability risks: even short documents may trigger many inspection, coding, debugging, and validation steps, and unconstrained agents can run for many steps before producing a small extraction. Coding agents also require an agent runtime with filesystem access and validation, depend on how documents are rendered and which tools are available, and are not designed around extraction-specific grounding metadata.

Specialized document extractors.

Commercial platforms [ 36 , 10 , 6 , 24 ] expose extraction as a purpose-built API rather than a raw model prompt or coding environment. As managed services, they let users upload files directly and handle format support, preprocessing, document parsing, and schema-guided extraction with little configuration. They can also expose visual grounding and other extraction metadata, such as source pages and, in some cases, word-level boxes, making outputs easier to audit than raw model responses. However, specialized APIs still differ substantially in completeness, robustness, grounding quality, and cost.

5 Conclusion

We introduced ExtractBench, a challenge-tagged benchmark that brings the core requirements of real schema-guided extraction into one evaluation: correct and complete outputs, source traceability, robustness across document challenges, and cost at scale. By measuring these dimensions together, ExtractBench shows not only which systems perform well, but where and why they fail.

Direct VLM extraction is inexpensive but often truncates long record lists; coding agents are more robust on these workloads but substantially more expensive. Specialized APIs span the quality–cost frontier, with LlamaExtract Agentic Plus achieving the strongest overall performance at a lower cost than the coding agents. Challenge-tagged results further show that systems degrade differently on long documents, dense schemas, scans, handwriting, and enormous tables.

Grounding remains the clearest area for improvement. The evaluated VLMs and coding agents do not return source evidence by default, while word-level grounding F1 remains at 46.4% even for specialized systems that return boxes. Together, these results set a clear bar for reliable extraction: complete outputs, traceable evidence, and sustainable cost at scale.