Evidence-Grounded Constraint Checking in Construction Documents

Evidence-Grounded Constraint Checking in Construction Documents

Rashid Mushkani

Hugo Bérard

Shin Koseki

Abstract Professional-document review is a constraint-checking problem in which the answer depends on relations among text, geometry, pages, and document revisions. We present an evidence-grounded pipeline that normalizes extracted facts, executes four-state rules deterministically, retains source spans, and escalates unresolved cases. We evaluate its PDF evidence allocator on 160 expert-referenced tasks from 29 construction projects, with a repeated four-system test and a disjoint two-system breadth extension. In the repeated test, reallocating a four-image cap from retrieved page overviews to one overview plus three overlapping tiles improves project–family standardized decision accuracy by 10.6 percentage points (95% project-cluster bootstrap CI 4.3–18.0; exact p = .031 p=.031 ). The contrast does not persist in the larger breadth block: Region-RAG changes accuracy by -4.1 points (95% CI -10.2–1.9; exact p = .209 p=.209 ), and the equal-image sensitivity favors page breadth. Exact finding localization remains low, false passes remain common, and repeated-run agreement is poorly calibrated. The central result is therefore not a universal advantage for tiling, but a measured resolution–breadth tradeoff that requires rule- and evidence-aware routing before expert review. multimodal document understanding, constraint checking, construction documents, evidence grounding, selective prediction

1 Introduction

Reviewing a construction set requires relations that an unordered text stream does not preserve. A detail callout is meaningful only at the object where its leader terminates; a sheet reference is valid only if its destination exists; and a product submittal satisfies a specification only when values agree across documents. These are typed constraints over text, geometry, document structure, and revision state. Retrieval can expose the right terms while still losing the relation needed for a correct professional judgment.

This paper studies a general machine-learning problem: evidence-grounded constraint checking in multimodal professional documents . Construction documents provide a demanding testbed because pages are large, evidence is sparse, and an unchecked false pass can be more consequential than a flagged case. We use AEC-Bench (Mankodiya et al. , 2026 ) and score predictions against a typed inventory of its released expert-generated references. The primary endpoint is a four-way decision— Meets , Does Not Meet , Missing Information , or Uncertain —and the secondary endpoints include finding precision and recall, exact task correctness, false-pass and false-fail rates, calibration, and selective risk.

We implement an evidence-grounded constraint checker with four components: source adapters, typed facts, deterministic rules, and proof traces. The PDF adapter combines layout-preserving text retrieval with bounded visual regions. CAD and IFC use the same source-span interface but are not part of the present empirical data. A deterministic reducer converts extracted fact states into a decision, and a fail-closed policy identifies missing, uncertain, low-confidence, or untraced facts for expert review.

Our central test is deliberately narrow and falsifiable: under the same text retrieval and a four-image cap, does allocating visual capacity to local detail improve expert-reference decisions relative to spreading that capacity across retrieved page overviews? The repeated test contains 40 tasks from 6 projects, four Google/Azure systems, three evidence policies, and five calls per cell (2,400 trials). A breadth extension adds 120 tasks from 23 disjoint projects for two Google systems. Independent inference is performed at the source-project level.

The contributions are:

• a typed constraint-checking architecture with deterministic reduction, source-level evidence contracts, and explicit expert escalation;

• a machine-readable expert-reference suite of 160 tasks in seven constraint families and 29 source projects; and

• a controlled evidence-allocation experiment where a 10.6-point repeated-test gain changes to -4.1 points on 23 disjoint projects, exposing a resolution–breadth tradeoff.

2 Related Work

Multimodal documents.

DocVQA established visual question answering over document images (Mathew et al. , 2021 ) ; MMLongBench-Doc studies long documents containing text and graphics (Ma et al. , 2024 ) ; and M3DocRAG and ColPali examine multimodal retrieval for multi-page reasoning (Cho et al. , 2024 ; Faysse et al. , 2025 ) . Layout-aware encoders and OCR-free document models learn page structure from pixels and text (Huang et al. , 2022 ; Kim et al. , 2022 ) . These settings make retrieval or representation part of the evaluated system. Our focus is narrower than general document QA: the output is a typed constraint decision with asymmetric safety errors, explicit source provenance, and source-project grouped evaluation.

Multimodal retrieval work typically optimizes which pages enter a long-context answering model. Our intervention addresses a subsequent allocation problem: given a small image budget and a fixed ranked text set, should the packet preserve page breadth or spend capacity on local resolution? The distinction matters for large-format drawings, where a page overview can preserve global layout while making dimensions, leaders, and callout text unreadable. We hold retrieval fixed to isolate this resolution–breadth choice.

Construction-document understanding.

AEC-Bench evaluates executable workflows over drawings, specifications, and submittals (Mankodiya et al. , 2026 ) . DrawingVQA evaluates expert-authored questions over issued construction drawings, including resolution and text-only ablations (Jung et al. , 2026 ) . We retain the former’s diverse professional tasks but restrict the analysis to seven families that express a constraint judgment, and score generated findings against its expert-generated reference inventory. Earlier automated checking systems compile building rules against structured BIM representations (Eastman et al. , 2009 ; Solihin and Eastman, 2015 ) . Our architecture retains deterministic execution and auditable rule states while admitting facts extracted from PDFs as well as CAD and IFC sources.

Structured BIM checking assumes that relevant entities and properties already exist in a queryable model. PDF sets invert that premise: the checker must recover entities and relations from sparse text and graphics before applying a rule. This motivates our separation between a learned fact-extraction boundary and a deterministic rule boundary. It also permits the same rule and trace objects to accept future CAD/IFC facts without treating untested modalities as part of the present result.

Tool use, abstention, and measurement.

Tool-using language-model systems combine retrieval, actions, and generation (Yao et al. , 2023 ) ; their observed performance therefore depends on the complete inference pipeline. Selective prediction asks a system to trade coverage for lower risk (Geifman and El-Yaniv, 2017 ) , while calibration asks whether confidence corresponds to correctness (Guo et al. , 2017 ) . In professional review, both require validation at the level of independent projects rather than treating repeated calls as new cases. Our analysis reports these quantities but does not infer expert agreement, which the source release does not provide.

Repeated sampling is sometimes used as a proxy for confidence in generative systems. For constraint checking, identical answers can reflect a stable mistake, so repetition supplies an empirical class frequency rather than a guaranteed probability of correctness. We measure its calibration and retained risk directly, with task repetitions nested inside source projects.

3 Evidence-Grounded Constraint Checking

3.1 Problem formulation

Let a task provide documents D D , a constraint rule r r , and extracted facts F = { f j } F={f{j}} . Each fact has a typed predicate, subject, expected and observed values, state s j ∈ { verified , violated , missing , uncertain } s{j}\in{\text{verified},\text{violated},\text{missing},\text{uncertain}} , confidence, and one or more source references. A source reference identifies a PDF page and normalized region, a CAD entity, or an IFC entity and property path.

Let V V , M M , and U U denote the presence of a violated, missing, or uncertain fact. For a conjunctive review rule, the deterministic reducer is

g ​ ( F ) = { DNM , V , MI , ¬ V ∧ M , U , ¬ V ∧ ¬ M ∧ U , M , otherwise , g(F)=\begin{cases}\mathrm{DNM},&V,\ \mathrm{MI},&\neg V\land M,\ \mathrm{U},&\neg V\land\neg M\land U,\ \mathrm{M},&\text{otherwise},\end{cases} (1)

where DNM, MI, U, and M correspond to Does Not Meet , Missing Information , Uncertain , and Meets . This precedence is intentionally fail-closed: a supported violation is not overridden by absent evidence, and incomplete evidence cannot become a pass.

3.2 Source and fact contracts

The source layer separates document access from rule execution. Every source reference contains an immutable document identifier and a short excerpt. PDF references additionally require a one-indexed page and may carry a normalized bounding box. CAD and IFC references require an entity identifier and may carry a field or property path. Invalid page indices, degenerate regions, and unresolved entity references fail schema validation before a rule is run.

An extractor maps source evidence to typed facts f j = ( q j , o j , e j , s j , c j , Z j ) f{j}=(q{j},o{j},e{j},s{j},c{j},Z{j}) : predicate q j q{j} , observed value o j o{j} , expected value e j e{j} , state s j s{j} , confidence c j ∈ [ 0 , 1 ] c{j}\in[0,1] , and source set Z j Z_{j} . The rule declares the fact identifiers it requires. A verified or violated fact must cite at least one source; a required fact absent from the extractor output is inserted as missing . Duplicate fact identifiers are rejected. These invariants make extraction errors visible to the rule layer instead of silently converting them into successful checks.

Algorithm 1 Fail-closed constraint reduction

0: required fact IDs R R , extracted facts F F , threshold τ \tau

1: for each r ∈ R r\in R in rule order do

2: if r ∉ F r\notin F then

3: insert ( r , missing , 1.0 , ∅ ) (r,\texttt{missing},1.0,\varnothing)

4: end if

5: validate type, confidence, and source contract

6: end for

7: if any required state is violated then

8: d ← Does Not Meet d\leftarrow\textsc{Does Not Meet}

9: else if any required state is missing then

10: d ← Missing Information d\leftarrow\textsc{Missing Information}

11: else if any required state is uncertain then

12: d ← Uncertain d\leftarrow\textsc{Uncertain}

13: else

14: d ← Meets d\leftarrow\textsc{Meets}

15: end if

16: emit ordered fact steps, sources, and completeness

17: escalate if d d is unresolved, min j ⁡ c j < τ \min{j}c{j}<\tau , or a source is absent

18: return decision d d , proof trace, escalation reasons

The reducer is deliberately small: the same facts always yield the same decision, and every branch can be inspected without another generative pass. The present implementation evaluates conjunctions of required facts. More complex professional rules can expose disjunctions or exceptions as typed predicates upstream; evaluating that compilation is outside the current experiment.

3.3 Machine-readable rule families

Each family definition contains a stable rule identifier, a natural-language statement for authoring and review, a fact type, and the subject fields needed to bind a task instance. Reference-resolution rules bind a source region, target sheet, and target view; title and callout rules bind text to depicted content or a leader endpoint; index rules bind an index entry to a title block. Specification and submittal rules bind required and observed values to clause and drawing sources. The rule files also version the four decision labels and state precedence in Equation 1 .

This representation separates three operations that are often collapsed into one prompt. Authoring defines what must be true and which facts are required. Extraction populates those facts from document evidence. Execution applies the fixed state algebra. The distinction is material for missing information: failure to extract a required fact is represented as missing , whereas a source-supported contradiction is violated . Neither can be converted to Meets by fluent explanatory text.

Rules are stored independently from task references and model outputs. A task adapter instantiates subjects and expected values, while source hashes identify the documents against which the rule was evaluated. This permits a rule to be reviewed or revised without changing the source adapter, and it permits the same executor to consume facts from PDF, CAD, or IFC extraction. The seven family definitions used here are listed in Appendix A ; only their PDF instantiations are evaluated.

PDF pages evaluated CAD / IFC adapter interfaces Typed extraction text spans + visual regions Fact graph predicate, values, state, source Deterministic rule four-state reduction Decision Meets / Does Not Meet Missing / Uncertain Proof trace rule steps + source spans Escalation confidence + completeness Expert review when escalated Figure 1: Constraint-checking architecture. Solid paths are instantiated for PDFs in the experiment; the dashed CAD/IFC path is an adapter contract. The rule engine emits both a decision and an auditable trace. The benchmark lacks region-level reference annotations, so the experiment evaluates decisions and finding content, not trace localization accuracy.

3.4 PDF evidence allocation

Layout-preserving text is extracted deterministically and indexed with BM25 (Robertson and Zaragoza, 2009 ) . The query is the task instruction with procedural examples removed. Up to 12 page excerpts are retained, with document-balanced retrieval for tasks containing at most four files. Images are rendered with a maximum edge of 1,800 pixels.

The retriever indexes pages rather than arbitrary token windows so that every hit has a stable page reference. Explicit page numbers in the task instruction are inserted ahead of ranked candidates. For sets with at most four files, each file contributes candidates before the global top-12 truncation; this prevents a long specification or drawing set from suppressing every other document. Each excerpt records its document ID, page, score, extraction method, and content hash. These records are identical across the paired visual conditions.

The page-RAG policy uses up to four highest-ranked distinct page overviews. The region-RAG policy uses the same ordered text excerpts, one overview of the highest-ranked page, and three overlapping half-span crops of that page’s ink bounding box. The crops begin at offsets 0, .25, and .5 along the longer content axis and are rendered from a 3,000-pixel source. Both policies have a four-image cap; short documents can leave page-RAG slots unused. A sensitivity analysis restricts the comparison to tasks where both policies attach four images.

The crop axis is selected from the rendered page’s content aspect ratio, not from a task-specific annotation. Overlap avoids splitting a leader, note, or detail exactly at a crop boundary. No expert reference, defect region, or variant label enters retrieval or tiling. The intervention therefore tests a generic allocation rule rather than an oracle crop around the answer. Shared task instruction + ordered BM25 text excerpts Page-RAG Region-RAG P 1 P{1} P 2 P{2} P 3 P{3} P 4 P{4} P 1 P{1} R 1 R{1} R 2 R{2} R 3 R{3} up to four distinct retrieved pages top-page overview + three overlapping ink-region crops Figure 2: Controlled evidence-allocation intervention. Text evidence and response logic are fixed; only the four visual attachment slots differ.

3.5 Typed output, trace, and escalation

The model returns the public task’s typed records. A field-strict parser rejects missing, extra, null, or empty fields. The deterministic rule layer then maps records to Equation 1 ; it does not ask a second model to revise the answer. A proof-trace object records the rule, ordered fact states, confidence, and source references. Supported verified or violated facts fail validation when no source is attached. The present task outputs do not contain finding-level source references, so trace generation is an implemented contract rather than an evaluated outcome.

Formally, the trace for rule r r is π r = ( r , d r , { ( f j , s j , c j , Z j ) } j ∈ R r ) \pi{r}=(r,d{r},{(f{j},s{j},c{j},Z{j})}{j\in R{r}}) . It is complete only when every required fact is present, every fact has a source, and no state is uncertain . Cross-document facts may cite multiple source references; for example, a specification–drawing comparison can attach one clause span and one drawing region to the same observed conflict. The trace therefore preserves the evidence used by the reducer rather than a free-form rationale generated after the decision.

The escalation interface flags uncertain decisions, missing information, facts below a confidence threshold, and facts without sources. In the repeated experiment, the available empirical confidence is the class frequency across five calls. We evaluate whether a threshold of .8 lowers retained risk rather than assuming that repeated agreement is calibrated.

Escalation is kept separate from the rule result. With threshold τ \tau , the default action is

a r = 𝕀 [ d r ∈ { MI , U } ∨ min j ∈ R r c j < τ ∨ ∃ j : | Z j | = 0 ] . a{r}=\mathbb{I}!\left[d{r}\in{\mathrm{MI},\mathrm{U}};\lor;\min{j\in R{r}}c{j}<\tau;\lor;\exists j:|Z{j}|=0\right]. (2)

Thus a low-confidence supported violation remains Does Not Meet but is also queued for review; escalation never rewrites it as a pass. The rule result, trace completeness, and review action can consequently be audited as three distinct outputs.

4 Evaluation

4.1 Expert-reference constraint suite

We include all 160 tasks in seven AEC-Bench families that require a compliance, consistency, or defect judgment: reference resolution, technical detail review, title accuracy, callout accuracy, sheet-index consistency, specification–drawing synchronization, and submittal review. Retrieval-only navigation and tracing tasks are excluded before analysis. The resulting suite contains 29 source projects and 160 task-level references. The dataset card describes the annotations as expert-generated; no new labels or inter-annotator agreement measurements are introduced here.

The suite is organized by source project, not by page or task variant. Its repeated-test, development, and validation partitions contain 40, 74, and 46 tasks from 6, 18, and 5 disjoint projects. Across all partitions, the reference decisions are 126 Does Not Meet , 26 Meets , and eight Missing Information . The Uncertain class is available to the checker when an output is indeterminate; it is not assigned to an expert reference. Each task is paired with a machine-readable family rule specifying the relation and subject fields to be checked. The complete family definitions appear in Appendix A .

Each reference is normalized to one of four decisions. A nonempty defect list is Does Not Meet and an empty list is Meets . For submittals, any NOTMET item is Does Not Meet ; otherwise any CANNOTVERIFY item is Missing Information ; the remainder is Meets . Invalid or semantically indeterminate predictions are Uncertain . The complete reference inventory, source hashes, and family rule definitions are machine-readable.

Inclusion is registry-driven: every task in an included split and one of the seven judgment families enters the suite. Tasks that only ask a system to find a page or enumerate references are outside the target construct and are excluded by family before comparative scoring. Several tasks are controlled variants of the same documents; all such variants retain a common project ID and are kept in the same split. This is why project, rather than task or model query, is the unit used for resampling and exact tests.

Across the full suite, 146 references come from task-level ground-truth files, 12 from machine-readable reference lists embedded in the release, and two from explicit clean-case sentinels. The build script records the source type and a content hash for every normalized reference. This provenance makes the label transformation auditable while preserving the source annotations as released. It does not add an independent expert-adjudication layer.

4.2 Systems and experimental blocks

The repeated test uses four fixed deployments: Gemini 3.1 Pro Preview, Gemini 3.6 Flash, GPT-5.6 Sol, and GPT-5.6 Luna. It contains all 40 constraint tasks from the six-project held-out split. Each system receives page-RAG, region-RAG, and a fixed family policy that selects one of the two packets, with five calls per task and condition. This yields 2,400 trials. The family policy is a routing ablation, not a learned or uncertainty-adaptive router.

The panel contains two higher-capability and two efficient deployments. It is used as a balanced intervention panel rather than a leaderboard: the primary estimand gives each system equal weight, while system-specific effects remain secondary. The five repetitions are fresh queries to the same fixed deployment and packet; they measure response variability under the deployed inference route.

The breadth extension was frozen before its model queries. It uses the remaining 120 tasks from 23 disjoint development and validation projects, the two Gemini systems, the two fixed evidence policies, and one call per cell (480 trials). Because this block was designed after inspecting the repeated test, it is a breadth extension, not an independent confirmation. Results from the two blocks are reported separately.

4.3 Experimental controls

Evidence packets are materialized before inference. Both fixed policies use the unchanged task instruction, the same layout-preserving text extraction, the same BM25 ranking, at most 12 text excerpts, the same typed response contract, and the same model deployment. Only the allocation of up to four image attachments changes. Filenames that directly expose a controlled task variant are replaced by neutral document identifiers. Each packet is hashed from its instruction, ordered excerpts, and image content; trial order is randomized with a recorded seed. The response budget is 8,192 tokens.

A record-level pairing audit found no differences in instruction hash, ordered text evidence, or required output fields across the 800 page/region trial pairs in the repeated test and 240 paired cells in the breadth extension. This check verifies the intended control in the archived packets rather than assuming it from configuration alone.

The comparison is paired at the task–system level. A model first emits the public task’s structured records; the strict parser and deterministic reducer then produce the four-way decision and typed finding set. Consequently, page-RAG and region-RAG differ in visual evidence allocation, not in prompts, text retrieval, output parsing, or decision logic. The family policy selects one already-materialized packet by task family and introduces no additional model call.

4.4 Outcomes and inference

The primary endpoint is four-way decision accuracy. Repetitions are first averaged within task–system–condition cells. Tasks are then averaged within project–family, projects within family, and the seven family means equally:

Y ¯ m ​ h = 1 7 ​ ∑ f 1 | P f | ​ ∑ p ∈ P f 1 | I p ​ f | ​ ∑ i ∈ I p ​ f Y i ​ p ​ f ​ m ​ h . \bar{Y}{mh}=\frac{1}{7}\sum{f}\frac{1}{|P{f}|}\sum{p\in P{f}}\frac{1}{|I{pf}|}\sum{i\in I{pf}}Y_{ipfmh}. (3)

This prevents projects with many variants and large task families from dominating the estimate.

Secondary outcomes are false-pass rate (predicting Meets for a Does Not Meet reference), false-fail rate, exact task correctness, and typed finding precision, recall, and F1. Findings are matched one-to-one: defect matches require at least two distinct reference phrases; submittal matches also require exact status and clause. Calibration uses the empirical five-call class distribution, multiclass Brier score, and five-bin expected calibration error. Selective risk is one minus accuracy among cells retained at confidence at least .8.

False-pass rates are computed only over Does Not Meet references; false-fail rates are computed only over Meets references and count an asserted Does Not Meet decision. Typed finding scores use maximum-cardinality bipartite matching so one broad prediction cannot receive credit for several reference defects. Exact task correctness requires no unmatched prediction, no unmatched reference, and no invalid record. This separates a correct four-way triage decision from a complete professional finding set.

The primary contrast is region-RAG minus page-RAG, averaged over the fixed systems in each block. We decompose this fixed-weight estimand into additive source-project contributions. Percentile intervals resample those independent contributions with replacement (10,000 draws), and exact two-sided sign-flip tests enumerate their signs. System-specific contrasts are secondary and Holm adjusted (Holm, 1979 ) .

These two uncertainty summaries answer different finite-sample questions: the bootstrap describes variation under project resampling, whereas the exact test evaluates sign assignments to the observed contributions. With few clusters they need not cross zero at the same threshold, so we report both and do not infer significance from a percentile interval alone.

The contribution bootstrap keeps the family, project, and task weights in Equation 3 fixed while resampling the independent project clusters. The exact test enumerates every sign assignment to the same contributions. Repetitions reduce within-cell noise but do not increase the independent-project count. We report the repeated and breadth blocks separately because they answer different questions: repeated stability across four systems versus coverage across more projects for two systems.

5 Results

Table 1: Repeated-test performance against expert references. Decision values are project–family standardized percentages. FP is the false-pass rate among Does Not Meet references; finding F1 uses one-to-one typed matching. The family policy is a fixed routing ablation.

System Page-RAG Region-RAG Family policy Δ \Delta Region FP Region finding F1

Gemini 3.1 Pro 38.6 56.1 50.5 +17.5 37.9 12.6

Gemini 3.6 Flash 49.0 51.8 54.0 +2.9 43.7 16.2

GPT-5.6 Sol 38.6 46.9 46.4 +8.3 43.3 4.2

GPT-5.6 Luna 35.2 49.0 46.6 +13.8 39.4 2.9

Panel mean 40.3 51.0 49.4 +10.6 41.1 9.0

5.1 Repeated four-system test

Region-RAG improves pooled standardized decision accuracy from 40.3% to 51.0%, a gain of 10.6 percentage points (95% CI 4.3–18.0; exact p = .031 p=.031 ). Every source project’s weighted contribution to the pooled contrast is positive. The four system-specific estimates are also positive, ranging from 2.9 to 17.5 percentage points, but none is individually significant after Holm correction. The pooled panel is the primary contrast of this reanalysis, not a claim about any one deployment.

Table 1 shows the associated safety and localization outcomes. The mean standardized false-pass rate falls from 51.6% under page-RAG to 41.1% under region-RAG, and the false-fail rate falls from 35.2% to 29.4%. These are meaningful reductions but leave absolute error rates too high for unsupervised use. Exact task correctness rises only from 15.9% to 20.5%, and mean typed finding F1 from 5.2% to 9.0%. Region-focused evidence therefore improves coarse triage more than complete defect localization.

The fixed family policy reaches 49.4% decision accuracy, below always using region-RAG (51.0%). Its false-pass rate is slightly lower, but its false-fail rate is higher. Family identity alone is not sufficient routing information; an adaptive policy would need evidence-quality signals that generalize across projects.

5.2 Heterogeneity and paired error audit

The descriptive family contrasts are positive in all seven families but vary substantially. Region-minus-page accuracy is 26.1 percentage points for cross-reference resolution, 16.2 for technical-detail review, 13.3 for sheet-index checks, and 10.0 for title accuracy. It is 4.6 for submittal review, 2.5 for specification–drawing synchronization, and 1.7 for note-callout accuracy. At the task level, 25 of 40 pooled contrasts are positive, eight are zero, and seven are negative. These counts are descriptive because variants from the same source project are not independent.

The largest positive and negative task contrasts illustrate both directions. On a local reference-resolution task containing an incorrect 3/A355 citation, region-RAG produces the correct decision in 18 of 20 system–repetition trials, compared with 9 of 20 for page-RAG. Yet only seven region-RAG outputs match the typed defect, showing that decision correctness can overstate report completeness. On a cross-document door and glazing task with three specification conflicts, page-RAG is correct in 7 of 20 trials and region-RAG in only 1 of 20. The focused packet can expose small local text while withholding visual breadth needed for a multi-document comparison.

Figure 3: Region-RAG minus page-RAG decision accuracy. Intervals are source-project cluster bootstraps. The repeated-test panel is the primary contrast; system rows and the breadth extension are secondary.

5.3 Disjoint project breadth extension

Across 120 additional tasks and 23 disjoint projects, the two-system pooled contrast is -4.1 percentage points (95% CI -10.2–1.9; exact p = .209 p=.209 ). The equal-four-image sensitivity is − 9.7 -9.7 percentage points over 86 tasks from 16 projects (95% CI − 19.2 -19.2 – 0.0 0.0 ; exact p = .082 p=.082 ). Development and validation estimates are − 3.1 -3.1 and − 10.4 -10.4 percentage points, respectively. Both system-specific breadth contrasts are negative ( − 4.8 -4.8 and − 3.4 -3.4 percentage points; Holm-adjusted p = .806 p=.806 ). Five project contributions are positive, 11 negative, and seven zero. Appendix D gives the full sensitivities.

Average standardized decision accuracy in this block is 55.5% for page-RAG and 51.3% for region-RAG. Region-RAG reduces the mean false-fail rate from 41.5% to 27.9% but increases the false-pass rate from 43.0% to 46.7%. Finding F1 also falls from 18.0% to 16.1%. This block tests project breadth with one call per cell; it does not estimate repeated-run uncertainty. Its opposite point estimate shows that the repeated six-project gain is not stable under a fixed region policy across the remaining projects.

5.4 Calibration and selective risk

Across systems and evidence policies, five-call majority confidence is high (mean .86–.93) even when decisions are wrong. Expected calibration error ranges from .36 to .67. Escalating cells below .8 confidence retains 75–93% coverage, yet retained decision risk remains 43–80%. Repetition is useful for measuring instability, but agreement frequency is not a validated confidence score for this task. A deployable escalation policy requires confidence tied to evidence completeness or an independently calibrated correctness model.

6 Discussion

What the evidence contrast identifies.

The intervention keeps the instruction, text retrieval, output schema, and deployment fixed while changing how a maximum of four image attachments are allocated. Region-focused evidence improves the repeated six-project panel, whereas the 23-project breadth estimate favors page coverage and includes zero. The intervention therefore identifies a resolution–breadth tradeoff, not a dominant packet. Page breadth can be essential for cross-document checks, and short documents allow region-RAG to use attachment slots that page-RAG cannot fill. The equal-four-image sensitivity removes that slot-count difference and still favors page-RAG, although its exact project test remains inconclusive.

The estimand is architecture-level.

The pooled contrast gives equal weight to four fixed deployments after pairing conditions within task and system. It therefore asks whether changing the evidence packet shifts decisions across this panel; it is not an average model score or a system ranking. All four system estimates are positive, but their magnitudes differ and the six-project system-specific tests are coarse. Both systems in the broader block have negative point estimates. The combined result does not justify a model-independent recommendation for either fixed packet and instead motivates validation of a routing policy on new projects.

Decision and finding accuracy differ.

A ten-point decision gain can coexist with very low exact finding accuracy. The four-way label asks whether a constraint fails; a professional report must also identify the right object, location, clause, and explanation. This gap is why the architecture separates deterministic decision reduction from proof traces. Future datasets should annotate source regions and cross-document links so that trace correctness can be measured directly rather than inferred from answer text.

Implications for hybrid systems.

The family-policy ablation does not outperform a fixed region policy, and replicate agreement does not produce reliable selective risk. Useful routing signals must therefore come from the evidence state: retrieval coverage, unresolved entities, missing source links, conflicting observations, or calibrated fact confidence. The typed engine exposes these variables and can escalate before a pass is issued. The present experiment does not evaluate a learned router or establish a production-ready confidence model.

Passes require evidence completeness.

The four-state output prevents two operationally different cases from sharing a pass label. Missing Information means a required source or value was absent; Uncertain means the available evidence did not support a stable resolution. Only Meets requires every rule fact to be verified. This distinction is especially important when false passes remain common: an extractor that does not observe a defect has not thereby produced evidence that the constraint is satisfied. In a review interface, missing and uncertain cases can be routed to different queues while a trace-complete Does Not Meet case can present the cited conflict directly to an expert.

The benchmark outputs do not expose fact-level source links, so this operational policy is an architectural contract rather than a measured safety improvement. A deployment study would need experts to adjudicate both the decision and every cited region, then estimate risk at each escalation threshold on genuinely new projects.

Evidence budgets should follow constraint structure.

Region-RAG spends three attachment slots increasing spatial resolution on the highest-ranked page. This is well matched to titles, details, leaders, and local references, but it narrows visual page coverage. A production allocator should therefore condition on the rule’s evidence graph: local geometric predicates need resolution, whereas index checks and specification comparisons need breadth across pages and files. The fixed family policy is only a coarse version of this decision and leaves no mechanism to detect an atypical task within a family.

Transfer to other professional documents.

The source contract and four-state reducer do not depend on construction terminology. They can represent a clause linked to an exhibit, a maintenance requirement linked to an equipment schedule, or a clinical criterion linked to a report region. Transfer would still require domain-authored rules, source adapters, and project-level validation. The construction results alone do not establish performance in those domains.

7 Limitations

The expert references are inherited from the public release. Their provenance is documented as expert-generated, but independent relabeling and inter-annotator agreement are unavailable. The strict phrase matcher can miss valid paraphrases and should be replaced by adjudicated labels in a deployment study. The label distribution is also strongly weighted toward Does Not Meet cases; project–family standardization controls task volume, not class prevalence.

The repeated test is a reference-based analysis of an existing experiment and has only six independent projects. The 23-project breadth extension was frozen before its model queries, but it was designed after the repeated result was known; it is not an independent confirmation. It also has one call per cell and only two systems. The source tasks are public, so training-data exposure cannot be excluded. Inference concerns these project distributions and the named hosted deployments at the evaluation date, not all construction documents or future checkpoints.

With six clusters, the repeated-test bootstrap interval is necessarily coarse and cannot create project diversity absent from the data. The exact project test is reported alongside it, and no system-specific repeated-test contrast survives multiplicity correction.

Only PDF inputs are evaluated. CAD and IFC are represented by tested source and adapter contracts, not empirical modality results. The benchmark does not provide region-level proof annotations, so finding-level trace correctness is not measured. Finally, the image policies share a four-attachment cap rather than exact pixel equality, and page-RAG can use fewer than four images on short documents; the restricted sensitivity quantifies this design boundary.

The experiment isolates a complete packet-level intervention, not every mechanism inside it. Region-RAG changes page coverage, local resolution, and the number of occupied attachment slots on some tasks. The equal-four-image sensitivity removes the last difference but does not equalize rendered pixels or visual tokenization. Likewise, the fixed family policy is not a substitute for a trained evidence router, and the repeated-agreement analysis does not evaluate fact-level confidence from the typed engine.

8 Conclusion

Evidence allocation materially changes constraint decisions in professional documents. Across a repeated four-system test, concentrating a bounded visual packet on one retrieved page and its regions improves expert-reference decision accuracy by 10.6 percentage points, with positive contributions from all six source projects. Across 23 disjoint breadth projects, however, the estimate is -4.1 percentage points and both tested systems favor page-RAG. A fixed region-focused packet is therefore not a general solution to large-format document review. The practical direction is a hybrid checker that routes evidence according to the constraint, treats missing and uncertain facts explicitly, and exposes source spans and proof completeness to experts. The current systems remain unsuitable for autonomous approval: false passes are frequent, exact localization is low, and repeated agreement is miscalibrated.

Impact Statement

Construction-document review affects safety, accessibility, cost, and legal compliance. The evaluated systems have high false-pass rates and low exact finding localization; they are not suitable for autonomous approval. The intended use of this work is to improve evidence presentation, auditing, and expert triage while preserving professional responsibility.

Reproducibility Statement

The artifact freezes task and rule registries, canonical responses, scoring code, and figure sources. Recorded hashes regenerate every table and figure from archived records. Tests cover source contracts and analysis cardinalities; rebuilding the 160-task suite reproduces it byte for byte.