Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation

Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation

Fan Wu &Cuiyun Gao &Yiming Huang 1 1 footnotemark: 1 Computer Science and Technolog, Harbin Institute of Technology. Email: codenobuge@163.com .Corresponding author. Yang Xiao &Yujia Chen 1 1 footnotemark: 1 &Qing Liao 1 1 footnotemark: 1 Institute of Information Engineering, Chinese Academy of Sciences.

Abstract Recent multimodal large language models can convert visual designs directly into executable code, but real mobile products require multiple screenshots to become a buildable codebase with shared components and working navigation. This project-level setting exposes three limits of existing design-to-code benchmarks: they focus on single-page generation rather than complete codebases, cannot evaluate cross-page navigation, and do not measure project-wide maintainability. We introduce MobileForge , the first benchmark for project-level multi-screen mobile app generation, comprising 29 29 real mobile apps, 309 309 human-reviewed screens, structured page-relationship annotations, and 701 701 navigation test specifications. MobileForge supports five-axis evaluation of build, navigation, visual fidelity, code maintainability, and efficiency. We also propose state-isolated navigation testing to avoid cascading failures in navigation evaluation and an anchor-referenced list-wise visual evaluation protocol to improve visual-judge reliability. Across 174 174 end-to-end runs on six frontier multimodal LLMs, current models can build mobile-app projects that compile and reach the correct pages, but interactive navigation remains unreliable and visual fidelity and maintainability still lag. The benchmark and supporting materials are available at https://github.com/anoa12159-hue/mobileforge_eval .

Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation

Fan Wu † † thanks: Computer Science and Technolog, Harbin Institute of Technology. Email: codenobuge@163.com .

Cuiyun Gao † † thanks: Corresponding author.

Yiming Huang 1 1 footnotemark: 1

Yang Xiao † † thanks: Institute of Information Engineering, Chinese Academy of Sciences.

Yujia Chen 1 1 footnotemark: 1

Qing Liao 1 1 footnotemark: 1

1 Introduction

Can a multimodal LLM read a folder of mobile-app screenshots and generate the source code for that app? The idea of turning a designer’s screenshots directly into a running application has motivated a decade of research, from pix2code ’s CNN-LSTM screen-to-DSL pipeline Beltramelli ( 2018 ) to multimodal LLMs that take a webpage screenshot as input and generate HTML+CSS in one shot Si et al. ( 2025 ); Laurençon et al. ( 2024 ); Gui et al. ( 2025 ) . Recent work on single-page design-to-code with rich design-tool metadata further reflects this continued focus on individual pages Gui et al. ( 2026 ) .

A real product, however, is never a single screen. It is a folder of screenshots that must collectively become a buildable codebase with shared components, consistent design tokens, working routing, and a navigable user experience. Existing design-to-code benchmarks fall short in three ways exposed by this project-level setting. (1) Single-page focus. They target one webpage at a time and do not evaluate a multi-screen project as a whole, leaving cross-page coherence and shared-component reuse out of scope. (2) No evaluation of interactive navigation. They cannot test whether the generated app’s global routing correctly connects tab transitions and parent–child transitions to the intended targets, even though navigation correctness is a basic functional requirement of a usable app. (3) No measure of project-wide code maintainability. They omit indicators such as component reuse, dead-component prevalence, and cross-page design-token consistency, which directly affect downstream engineering cost when a generated project is handed off to a team.

To address these gaps, we introduce MobileForge , the first project-level multi-screen mobile app generation benchmark, together with a five-axis evaluation framework for multi-screen projects (Figure 1 ). Our experiments on six frontier multimodal LLMs across 174 174 end-to-end runs show clear gaps between current models and the requirements of project-level interactive code generation, gaps that single-page benchmarks cannot expose.

Figure 1: Prior design-to-code benchmarks take a single screenshot as input and score the resulting HTML page by visual similarity alone (top). MobileForge takes the full set of screenshots of a real mobile app and a page-relationship description as input, asks an agentic harness to produce a runnable multi-page project, and scores it on five orthogonal axes: build, navigation, visual fidelity, code maintainability, and efficiency (bottom).

Key Contributions

• Task and Benchmark. We formulate project-level multi-screen mobile app generation as a new task and release MobileForge , the first benchmark for this setting, comprising 29 29 in-market consumer apps with 309 309 human-reviewed screens, structured page-relationship descriptions, and 701 701 navigation test specifications.

• Evaluation Framework. We develop a five-axis evaluation framework covering build, navigation, visual fidelity, code maintainability, and efficiency. Two new evaluation techniques anchor the framework: state-isolated navigation testing , which runs each navigation specification from a fixed source-page route rather than a chained walkthrough, and an anchor-reference list-wise visual evaluation protocol , calibrated against human raters and motivated by the documented reliability issues of point-wise VLM judging.

• Large-Scale Empirical Study. We evaluate six frontier multimodal LLMs across 174 174 end-to-end runs and find that current models can build mobile-app projects that compile and reach the correct pages, yet interactive navigation is unreliable and visual fidelity and code maintainability leave substantial room for improvement. We release MobileForge, the evaluation harness, and all run artifacts to support reproducibility.

2 Related Work

Screenshot-to-code and design-to-code.

Pix2code Beltramelli ( 2018 ) introduced neural code generation from a single GUI screenshot. The modern wave, including Design2Code Si et al. ( 2025 ) , WebSight Laurençon et al. ( 2024 ) , WebCode2M Gui et al. ( 2025 ) , and recent VLM-centric efforts Ge et al. ( 2025 ); Jiang et al. ( 2025 ); Yang et al. ( 2025 ) , reframes the task as multimodal prompting over web screenshots. However, they still treat each example as a single webpage. Figma2Code Gui et al. ( 2026 ) studies a complementary setting based on Figma design files, in which structural component definitions are available to the model. A concurrent submission by a partially overlapping author set Anonymous ( 2026 ) benchmarks MLLMs on interactive single-file HTML/CSS/JavaScript webpages and explicitly scopes out component frameworks, multi-file projects, and multi-page navigation, which are the regimes MobileForge targets. MobileForge differs from all the above in two ways. First, it uses only screenshots and does not assume any structural prior. Second, it targets generation at the project level across multiple screens rather than single page generation.

Mobile UI corpora, agents, and code-generation benchmarks.

A large body of work targets static-UI perception , including Rico Deka et al. ( 2017 ) and downstream understanding work Wang et al. ( 2021 ); Li et al. ( 2020 ); Baechler et al. ( 2024 ); You et al. ( 2024 ); Hong et al. ( 2024b ); Lu et al. ( 2024 ) , or trains agents to operate running apps Zhang et al. ( 2025a ); Wang et al. ( 2024a ); Cheng et al. ( 2024 ); Xie et al. ( 2024 ) . MobileForge inverts both directions and asks models to generate the project. On the code side, benchmarks have moved from function-level (HumanEval Chen et al. ( 2021 ) , MBPP Austin et al. ( 2021 ) ) to repository-level (SWE-bench and variants Jimenez et al. ( 2024 ); Yang et al. ( 2024 ); Deng et al. ( 2024 ); Tian et al. ( 2026 ) ) and broader agentic suites Liu et al. ( 2024b ); Ma et al. ( 2024 ) , but none takes a visual design as input. Our single-agent harness draws on the iterative tool-use paradigm Yao et al. ( 2023b , a ); Madaan et al. ( 2023 ); Schick et al. ( 2023 ); Wang et al. ( 2024b ); Xia et al. ( 2025 ); Wang et al. ( 2025 ); Liu et al. ( 2024a ) ; we deliberately fix a single-agent setup so that observed differences are attributable to the model rather than the orchestration Hong et al. ( 2024a ); Qian et al. ( 2024 ); Chen et al. ( 2024 ); Zhang et al. ( 2025b ) .

VLM-as-a-Judge.

Vision-language judges have become standard evaluators when no gold output exists Lee et al. ( 2024 ); Li et al. ( 2026 ); Feizi et al. ( 2025 ) , and the literature documents two failure modes of point-wise scoring: scale drift across sessions and ties-in-the-middle compression. Pair-wise and list-wise protocols are more reliable, with list-wise producing a full ranking at O ​ ( N ) O(N) rather than O ​ ( N 2 ) O(N^{2}) call cost. Our anchor-reference list-wise protocol (§ 4.3 ) extends list-wise judging with an in-evaluation sanity check, and we substantiate the choice with empirical comparison (Appendix C ).

3 The MobileForge Benchmark

Figure 2 summarizes the four stages of MobileForge: dataset construction, annotation, generation, and five-axis evaluation. The remainder of this section and § 4 describe them in detail.

Figure 2: MobileForge pipeline. (1) Dataset construction starts from three design principles: realism, project granularity, and runnable evaluation. It collects 29 29 in-market apps across 14 14 category buckets with coherent multi-page screenshot sets ( 309 309 screens, mean 10.7 10.7 per app). (2) Annotation runs a VLM auto-draft over the screenshots, and then a human reviewer edits the page-relationship description and the navigation test specifications. (3) Generation hands the reviewed screenshots and page-relationship description to a single-agent harness ( 8 8 tools, pre-scaffolded directory, verify-repair loop). (4) Evaluation scores the resulting project on five axes, namely build, navigation, visual fidelity, code maintainability, and efficiency, using state-isolated navigation testing (§ 4.2 ) and an anchor-reference list-wise visual judge (§ 4.3 ).

3.1 Design Principles

MobileForge is shaped by three principles: (i) realism , where every instance is an in-market consumer app rather than a synthetic mockup; (ii) project granularity , where each instance is a folder of screens so that cross-page consistency and routing are exercised; and (iii) runnable evaluation , where ground truth is the combination of building , navigating , and visually matching , not pixel-perfect reproduction, which is brittle and underspecified.

3.2 App Selection and Capture

We sampled 29 29 apps spanning 14 14 category buckets (media, finance, social, navigation, e-commerce, productivity, communication, etc.), prioritizing broad cultural recognition. For each app, we captured a coherent screenshot set covering the primary tab structure plus key sub-screens, totaling 309 309 screens (mean 10.7 10.7 , std 3.8 3.8 , range 5 5 – 21 21 per app). The full app list and per-app statistics are in Appendix B (Table 5 ).

3.3 Annotation Schema

Each app’s annotation comprises two parts. The first is a page-relationship description that enters the generation pipeline as model input. It specifies the application’s tab structure, namely the main navigation tabs visible at the application root, and the parent–child relationships between screens, encoded as natural-language statements of the form “ Screen X is a sub-page of Screen Y, triggered by tapping the search icon .” This is the structural information the model needs to plan multi-page routing. The second is a set of navigation test specifications used only at evaluation time. Each specification belongs to one of three categories: page existence (is a given screen reachable at the expected route), tab navigation (does tapping a tab item route to the correct screen), and parent–child navigation (does triggering a known affordance on a parent screen transition to the expected child). Each test records a starting screen, an action with strategy hints (DOM selectors, text matches, tab positions), and the expected target screen. The full benchmark contains 701 701 such specifications across 29 29 apps (mean 24.2 24.2 , range 11 11 – 41 41 per app), distributed as 120 120 page-existence, 392 392 tab-navigation, and 189 189 parent–child cases.

3.4 Annotation Pipeline and Quality

Annotation follows a two-stage pipeline: a vision-language model drafts the schema entries of § 3.3 , and human annotators then review every field in a labelling interface (Figure 4 ) and may edit, add, or remove entries before producing the final ground truth. Three properties anchor the dataset’s reliability. (i) Data quality : every retained annotation has passed human review. (ii) Annotation-framework accuracy : against the post-review ground truth, the auto-drafts achieve 96.8 % 96.8% recall and 94.3 % 94.3% precision at the page level, and 74.1 % 74.1% recall and 83.9 % 83.9% precision at the test-case level. (iii) Low edit rate : 64.7 % 64.7% of drafted page annotations and 76.5 % 76.5% of test cases are accepted unchanged, confirming that the auto-drafting stage produces candidates that the human reviewer only lightly edits.

3.5 Benchmark Statistics

The benchmark spans 309 309 screens, 701 701 test cases, 10.7 10.7 avg. screens per app, and a long-tail distribution of complexity (Khan Academy: 5 5 screens / 11 11 tests; Klook: 21 21 screens / 41 41 tests). Table 1 contrasts MobileForge with prior design-to-code benchmarks along the dimensions that matter for project-level evaluation.

Benchmark Unit of evaluation Evaluation Axes

Build Nav. Visual Maint. Eff.

Design2Code Si et al. ( 2025 )

484 484 real webpages – – ✓ – –

WebSight Laurençon et al. ( 2024 )

2 2 M synth. webpages – – ✓ – –

WebCode2M Gui et al. ( 2025 )

2.56 2.56 M real webpages – – ✓ – –

Figma2Code Gui et al. ( 2026 ) real Figma files – – ✓ ✓ –

MobileForge (ours) 29 29 real apps / / 309 309 screens ✓ ✓ ✓ ✓ ✓

Table 1: Positioning of MobileForge among design-to-code benchmarks. Prior work evaluates single webpages, primarily along visual similarity; MobileForge introduces project-level evaluation along five axes. Inputs are screenshots throughout, except Figma2Code, which takes Figma metadata. ✓indicates that the axis is evaluated.

4 Evaluation Protocol

4.1 The Five Axes

We evaluate each project along five axes. Three use standard measurements, summarized below, and two rest on the new evaluation techniques of § 4.2 –§ 4.3 .

Build Success. Each project is shipped to a clean Vite scaffold (React 19 19 + TypeScript + Tailwind). We run npx tsc --noEmit and vite build ; a run builds only if both pass with zero errors. We additionally report route count, page-file count, and route coverage.

Navigation Correctness. NavPassRate : the fraction of the 701 701 navigation specifications (page-existence, tab-navigation, parent–child) passed under state-isolated execution (§ 4.2 ).

Visual Fidelity. A per-model Borda score in [ 0 , 1 ] [0,1] aggregated from anchor-reference list-wise ranking by a fixed vision-language judge; a 4 4 -dimension point-wise rubric is retained as an auditable reference baseline (§ 4.3 ).

Code Maintainability. Five indicators capture post-generation engineering upkeep: total LoC ; mean LoC/file (conciseness); ReuseRate , the average import count per shared component; DeadCompRate , the fraction of declared components that are never imported; and ColorConsistency , the fraction of Tailwind color tokens from a small recurring palette versus one-off shades. We use the term code maintainability rather than code quality because the former clearly excludes correctness, which is already captured by Build, and keeps the indicators focused on engineering handoff.

Efficiency. For each run, we log input and output tokens, LLM call count, wall-clock LLM time, and a dollar-cost estimate from the model card list price. Efficiency is reported as a first-class axis rather than a footnote, since cost–quality trade-offs surface a distinct finding (§ 6.3 ).

4.2 State-Isolated Navigation Testing

Evaluating multi-screen navigation under the standard chained-walkthrough protocol suffers from error cascading . Under this protocol, the test agent navigates from the home screen and exercises the entire app as a sequence of clicks. If the bottom-tab routing is broken at step 3 3 , every downstream test on the same chain fails too, and the cause of each later failure becomes ambiguous: is it the screen itself, or the upstream tab? In a project with 20 20 + test cases per app, a single early break can invalidate half of the chain. This protocol cannot tell us which failure mode dominates, and that is precisely the question we want to answer.

We propose state-isolated navigation testing : each test case begins from a fixed source-page route provided to the Playwright driver before the action is executed, rather than from a chained sequence starting at the home screen. Concretely, the driver navigates the browser directly to the route of the source page (e.g., /search rather than / → \rightarrow /search ), waits for content to settle, executes the action under test, and observes the post-action page. The starting state of every test is therefore independent of every other test.

State isolation produces three benefits. (i) Statistical independence : failures are independent events, so per-model failure rates can be compared without inflating denominators on chained failure cascades. (ii) Fine-grained attribution : a failure observed on test N N is genuinely caused by interaction N N , which is the prerequisite for the C1–C4 user-perception failure taxonomy we develop in § 7 . (iii) Reachability decoupled from operability : tests of the form “does the target page render at its route?” (page-existence cases) are separated from tests of the form “does the trigger fire?” (parent–child and tab-navigation cases), so reachability and operability can be diagnosed separately.

We render the built project in a headless Playwright browser ( 375 × 812 375{\times}812 viewport, simulating an iPhone X) and execute the 701 701 test specifications. A target is reached if the URL matches the expected page or, for same-route overlays, the page content hash changes meaningfully.

4.3 Anchor-Reference List-Wise Visual Evaluation

Cross-model visual evaluation is structurally hard for VLM-as-Judge. Point-wise scoring, the default in prior design-to-code work, asks the judge to assign a 1 1 – 5 5 score to a single candidate, ignoring how it compares with the alternatives. This produces two well-documented failure modes: scale drift across sessions, where the same image receives different scores at different times, and ties-in-the-middle compression, where close candidates collapse to the same integer score. Pair-wise judging adds a relative anchor but costs O ​ ( N 2 ) O(N^{2}) comparisons per scenario, where N N is the number of candidate models compared per scenario. Our empirical study (Appendix C ) further finds that it suffers from 23.3 % 23.3% position bias that requires swap-augmented re-runs. Standard list-wise judging is cheaper ( O ​ ( N ) O(N) calls) and produces a full ranking, but it provides no signal on whether the judge actually performed the comparison or simply guessed.

We add a small but consequential modification: we mix the original design screenshot itself, anonymized, into the candidate set as a hidden anchor. The judge ranks N + 1 N{+}1 candidates ( N N model outputs plus the anchor) from most to least visually faithful. A ranking is valid only when the anchor is placed first, since the anchor is pixel-identical to the reference and any judge that correctly sees the candidates must rank it first. If the anchor is misplaced, the ranking is discarded and the call is re-run.

This anchor mechanism functions on two levels.

(i) In-evaluation sanity check. Every individual judge call is self-validating, and we discard rankings that fail anchor placement rather than including them in the Borda aggregation. The per-call anchor pass rate is itself a quality indicator we report alongside the Borda scores.

(ii) Judge-model selection criterion. The same mechanism becomes a selection criterion for VLM judges. A candidate judge that cannot consistently place the anchor first on a screening set is not eligible to score the benchmark. In our experiments, Gemini 2.5 Pro achieves ∼ 100 % \sim!100% anchor pass on a 33 33 -scenario screening set; Claude Sonnet 4.5 also achieves ∼ 100 % \sim!100% ; GPT-5.1 fails at ∼ 35 % \sim!35% because it explicitly reasons that the reference “is not an independent candidate” and ranks it last, disqualifying it under this criterion. Appendix D reports the full cross-judge robustness analysis and a methodological note on the GPT-5.1 failure.

The judge produces a per-scenario ranking, which we aggregate to a per-model Borda score in [ 0 , 1 ] [0,1] via the 701 701 -scenario corpus. Borda is the primary visual metric. We additionally retain the per-pair point-wise protocol, a 4 4 -dimension rubric (Layout 30 % 30% , Style 25 % 25% , Content 25 % 25% , Consistency 20 % 20% , each 1 1 – 5 5 ), as an auditable reference baseline. This baseline supplies the absolute-ceiling reading (“no model exceeds 3.0 / 5 3.0/5 ”) that Borda’s ordinal scale cannot. Full justification combining theoretical analysis, literature support, and the three-way (point-wise / pair-wise / list-wise) empirical comparison is in Appendices C and D .

5 Experimental Setup

5.1 Agentic Harness

We deliberately fix the prompting strategy across models to isolate model capability rather than prompt engineering. All models share an identical agentic pipeline: a single tool-augmented agent receives the design screenshots and the page-relationship description from § 3.3 , and a system prompt instructs it to produce a React+TypeScript+Tailwind project under a pre-scaffolded directory. The agent has eight tools: writefile , readfile , strreplace , batchstrreplace , listfiles , deletefile , runcommand (for tsc /build), and taskdone . The agent follows a soft six-phase plan (analyse, foundation, components, pages, assembly, verify) and may iterate up to 50 50 times. On taskdone , the harness triggers an automated tsc + vite build; build errors are returned to the agent, which may continue iterating. Smaller models reach the same endpoint by writing files in many small edits and self-repairing more often; the per-phase token attribution accounts for 18 18 – 35 % 35% of total tokens in the verify phase, with the smaller models at the high end.

5.2 Models

We evaluate six closed-weight frontier LLMs available with multimodal input as of April 2026: Claude Opus 4.6 and Claude Haiku 4.5 (Anthropic), GPT-5 and GPT-5 Mini (OpenAI), and Gemini 2.5 Pro and Gemini 2.5 Flash (Google). For each app–model cell, we run a single trial, totalling 29 × 6 = 174 29\times 6=174 runs, all of which completed and were evaluated end-to-end.

5.3 Judge Selection

The vision-language judge for visual fidelity is fixed across all runs as Gemini 2.5 Pro. The selection criterion is that the candidate judge must reliably pass the anchor sanity check on a screening set: when the reference design is mixed into the candidate ranking, a competent judge places it first. Models that fail this check are not eligible. A cross-judge robustness check with Claude Sonnet 4.5 is reported in Appendix D .

6 Results

6.1 Visual Fidelity Has Not Saturated

Table 2 reports per-model means across all 29 29 apps. List-wise Borda is the primary visual metric, and point-wise visual scores are reported alongside as an auditable absolute-ceiling reference.

Model Tok.(K) $ Calls Build% NavPass ↑ \uparrow Borda ↑ \uparrow Vis(pt) ↑ \uparrow LoC

Claude Opus 4.6 488 488 2.90 2.90 14.7 14.7 100 100 𝟗𝟐 % \mathbf{92%} 0.906 \mathbf{0.906} 2.99 \mathbf{2.99} 1 , 441 1{,}441

Claude Haiku 4.5 1 , 308 1{,}308 1.40 1.40 38.5 38.5 100 100 85 % 85% 0.339 0.339 2.37 2.37 1 , 290 1{,}290

GPT-5 304 304 0.50 0.50 14.7 14.7 100 100 89 % 89% 0.542 0.542 2.64 2.64 668 668

GPT-5 Mini 542 542 0.15 0.15 23.9 23.9 100 100 58 % 58% 0.130 0.130 1.60 1.60 436 436

Gemini 2.5 Pro 336 336 0.64 0.64 16.0 16.0 100 100 82 % 82% 0.513 0.513 2.56 2.56 920 920

Gemini 2.5 Flash 839 839 0.33 0.33 34.5 34.5 100 100 62 % 62% 0.186 0.186 2.00 2.00 1 , 176 1{,}176

Table 2: Per-model results aggregated over all 29 29 apps ( 174 174 runs). Borda : list-wise Borda score on the 701 701 -scenario judge run, normalised to [ 0 , 1 ] [0,1] and used as the primary visual metric. Vis(pt) : point-wise visual mean (1–5 scale; reference baseline). Tok. : input and output token mean. Calls : mean LLM tool-call iteration count. Token differences are dominated by iteration count: Haiku averages 38.5 38.5 iterations versus 14.7 14.7 for Opus and GPT-5, with smaller models writing many small edits and self-repairing more often. Best per column in bold .

Build success has saturated at 100 % 100% , but visual fidelity has not. List-wise judging shows substantial between-model differences: Claude Opus 4.6 ranks first in 26 26 of 29 29 apps, the bottom model wins zero apps, and the global Borda scores span the full [ 0.13 , 0.91 ] [0.13,0.91] range. Point-wise scoring further shows that even the leader does not approach absolute visual faithfulness: the top mean is 2.99 / 5 2.99/5 (Opus), with all six models falling in [ 1.60 , 2.99 ] [1.60,2.99] . The gap is concentrated in the cross-page Consistency dimension. Models routinely re-implement navigation chrome inline on each page rather than sharing a single component, which produces subtle color and sizing drift across screens. Cross-page consistency, not single-screen rendering, is the dominant remaining bottleneck, and it is invisible to single-page benchmarks.

6.2 Code Maintainability Is Orthogonal to Visual Fidelity

Table 3 reveals a sharp dissociation between visual leadership and code maintainability leadership. Claude Opus 4.6, the visual leader, writes 2.2 × 2.2\times the lines of code of GPT-5 ( 1 , 441 1{,}441 versus 668 668 ) at 2.5 × 2.5\times the average file size ( 82 82 versus 33 33 LoC/file). GPT-5 also achieves higher component reuse ( 4.23 4.23 versus 3.83 3.83 imports per shared component) and 2.5 × 2.5\times fewer dead components ( 7.8 % 7.8% versus 19.9 % 19.9% ). Design-token color consistency has largely saturated across all models ( 0.74 0.74 – 0.85 0.85 , a 15 % 15% spread), whereas the structural engineering metrics differentiate sharply.

Model LoC LoC/file Reuse ↑ \uparrow Dead% ↓ \downarrow Color ↑ \uparrow

Claude Opus 4.6 1 , 441 1{,}441 82 82 3.83 3.83 19.9 19.9 0.81 0.81

Claude Haiku 4.5 1 , 290 1{,}290 68 68 3.04 3.04 12.9 12.9 0.85 0.85

GPT-5 668 668 33 33 4.23 \mathbf{4.23} 7.8 \mathbf{7.8} 0.80 0.80

GPT-5 Mini 436 436 26 26 3.49 3.49 12.8 12.8 0.74 0.74

Gemini 2.5 Pro 920 920 41 41 2.98 2.98 21.4 21.4 0.80 0.80

Gemini 2.5 Flash 1 , 176 1{,}176 52 52 2.84 2.84 17.6 17.6 0.78 0.78

Table 3: Code maintainability metrics per model (means over 29 29 apps). LoC/file : mean lines per source file, used as a conciseness proxy. Reuse : mean import count per shared component. Dead% : fraction of declared shared components never imported. Color : fraction of Tailwind color tokens from a small recurring palette. Full per-model breakdown including file count, shared-component count, and type utilization is in Appendix J .

The visual leader ranks third on component reuse, fifth on dead-component rate, and last on code conciseness. A team optimizing for designer fidelity at hand-off would choose Opus; a team optimizing for downstream code maintenance would choose GPT-5. Existing single-axis benchmarks conflate these two operating points.

6.3 Capability-Cost Pareto Frontier

Plotting list-wise Borda against per-app dollar cost (Figure 3 ) yields a Pareto frontier where GPT-5 dominates two mid-tier models on cost and quality jointly . GPT-5 ($0.50/app, Borda 0.542 0.542 , NavPass 89 % 89% ) sits strictly inside the dominance region of both Claude Haiku 4.5 ($1.40, Borda 0.339 0.339 , NavPass 85 % 85% ) and Gemini 2.5 Pro ($0.64, Borda 0.513 0.513 , NavPass 82 % 82% ). The 19 × 19\times cost spread between the cheapest ($0.15) and most expensive ($2.90) model does not translate into a proportional quality spread in the relative ranking, and two mid-tier models are dominated. We also observe content fabrication, where models fill otherwise complete UI slots with Lorem ipsum text or stock-photo imagery instead of the screenshot’s actual content, predominantly in the lowest-cost models (GPT-5 Mini and Gemini 2.5 Flash); we treat this as part of the cost-quality picture rather than as a separate navigation failure mode.

Figure 3: Capability-cost frontier on MobileForge. Each point is a model’s mean list-wise Borda score versus mean dollar cost across 29 29 apps; error bars are ± 1 \pm 1 standard deviation across apps.

6.4 Per-App Variance

Difficulty varies sharply across apps. The easiest apps, Notion ( 3.62 3.62 ), Khan Academy ( 3.50 3.50 ), and WhatsApp ( 3.33 3.33 ), are visually clean with compact tab structures. The hardest apps, Apple News ( 2.01 2.01 ), Airbnb ( 2.02 2.02 ), and Coinbase ( 2.07 2.07 ), combine dense data presentation with custom typography and information-rich imagery. Klook ( 21 21 screens, 0.48 0.48 NavPass) and Medium ( 10 10 screens, 0.45 0.45 NavPass) illustrate the cross-page consistency penalty: many screens magnify per-screen weaknesses and stress component-sharing decisions. The full per-app table is in Appendix B ; we read this distribution as evidence that the 29 29 -app pool covers a wide spread of project-level difficulty.

7 Failure Analysis

We focus our failure analysis on interactive navigation, namely the 581 581 test specifications for tab navigation and parent–child navigation, which directly test the model’s overall routing structure. Across all 174 174 runs ( 6 6 models × \times 29 29 apps), these specifications generate 3 , 486 3{,}486 test-case executions. We hand-label a stratified sample of 64 64 failures under a four-class user-perception schema (C1–C4) and project the sample proportions to the population with Wilson 95 % 95% confidence intervals. State-isolated testing (§ 4.2 ) makes this fine-grained attribution possible: a failure observed on a given test cannot be blamed on an upstream chained interaction.

Three modes account for roughly 90 % 90% of interactive navigation failures (Table 4 ).

Code Description p ^ \hat{p} 95 % 95% CI

C1 Blank starting page 7.8 % 7.8% [ 3.0 , 18.3 ] [3.0,18.3]

C2 Route mapping error 10.9 % 10.9% [ 5.4 , 20.9 ] [5.4,20.9]

C3 Target unreachable 32.8 % 32.8% [ 22.6 , 45.0 ] [22.6,45.0]

C4 Click has no effect 48.4 % 48.4% [ 36.6 , 60.4 ] [36.6,60.4]

Table 4: Distribution of interactive navigation failure modes from a stratified sample of n = 64 n{=}64 .

C1: blank starting page ( 7.8 % 7.8% ).

The source page renders empty or as a stub placeholder without substantive content or interaction targets, so no navigation action can be dispatched (Figures 7 – 8 ). This is distinct from the filled-but-fake content fabrication discussed in § 6.3 : in C1 the relevant affordances are absent, whereas in fabrication the UI structure is present but its text or imagery is placeholder content. C1 is the distinctive failure mode for Gemini 2.5 Flash, which alone accounts for an outsized share of C1 occurrences: 42 % 42% of its labelled failures are C1, versus 0 % 0% for the four next-best models. Flash’s high iteration count ( 34.5 34.5 on average) does not translate into substantive page content for failed cases.

C2: route mapping error ( 10.9 % 10.9% ).

The agent declared the route, but the URL resolves to a different page than the one referenced by the test (Figure 9 ). The model has correctly produced the page; the global routing table simply maps the navigation action to the wrong page.

C3: target unreachable or occluded ( 32.8 % 32.8% ).

The action target is not present in the rendered viewport, either because the layout omits it entirely or because another element covers it (Figures 10 , 11 , 12 ). This is structurally distinct from C4: in C3 the page is malformed, whereas in C4 it is well-formed but inert. C3 dominates the smaller models, where layout sparsity is the recurring failure mode.

C4: target clickable but unresponsive ( 48.4 % 48.4% ).

The page renders, the affordance is in the right place, but the click registers no effect (Figures 13 , 14 , 15 ). This is consistent with a missing or mis-wired onClick handler. Build verification catches type errors but not silent event-handler bugs; this is the dominant failure mode for the stronger models.

Per-model headline observations.

The failure-mode distribution differs sharply across models (per-model n = 8 n{=}8 – 12 12 ; we read the per-model proportions as descriptive signatures rather than precise point estimates). Claude Opus 4.6 and GPT-5 show identical signatures: 75 % 75% C4 in our labelled sample (Table 10 ), indicating that their failures concentrate on broken interaction wiring rather than missing UI. Gemini 2.5 Pro is similar but less extreme ( 55 % 55% C4, 45 % 45% C3). GPT-5 Mini reverses the pattern: 67 % 67% C3, so when GPT-5 Mini fails, it most often produces a layout that simply lacks the target affordance. Gemini 2.5 Flash is the C1 outlier ( 42 % 42% ). Claude Haiku 4.5’s failures are more uniformly spread across C2–C4, consistent with its higher route-mapping error contribution. Practically, the remaining failures are not subtle: handlers that do not fire and targets that are not drawn, both diagnosable from the generated component tree before runtime. Full per-model and per-scenario-type tables are in Tables 10 and 11 .

8 Conclusion

We introduce MobileForge, the first project-level benchmark for multi-screen mobile app generation, and evaluate six frontier LLMs end-to-end on 174 174 runs along a five-axis protocol. The answer to our opening question is partly yes : every model produces a buildable project and the strongest reaches 92 % 92% navigation correctness, yet visual fidelity has not saturated, maintainability is uncorrelated with visual fidelity, and roughly 90 % 90% of navigation failures concentrate in three predictable modes.

Limitations

Single trial per cell.

Each (app, model) pair runs once. We chose 29 × 6 29\times 6 apps over fewer apps with more trials to maximise diversity; a multi-trial replication is left to future work.

Vision-language judge.

Visual scores are produced by Gemini 2.5 Pro, a competitor model class. We fix the judge across all runs, so all models share the same judge bias, calibrate the judge against human ratings on 55 55 scenarios (strict Kendall τ = 0.716 \tau{=}0.716 , top-1 accuracy 83.6 % 83.6% ), and cross-check with Claude Sonnet 4.5 on 139 139 scenarios (inter-judge τ = 0.867 \tau{=}0.867 ). No self-preference bias toward either judge family is observed.

Web-rendering proxy.

The output target is web (React+Tailwind), not native iOS/Android. We chose web for evaluability, including browser automation and deterministic builds, at the cost of not exercising native APIs. Cross-platform extension is a natural follow-up.

Ethical Considerations

Data sources and licensing.

The 29 29 apps in MobileForge are widely distributed consumer apps obtained through their official public distribution channels, and screenshots were captured manually from default-state or demo-mode screens without bypassing paywalls, login walls, or rate-limit protections. To respect the intellectual property of the source apps, our public release contains the captured screenshot set together with the structured page-relationship annotations and navigation test specifications, framed as a research benchmark rather than a verbatim re-distribution of any app’s UI assets. The benchmark and the accompanying harness and evaluator are released under a research-only license that prohibits commercial reuse of the screenshot artifacts and any derivative product that re-skins them.

Privacy and PII.

MobileForge focuses on UI-level structure and navigation on the public surfaces of consumer apps. Annotators captured screens without logging into real user accounts, and no personal profiles, payment information, private messages, or location histories were accessed or collected. We manually inspected every screenshot and redacted incidental content that could disclose third-party PII before release.

Human annotators.

The page-relationship descriptions and navigation test specifications were authored by members of our research team rather than crowdworkers. Annotators were fully informed about the purpose and intended release of the dataset, participated voluntarily, and were compensated above the local minimum wage standard of their region.

Use of pre-trained LLMs and compute.

Our experiments rely on publicly available commercial LLM inference APIs and a publicly available VLM judge. API usage conforms to the respective providers’ Terms of Service. Because agentic code generation with frontier LLMs is energy-intensive, our pipeline runs each (app, model) cell exactly once and caches all intermediate artifacts so that downstream metric recomputation does not re-trigger generation or browser automation. Token consumption and wall-clock time per app are reported transparently as part of the efficiency axis (§ 4.1 ).

Potential misuse and mitigation.

A capable design-to-code model could in principle be misused to clone legitimate apps for phishing, impersonation, or counterfeit distribution. We mitigate this risk in several ways. MobileForge is a benchmark, not a generator or a fine-tuning corpus, and it does not lower the barrier to producing malicious clones beyond what off-the-shelf multimodal LLMs already provide. The five-axis evaluation rewards build, navigation correctness, visual fidelity, maintainability, and efficiency, none of which is informative for malicious intent. The research-only license further forbids commercial use, ruling out the most plausible misuse vectors such as deploying re-skins of the source apps for profit.

AI writing and coding assistance.

Per the ACL policy on AI writing assistance, we disclose that general-purpose AI assistants were used solely for language polishing and for low-level coding assistance during the implementation of the agentic harness and the evaluator. All scientific claims, experimental design, data analysis, and conclusions are the work of the human authors, who take full responsibility for the correctness of the content.

Reproducibility Statement

The benchmark, the agentic harness, and the five-axis evaluator are released as a single repository, together with the 174 174 run artifacts. Each run includes the generated source tree, the build log, the navigation-test screenshots, the VLM judge rationales, and a structured run-summary record. A one-command replay reproduces the comparison table verbatim.