DungeonBench: A Benchmark for Rules-Rich Tactical Reasoning in Dungeons & Dragons Combat

DungeonBench: A Benchmark for Rules-Rich Tactical Reasoning in Dungeons & Dragons Combat

Ismayil Ismayilov fal &Atakan Kara fal &Kaan Oktay fal

Abstract Games and simulators make valuable benchmarks by turning decisions into measurable outcomes, but many current suites under-test rules-rich tactical reasoning: the ability to choose well when geometry, timing, resources, objectives, and rule interactions all matter at once. We introduce DungeonBench , a benchmark for tactical reasoning in Dungeons & Dragons combat, built to cover the vast majority of combat-relevant 2014 System Reference Document content whose effects can be resolved by the simulator while retaining mechanics that simplified combat simulators often abstract away. At each step, DungeonBench exposes a complete tactical observation, a pending decision, and an indexed list of executable options spanning movement, attacks, spells, reactions, objectives, preparation, and scarce resources. The task is to value legal choices whose consequences depend on action economy, creature traits, battlefield geometry, timing windows, and future encounters. DungeonBench has two tracks: Encounter , which evaluates local tactical play in single fights, and Day , which links encounters through persistent hit points, spell slots, consumables, preparation, and short-rest timing, forcing policies to trade off immediate tactical advantage against future survivability. The same engine-generated decision stream supports heuristic controllers, language-model policies, learned option rankers, and masked-action reinforcement-learning agents. We evaluate frontier language-model policies on this shared decision stream. Results show that full tactical observations do not saturate the benchmark: frontier policies often win direct encounters, but linked encounter days expose failures in resource budgeting, rest timing, and rule-aware tactical discipline.

1 Introduction

Figure 1: Overview of DungeonBench. The engine maps tactical state to legal executable options; policies choose over the same decision stream, and the simulator scores Encounter and Day outcomes.

Games make robust testbeds for artificial intelligence: explicit rules, measurable outcomes, and repeatable interaction loops give an unambiguous account of what an agent did and what its decision produced. As a result of this, they have supported progress in continuous control, board games, multi-agent sports, procedural navigation, and open-ended worlds (Brockman et al. , 2016 ; Cobbe et al. , 2020 ; Lanctot et al. , 2019 ; Xu et al. , 2025 ) . Yet one kind of reasoning remains underrepresented: tactical judgment in domains where legal actions are heterogeneous and their value is determined by rule interactions, timing windows, objectives, and scarce resources.

In a tactical domain, one decision may be about where to stand, another about which enemy to target, another about whether to spend a scarce spell slot, trigger a reaction, protect concentration, exploit terrain, or hold a resource for a later fight. These choices draw on different action types and acquire value through the rules around them. A locally strong action can be strategically wrong; a low-damage move can be decisive because it preserves concentration, blocks a doorway, saves a reaction, or keeps a resource for the next encounter. The challenge is not only long horizon or large branching factor. It is state-contingent option valuation under executable rules.

The source domain is the combat-relevant portion of 2014 Dungeons & Dragons. In Dungeons & Dragons combat, parties and opposing creatures act in initiative order on a battlefield; turns combine movement, attacks, spells, object interactions, reactions, and resource spending. We use this domain because it offers a mature library of tactical primitives rather than a rule system invented solely for the benchmark. Its mechanics have been accumulated, revised, and stress-tested through years of play, giving DungeonBench a broad rule surface where positioning, resources, timing, and rule interactions all matter without being tuned to any particular model class. We restrict attention to combat-relevant rules that can be represented and resolved by the engine.

DungeonBench turns this problem into a benchmark for tactical reasoning. It exposes each decision as a complete tactical state together with an indexed list of legal options generated by the rules engine. Agents therefore do not need to produce command syntax or discover the legal action set; they must rank legal choices of different kinds whose consequences are resolved by the simulator as illustrated in 1 .

The benchmark has two tracks. The Encounter track tests local tactical play in single fights: positioning, target priority, reactions, spell interactions, party coordination, boss mechanics, and objectives. The Day track links encounters through persistent party state, scarce resources, preparation choices, and short rests. This is where “win the fight” and “win the day” diverge. A high-level spell, an ability, a hit die, or a consumable can be correct to spend now if it ends a fight, or wrong if it leaves the party unable to answer a later room. DungeonBench makes that pressure visible and measurable by carrying party state, resources, active effects, and rest choices across encounters rather than treating each battle as an isolated episode.

We make three contributions. (i) We introduce DungeonBench , a combat-rules engine and benchmark for reasoning over the 2014 D&D tactical kernel, in which agents rank state-dependent options enumerated by the engine. (ii) We introduce Encounter and Day tracks that test local tactics, reactions, spell and item semantics, boss mechanics, objectives, preparation, and medium-horizon resource budgeting. (iii) We provide direct heuristic, text-prompt, tensorized, and Gymnasium-compatible masked-action interfaces over the same engine-generated decision stream, and use this shared surface to evaluate frontier language-model policies.

2 Related Work

RL environments and benchmark suites. OpenAI Gym and successor interfaces established a common pattern for environment interaction (Brockman et al. , 2016 ) . PettingZoo extends related abstractions to multi-agent environments (Terry et al. , 2021 ) , while OpenSpiel collects game-theoretic environments and algorithms (Lanctot et al. , 2019 ) . Procgen emphasizes generalization across generated levels (Cobbe et al. , 2020 ) , and Jumanji provides scalable environments for combinatorial decision problems (Bonnet et al. , 2024 ) . These suites provide reproducible interaction protocols and comparable agent interfaces across broad classes of tasks. DungeonBench follows this benchmark tradition, but centers evaluation on rule-mediated tactical option valuation.

Capability-focused game benchmarks show how a game environment can be useful because it isolates a particular form of reasoning. Hanabi foregrounds belief, intent, and ad-hoc cooperation (Bard et al. , 2020 ) . DungeonBench is analogous in spirit, but isolates tactical choice under structured combat rules rather than hidden-information cooperation.

Tactical and symbolic planning benchmarks. SMAC and SMACv2 evaluate multi-agent tactical micromanagement in StarCraft scenarios (Samvelyan et al. , 2019 ; Ellis et al. , 2023 ) , and MicroRTS provides a lightweight real-time strategy benchmark (Huang and Ontañón, 2021 ) . MiniGrid and Boxoban/Sokoban-style domains provide compact planning problems with controllable structure (Chevalier-Boisvert et al. , 2018 ; Guez et al. , 2019 ) . PDDLGym bridges reinforcement learning and classical planning with relational observations and dynamically valid actions from PDDL domains (Silver and Chitnis, 2020 ) . NetHack, MiniHack, Crafter, Craftax, and MineRL expose agents to large combinatorial state spaces and long horizons (Küttler et al. , 2020 ; Samvelyan et al. , 2021 ; Hafner, 2021 ; Matthews et al. , 2024 ; Guss et al. , 2019 ) . The Factorio Learning Environment evaluates long-horizon resource optimization through factory construction in a procedurally generated world, with task difficulty scaling through automation depth (Hopkins et al. , 2025 ) .

D&D combat simulation. FIREBALL and CALYPSO show that D&D is a fertile AI domain because structured game state, natural-language play, and human tabletop workflows meet in one setting. FIREBALL collects actual-play sessions with structured game state for language and command-generation tasks (Zhu et al. , 2023a ) , while CALYPSO studies LLM interfaces for dungeon masters (Zhu et al. , 2023b ) . Recent D&D combat work includes an RL environment with LLM-controlled adversaries in restricted scenarios (Dayo et al. , 2025 ) and NTRL, which uses reinforcement learning for encounter generation and dynamic difficulty adjustment rather than agent tactical control (Romeo and Bagdanov, 2025 ) .

Language models as decision policies. Frontier language models are increasingly used as policies over structured choices: they read state, deliberate over candidates, and return a selected action (Yao et al. , 2023 ; Wang et al. , 2023 ; Wu et al. , 2024 ; Paglieri et al. , 2025 ) .

DungeonBench differs from prior environments by focusing on tactical control over a broad executable combat surface. It deliberately retains mechanics that simplified combat benchmarks often abstract away: level-1-to-20 level class and subclass features, multi-classing, conditions, boss mechanics, and encounter-to-encounter resource persistence. These mechanics are not edge clutter; they are the interactions that make tactical option valuation difficult.

3 Benchmark Design

3.1 D&D combat background

Dungeons & Dragons is a tabletop role-playing game whose combat rules specify how player characters, monsters, spells, equipment, terrain, and battlefield effects interact. DungeonBench is grounded in the 2014 System Reference Document (SRD), the publicly available rules corpus for that edition. The 2014 SRD includes twelve base classes, with subclasses adding further rule packages. These choices are mechanically consequential: a fighter, cleric, rogue, and wizard differ not only in theme or statistics, but in the actions they can take, resources they spend, and tactical problems they create. Monsters are defined by stat blocks with hit points, defenses, actions, traits, reactions, and sometimes special mechanics such as legendary actions.

D&D combat is organized into rounds, with creatures acting in initiative order. A turn is not a single atomic action: a creature may split movement around an action, use a bonus action when available, interact with objects or terrain, and trigger reactions from other creatures. Tactical state also persists across turns and encounters through hit points, spell slots, class-feature uses, item charges, conditions, concentration effects, and rests. A short rest represents about one hour and restores some, but not all, resources. Thus combat choices are not only immediate damage choices; they depend on timing, action economy, interrupt windows, positioning, and resource allocation across future decision points.

3.2 Executable combat scope

DungeonBench targets the combat-relevant portion of the 2014 D&D SRD. The design rule is simple: if a spell, feature, item, monster trait, or combat rule can matter tactically and can be represented by the engine without inventing an open-ended ruling, it belongs in scope. Effects whose primary consequence depends on dungeon-master adjudication, campaign fiction, or creature interpretation are excluded or reduced to mechanically specified cases. For example, illusion spells such as Minor Illusion are outside the main benchmark contract when their value depends on what a creature believes. The open-ended reality-rewriting use of Wish is likewise outside scope, while mechanically specified uses such as spell duplication or rollback-style effects are modelable.

This boundary is meant to preserve breadth, not to make the simulator small. DungeonBench includes rules that are often dropped from simplified combat environments: reactions, concentration, persistent zones, summons, legendary actions, object interactions, mounted combat, underwater combat, flight, falling, vertical range, and hard-to-place effects such as Wall of Force . The benchmarked rule surface contains 561 combatants, including 93 spellcasters and 19 combatants with legendary actions; 229 spells; 275 items; and class, subclass, ancestry, monster, condition, and persistent-effect mechanics. These numbers characterize the executable rule surface available to the task registry rather than a claim that every tabletop use of every rule is modeled.

3.3 Data-driven engine and battlefield

DungeonBench is both a fixed benchmark suite and a data-driven tactical-combat engine. Combatants, maps, spells, items, objectives, persistent effects, and encounter days are authored as data and resolved by a shared rules engine. This matters because the SRD supplies the underlying rule vocabulary—classes, subclasses, spells, items, monsters, conditions, and combat procedures—but not benchmark-ready parties, tactical maps, objectives, encounter-day sequences, or complete player-character rosters. DungeonBench therefore composes player archetypes and benchmark scenarios from SRD mechanics, adding authored maps, objectives, and bosses where needed to create controlled tactical tasks. Examples include a draconic sorcerer, an epic paladin/sorcerer multiclass, objective-oriented rogues and casters, and bosses such as the Tempest Lictor and the Black Bell of Ordrune. New scenarios can be built by composing existing abilities, rosters, maps, objectives, and encounter-day structure rather than changing the benchmark interface.

The battlefield uses a 2.5D grid: positions have horizontal coordinates and height. This keeps movement and targeting executable while preserving tactical effects that depend on verticality. A flying creature can gain line of sight over cover, fall, fight above a ground target, or cast an area spell downward. Terrain can block sight or movement, provide cover, create hazards, support mounted or underwater fights, and constrain vertical movement. The point is not graphical fidelity; it is to keep height, range, flight, falling, and upward or downward area templates inside the executable rules rather than flattening them away.

3.4 Observation and decision interface

The main observation mode is a complete tactical observation as shown in Table 1 : the agent receives a serialized state sufficient for tactical decision-making, excluding non-semantic bookkeeping. The observation includes combatant statistics, hit points, spell slots, feature uses, positions, map state, conditions, active effects, zones, summons, objectives, encounter-day state, initiative context, and the current decision. These fields determine whether a creature can hold a doorway, whether a spell can affect a target, whether a reaction is available, and whether spending a resource now changes the next fight.

Full observability is deliberate. The main benchmark asks whether a policy can use the relevant tactical facts once they are available, not whether it can infer hidden monster traits, remember a scouted room, or guess that a harder fight is coming. Hidden-information and scouting variants are natural extensions of the same interface, but the reported benchmark fixes information access so that failures reflect tactical valuation rather than perception or memory.

As mentioned earlier, D&D combat also does not fit a single atomic action interface. A turn can interleave movement, action, bonus action, object interaction, and more movement; a creature’s movement can trigger another creature’s opportunity attack; a spell can trigger Counterspell , which can itself be countered; a failed save can trigger an optional resource spend; a boss can act in another creature’s turn through a legendary action. DungeonBench therefore exposes combat as a sequence of rules-generated decisions rather than asking an agent to emit a full turn script.

3.5 Hierarchical action decomposition

The broad rule surface creates an action-space problem. A flat action space over movement, spell choice, slot level, targets, area placement, area orientation, summon placement, reaction use, and follow-up choices would be too large to present to agents and too brittle to enumerate by hand. This is especially acute in 3D. A spell with 150 feet of range at 5 feet per tile spans 61 coordinate values along each axis in a naive centered bounding cube, or more than 226,000 candidate origins before legality filtering. Even a flat plane contains 3,721 possible origins before considering other action parameters.

Point-targeted and area-origin effects illustrate the design. A turn-action option may already distinguish Fireball cast at different legal slot levels, but the area origin is not flattened into that same option. Instead, DungeonBench can expose point selection coordinate-wise: choose an x x value, then a y y value conditioned on x x , then a z z value conditioned on ( x , y ) (x,y) . The selected triple is still a legal simulator point, but the policy sees a smaller sequence of choices. This supports point-origin spells, placement effects, and summons while preserving vertical tactics.

Directional templates use the same approach. For cones, lines, and similar effects, the policy chooses a direction by selecting d ​ x dx , then d ​ y dy , then d ​ z dz from canonical legal directions. This allows effects such as Cone of Cold to remain cones in a 3D battlefield. A flying caster can aim downward, producing a different tactical footprint than a ground-level horizontal cone, without adding a free-form geometric language or flattening the spell into a simplified radius. The tradeoff is that one tabletop action may become several benchmark decisions, but this is the mechanism that lets DungeonBench retain rule breadth while keeping every choice executable and legal.

Movement creates a different decomposition problem because its consequences can depend on the route, not only the endpoint. Moving to the same destination can provoke different opportunity attacks, cross different hazards or zones, spend different movement, or interact differently with vertical terrain. DungeonBench therefore presents movement first as a reachable destination choice, then, only when needed, as a route choice. For a selected destination, the engine groups route variants by simulator-visible consequences and removes duplicates with identical consequences. The grouping is equivalence-based: routes are merged only when they expose the same simulator-visible consequences to the policy and engine. If several distinct routes remain, the agent sees the relevant path facts and chooses among them; if only one remains, movement resolves immediately. This preserves tactical route differences without exposing every geometrically distinct path through the 3D battlefield, making flight, swimming, climbing, and other 3D movement modes viable inside the same action interface.

3.6 Tracks and task families

DungeonBench defines two benchmark tracks. Encounter tasks test local combat planning, positioning, target selection, reactions, terrain use, spell interactions, and occasional objective play. In this setting, spending a high-level resource immediately can be correct if it wins the fight. Day tasks link encounters through persistent party state and limited short rests, testing whether a policy can carry tactical judgment across several rooms. The same spell slot that ends a wolf skirmish quickly may be the resource needed to survive a later boss. A druid’s Natural Recovery choice after a short rest can likewise determine whether the party regains a control spell for the next room or preserves recovery capacity for a harder encounter later. The two tracks therefore separate immediate tactical competence from resource-aware tactical planning.

Tasks combine combatants, spells, maps, objectives, encounters, and encounter-day scenarios. The suite is organized by tactical skill rather than rule-surface size: each task family isolates a different form of choice pressure while sharing the same API. Some tasks reward clean target priority; others reward countering an enemy caster, protecting concentration, timing a reaction, exploiting terrain, or declining an attractive action because the next fight matters more. Objective tasks include movement and object-interaction goals, where class features such as a thief rogue’s Fast Hands can change the value of reaching and activating a scene object. The scenarios cover reaction control through broken sightlines, gaze monsters where cover matters more than damage, underwater and vertical terrain, swarms and body-control threats, possession and affliction recovery, item-driven objectives, boss and lair tempo, and linked days that combine several of these pressures behind limited rest windows.

Day regimes. The Day track uses a known encounter-day setting in which the encounter sequence is visible to the agent. This is deliberate. The benchmark is not asking whether an agent can guess that a harder room exists; it is asking whether the agent can act on that knowledge. The agent can see the linked fights but must decide how to allocate hit points, spell slots, consumables, short rests, and long-duration effects across them. Known future pressure becomes part of present action choice, making resource budgeting an observable tactical problem rather than hidden-information inference.

Intermission abstraction. Between encounters, the Day track exposes an intermission decision phase for healing, consumables, preparation actions, persistent summons, short-rest choices, and long-duration buffs. The abstraction preserves duration-relevant tactical distinctions without modeling a full dungeon clock. Very short combat-only effects expire before the next encounter; next-encounter effects carry into one subsequent fight; one-hour effects persist until a short rest or equivalent time advance; and adventuring-day effects such as long-duration buffs persist across short rests. Some resources refresh on short rests while others do not, so limited rest windows create planning pressure over hit points, hit dice, spell slots, class features, consumables, and persistent effects. This makes preparation and rest timing part of the same decision problem as combat positioning and target choice: casting a high-level Magic Missile into an easy first encounter can be locally effective and strategically wrong. The bounded intermission model keeps episodes reproducible while preserving the resource-pressure structure of the source game.

Reading a trajectory. DungeonBench keeps this tactical layer deterministic and records both combat events and day-level state changes. The following excerpts illustrate the two tracks.

Encounter example. In Moonhook Eyrie, a ranger opens with Volley against a wyvern and two air elementals. An elemental uses Whirlwind , pushing Vael 20 feet and dealing 18 bludgeoning damage; Vael answers by casting Feather Fall on Aurel, slowing the descent. Aurel’s Guiding Bolt hits the wyvern for 31 radiant damage. Vael then casts Fireball , catching the wyvern and both elementals; on the next round, after the manticore hits Vael three times, Aurel spends a 6th-level Healing Word to restore 30 hit points before Vael retreats and defeats the wyvern with another Fireball . Day example. Graywatch March begins with four party members and one short rest. After North Nave, Cassian is at 19/33 hp and Nix at 30/38 hp. The party spends its only short rest: Nix spends a hit die to heal to 38/38, while Cassian spends hit dice to return to 33/33. After Lower Yard, Cassian drops to 8/33 and Nix to 32/38; with no short rest left, Ilyra uses Lay on Hands before the final Ridge Room. The day is cleared, but the final party is badly depleted: Caldrin 17/44 hp, Ilyra 0/44 hp, Cassian 0/33 hp, and Nix 0/38 hp.

Scenario registry. Each benchmark scenario is identified by a stable ID and binds together map, side rosters, seeds, round limit, reward, decision interface, and observation mode. The headline evaluation suite contains 25 scenarios total: 20 Encounter scenarios and 5 Day scenarios. Encounter evaluation uses three seeds per scenario, while Day evaluation uses one fixed seed per day, for 65 headline evaluation episodes. Encounter scenarios vary party and enemy composition, map geometry, objectives, environmental conditions, and boss or lair mechanics; Day scenarios vary encounter sequence, short-rest pressure, preparation opportunities, objective structure, and final boss difficulty. Detailed lists of both tracks are provided at Table 6 and 7 .

3.7 Shared policy surface

The same decomposed decision stream is the shared policy surface for DungeonBench. In the reported setting, symbolic baselines and language-model policies choose among engine-generated legal options; the simulator executes the selected option, so comparisons measure tactical option valuation rather than command parsing or action validity. The released interface also exposes the same decision stream as tensorized variable-option examples and as a Gymnasium-compatible wrapper for learned rankers and reinforcement-learning agents. Appendix A documents these text and tensor interfaces.

Language-model policies and rule lookups. Language-model policies use the shared interface through a structured text rendering. The prompt contains the current decision kind, actor, tactical state summary, encounter history, legal option identifiers, and a response schema requiring a legal choice ID. Because rule details can be too large to place inline at every decision, the prompt exposes visible rule and item indices and permits lookup tools for attacks, spells, features, traits, combat riders, items, conditions, and movement routes. Lookups return simulator-grounded mechanics, including route-specific consequences such as opportunity attacks, hazards, readied reactions, and zone triggers. Invalid responses are retried against the same schema, and no response is executed unless it selects a legal option.

4 Evaluation

The reported evaluation focuses on frontier language-model policies under a shared decision interface. In every episode, the tested model controls the benchmark party, and all opposing sides are controlled by the same heuristic planner. This fixes the adversary and isolates the question of interest: given the same complete tactical state and the same legal option set, how well does a model value executable tactical choices?

4.1 Evaluation Protocol

Each language-model policy receives a serialized tactical observation, the pending decision, recent encounter history, visible rule and item indices, and a list of legal option identifiers generated by the simulator. The model does not emit free-form commands. It must select one legal option identifier, and the simulator executes that option. The policy is invoked at every exposed decision point, including movement, turn actions, reactions, optional riders, target allocation, area placement, and day-track intermission choices.

Rules and route details are exposed through lookup tools rather than copied into every prompt. These tools return simulator-grounded mechanics for spells, attacks, features, traits, items, conditions, and movement routes. Route lookups can reveal movement cost, opportunity attacks, hazards, zone triggers, readied reactions, and other route-specific consequences.

The encounter-track comparison evaluates five models: GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, Grok 4.3, and DeepSeek V4. Because linked-day episodes expose several hundred surfaced decisions and dominate evaluation time and cost, the Day-track comparison evaluates each scenario on one fixed seed and uses Claude Sonnet 4.6 as the Anthropic model in place of Claude Opus 4.7.

The Encounter track uses 20 canonical encounter scenarios and the Day track uses 5 linked encounter-day scenarios listed at Table 6 and 7 . Each day consists of 3 to 5 encounters with persistent party state, intermission decisions, and limited short rests. Encounter scenarios are evaluated with 3 seeds each; Day scenarios are evaluated with 1 fixed seed each. The day sequence is known to the agent. This makes resource planning an observable tactical problem rather than a hidden-information problem.

The primary Encounter metric is win rate: the party must defeat the opposition or satisfy the scenario-specific objective before a wipe or timeout. Secondary Encounter metrics report the final party hit-point fraction, survivor fraction, rounds to termination, and decisions per episode. These distinguish clean wins from depleted wins and short failures from long failures.

The primary Day metric is full-clear rate: the party must clear every encounter in the linked day. Secondary Day metrics report final party hit-point fraction, survivor fraction, decisions per episode, short-rest use, and encounters won.

4.2 Results

Tables 1 and 2 report the frontier language-model comparison on the Encounter and Day tracks. The headline result is that strong single-encounter performance does not transfer cleanly to linked encounter days. The best performer models on Encounter track clear slightly more than 80% of episodes, while the best Day models clear only two of five days.

On the Encounter track, Gemini 3.1 Pro and GPT-5.5 are the strongest models, with win rates of 83 ± 5 83\pm 5 and 82 ± 5 82\pm 5 , respectively. They also preserve the most party health, ending with about 0.80 final HP and 0.93 survivor fraction on average. Grok 4.3 is lower at 72 ± 6 72\pm 6 , and Claude Opus 4.7 and DeepSeek V4 are tied at 68 ± 6 68\pm 6 . Secondary metrics show that performance is not explained by tool use alone: DeepSeek V4 uses the most tools and tokens per decision but does not achieve a higher win rate, while GPT-5.5 reaches near-top success with the fewest decisions per episode among the five models.

Table 1: Frontier language-model policies on the Encounter track. The opponent side is fixed to the heuristic planner across all evaluations. Win rate is reported with one standard error over the evaluated encounter episodes. Final HP is the party hit-point fraction at encounter end, and Survivors is the fraction of party members alive at encounter end. Decisions/ep. is the mean number of surfaced decisions per episode. Tools/dec. and Tokens/dec. summarize lookup-tool use and prompt-plus-completion tokens per surfaced decision.

Model Win rate Final HP Survivors Decisions/ep. Tools/dec. Tokens/dec.

Gemini 3.1 Pro 83 ± 5 83\pm 5 0.80 0.93 93 1.62 42k

GPT-5.5 82 ± 5 82\pm 5 0.79 0.93 79 1.38 15k

Grok 4.3 72 ± 6 72\pm 6 0.59 0.78 102 1.17 16k

DeepSeek V4 68 ± 6 68\pm 6 0.67 0.82 104 3.27 79k

Claude Opus 4.7 68 ± 6 68\pm 6 0.65 0.82 109 0.52 26k

The Day track is substantially harder. GPT-5.5 and Gemini 3.1 Pro each clear two of five days, Claude Sonnet 4.6 and DeepSeek V4 each clear one, and Grok 4.3 clears none. The decision horizon is also much longer: day episodes average roughly 328–391 surfaced decisions per episode, compared with 79–109 on the Encounter track. This gap is central to the benchmark. The model must not only choose locally useful actions, but also preserve hit points, spell slots, consumables, and recovery opportunities across several fights.

Final HP and survivor fractions on the Day track should be read together with full-clear rate. Some failed days end with substantial remaining party health, which indicates objective or sequencing failures rather than gradual wear-down. Conversely, some days are nearly cleared before a final wipe, showing that partial encounter progress is not equivalent to day success. The Day track therefore exposes failures that are largely invisible in isolated encounters.

Table 2: Frontier language-model policies on the Day track. Each day chains multiple encounters with persistent state, intermissions, and limited short rests. The opponent side is fixed to the heuristic planner across all evaluations. Day clear is the full-clear rate over the five evaluated day scenarios. Final HP is the party hit-point fraction at day end, and Survivors is the fraction of party members alive at day end. Decisions/ep. is the mean number of surfaced decisions per day episode. Tools/dec. and Tokens/dec. summarize lookup-tool use and prompt-plus-completion tokens per surfaced decision.

Model Day clear Final HP Survivors Decisions/ep. Tools/dec. Tokens/dec.

Gemini 3.1 Pro 0.40 0.40 0.58 0.80 328 1.27 124k

GPT-5.5 0.40 0.40 0.38 0.55 391 0.77 114k

DeepSeek V4 0.20 0.20 0.50 0.65 354 2.16 102k

Claude Sonnet 4.6 0.20 0.20 0.50 0.60 354 0.91 70k

Grok 4.3 0.00 0.00 0.20 0.20 361 0.63 163k

The terminal traces show that losses are qualitatively different. Of the 76 failed Encounter episodes, 38 are party wipes, 25 end with the party essentially intact but failing an objective, or losing the race to a non-HP win condition, and 13 are depleted losses with intermediate remaining HP. unseenpredator is the clearest wipe-driven failure: the party repeatedly fights a hidden, resistant enemy under unfavorable attack conditions. westtower is different: many losses occur before the party is meaningfully damaged because the opposing side secures the map objective first. stonecut_bridge exposes a third pattern, where the party often remains alive but spends too many turns on low-value movement and attacks. These distinctions are important because a single binary loss does not say whether the model misunderstood the rules, ignored the objective, chose inefficient routes, or simply lost an attrition fight.

Table 3 gives statistics of the decision space that help to explain why these failures persist under full observability. Objective encounters have the longest average horizon, while Boss and Objective encounters expose the largest local option sets. The median decision is much smaller than the tail, but some decisions expose hundreds of legal choices and the maximum option count exceeds 1,000. The model therefore has to solve two problems at once: choose a good local commitment from a large legal set, and understand how that commitment advances the scenario-level objective.

The Day track compounds these pressures. kestrel_approach is the cleanest example: every model fails the day while preserving the party, indicating an objective-recognition failure rather than a resource-depletion failure. Other days show the complementary pattern, where models win several encounters before the chain fails near the end. This is the intended distinction between the tracks. Encounter performance measures local tactical competence; Day performance measures whether that competence remains disciplined when hit points, spell slots, rests, and objective progress carry forward.

Table 3: Decision-space characterization of the Encounter track. Decisions/ep. is the mean number of surfaced decisions per episode. Options/dec. is the mean number of legal candidates exposed at each decision point, and Max options is the largest legal option set observed in the family. Objective encounters have the longest average horizons, while both Objective and Boss encounters expose large local choice sets.

Family Decisions/ep. Options/dec. Max options

Duel 52 67.7 169

Objective 112 47.3 1159

Boss 60 52.2 441

5 Limitations

DungeonBench scopes evaluation to tactical combat. It is designed for controlled comparison of rule-aware tactical decisions, not full tabletop play: social interaction, open-ended exploration, campaign-scale simulation, narrative adjudication, or broad party and build optimization. The main benchmark uses complete tactical observations and known encounter-day sequences. Strong DungeonBench performance should therefore be read as evidence about rule-aware tactical reasoning in this setting, not as a claim about all tabletop play or all long-horizon reasoning.

Future extensions. Several natural extensions would test capabilities outside the current combat-focused benchmark. A Dungeon track could chain multiple encounter days, extending the current Day setting to longer horizons. This would stress planning across long rests, spell preparation, consumable use, persistent injuries or afflictions, and the choice of which resources to commit when later days remain. Such a track could also introduce bounded noncombat choices, such as scouting, detours, stealth, and information-gathering spells like Arcane Eye , Speak with Dead , or Speak with Animals , without requiring full open-ended campaign simulation. Hidden-information variants could restrict what agents know about enemy traits, unseen positions, future rooms, objectives, immunities, and other tactical facts until they are discovered or inferred. Multi-agent variants could assign separate policies to party members, monsters, or opposing sides, enabling coordination, adversarial play, self-play, and population-based training.

6 Conclusion

DungeonBench provides a rules-rich tactical-combat benchmark for evaluating agents under heterogeneous state-dependent decisions, long horizons, and compositional rules. Its core setting focuses on fully observed tactical decision-making, curated scenario suites, and engine-generated legal options. The benchmark asks whether agents can reason over tactical choices whose meaning depends on geometry, action economy, spell and item semantics, objectives, reactions, and persistent resources. The resulting task is deliberately close to the texture of tactical play: survive the current fight, advance the objective, respect the rules, and keep enough resources to matter later. DungeonBench makes those tactical commitments executable, repeatable, and measurable. Broader framework integrations beyond the current Gymnasium-compatible interface would make the same decision stream easier to use in external RL libraries and multi-agent experiments.