Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures

\contact {harsh.raj, vipul.gupta, anas.mahmoud}@scale.com Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures

Harsh Raj

Vipul Gupta

Anas Mahmoud

Razvan-Gabriel Dumitru

Darvin Yi

Aakash Sabharwal

Yunzhong He

Abstract Existing evaluations often reduce agent failures to system-level outcomes, obscuring where the fault originated and which intervention would actually improve the next iteration of the agent system. This creates a repair-assignment problem: the same visible failure may call for model post-training, harness engineering, environment redesign, or benchmark repair depending on where it originated. Because an agent’s behavior emerges from interactions among its model, harness, users, tools, memory, and environment, outcome-level labels alone are often insufficient for improving agent performance. Most failure taxonomies do little to resolve this problem because they are typically benchmark-specific, capturing useful fine-grained failure modes without providing a shared structure. We introduce an interaction-centric taxonomy that localizes agent failures to the interaction in which they originate and identifies the component responsible. We treat interactions between components as the unit of analysis. The taxonomy organizes 41 failure modes by assigning each failure to an edge between two components and a fault side indicating where the repair belongs. This makes the taxonomy directly actionable: model-side failures identify targets for post-training, harness-side failures point to scaffolding and tool-integration fixes, and environment or grader failures reveal evaluation conditions that must be redesigned before they are used to judge agent capability. The schema applies across agent architectures, from coding assistants to long-horizon personal assistants and multi-agent systems. We ground the taxonomy in worked examples from public benchmarks, model system cards, published reports, and logged agent trajectories, and evaluate its operational reproducibility using independent reasoning agents as judges. Across four frontier models, the judges recover the human labels well above chance, with the strongest judge reaching Cohen’s κ = 0.76 \kappa=0.76 against human category labels, suggesting that the categories capture shared structure rather than annotator-specific labeling preferences.

1 Introduction

As LLMs are deployed in increasingly long-running and autonomous settings (Anthropic, 2026c ) , they interact repeatedly with users, tools, memory, harness, and environment. This broader interaction surface expands the failure surface of the agent system: when an agent fails under such complex scenarios, it is often difficult to determine where the failure originated and which component should be repaired (Zhang et al. , 2025a ; Cemri et al. , 2025 ; Zhu et al. , 2025a ) . For example, in a long-running Claude Code (Anthropic, 2025a ) session an agent may ignore an earlier user instruction because the harness’s context compaction removed it, or because the instruction remained available but the model failed to follow it. The observed behavior is the same, but the first case requires a harness-level fix, whereas the second requires a model-level intervention. Understanding agent failure modes is therefore essential for selecting effective post-hoc interventions, such as model post-training, harness engineering, environment redesign, or benchmark repair.

Component

Definition

Model

The policy that processes observations and produces outputs or actions.

Owner

The human or upstream system that gives the agent its task and defines what counts as success.

Grader

The mechanism used to evaluate whether the agent completed the task successfully; it is usually not visible to the agent.

Third party

An actor encountered during execution that does not act on behalf of the owner. The actor can be a human, organization, or agent, and the interaction may be adversarial, persuasive, or cooperative.

Context

The information available to the model during the current interaction, including instructions, conversation history, observations, and summaries.

Memory

A persistent store that outlives the active context, within or across sessions.

Tool

The bidirectional interface through which the model exchanges requests, messages, actions, observations, and responses with other components. This includes callable tools, communication channels, and wrappers that relay inputs and outputs.

Local env.

The agent’s immediate execution environment, such as the operating system, shell, filesystem, and runtimes.

External env.

Systems outside the agent’s immediate execution environment, such as remote services, websites, APIs, databases, and model-provider infrastructure.

Table 1: Component vocabulary used to localize agent failures. Each row defines a component that can form an endpoint of an interaction edge. In multi-agent settings, peer and subagent describe the role of the other model. In subagent interactions, the focal model acts as the orchestrator, while in peer interactions, neither model directs the other.

Prior work has classified agent failures by the internal agent module affected (Zhu et al. , 2025a ) . Without an explicit way to distinguish where a failure surfaces from which component caused it, outcome-level failure labels collapse distinct causes together and direct repairs toward the wrong part of the system. We represent an agent system as a set of interacting components, listed in Table 1 . The model is the LLM policy, while the owner specifies the task and what counts as success. The grader evaluates the result, and third parties are other actors the agent interacts with during execution. The harness manages the model’s context, memory, and tool access. The environment covers both the agent’s local execution setting and the external services it uses. We analyze failures at the interaction between two components. This interaction defines the edge , and the component responsible for the failure defines the fault side . Consider an agent that reports that a tool call succeeded when it actually failed. In one case, the tool wrapper suppresses the error, so the model never observes the failure. We label this failure tool — model ⋅ \cdot fault: tool . In another case, the wrapper returns the error, but the model ignores it. We label this failure tool — model ⋅ \cdot fault: model . The interaction is the same, but the responsible component differs. To show that these labels capture shared structure rather than one annotator’s intuition, we evaluate their reproducibility with independent reasoning agents as judges. Across four frontier models, the judges recover the human labels well above chance, reaching a Cohen’s κ \kappa of 0.76 0.76 .

Recent advances in the multi-step reasoning and evidence-synthesis capabilities of LLMs have motivated their use as agentic evaluators that independently reconstruct evidence and apply explicit criteria (OpenAI, 2024 ; Snell et al. , 2025 ; Zhuge et al. , 2024 ) . We therefore treat each judge as an independent analyst and measure pairwise agreement to test whether they converge on the same labels given the same definitions and evidence. They agree with one another about as strongly as they agree with the annotators, with the highest pairwise agreement reaching Cohen’s κ \kappa of 0.84 0.84 .

A single trajectory often contains many cascading failures. Without a fixed rule for which one to label, annotators would score the same trace inconsistently. An initiating failure can propagate into several later errors. We therefore begin with the observed system-level failure and trace its causal chain backward. We label the earliest failure from which execution does not recover, rather than its downstream symptoms (Jorf and Shamout, 2026 ; Zhu et al. , 2026b ; Qiao et al. , 2026 ) . An intervention at this point would have resulted in a different outcome, whereas the later errors may only be consequences of it. The taxonomy applies wherever a model or group of models interacts with users, tools, environments, memory, or other agents. Even the minimal case of a single LLM answering a user’s question involves an interaction between the model and the user. The same vocabulary applies to coding agents such as Claude Code (Anthropic, 2025a ) and Codex (OpenAI, 2025 ) ; long-running personal assistants that read mail, browse the web, execute shell commands, and maintain persistent memory, such as OpenClaw (OpenClaw, 2025 ) and Hermes Agent (Nous Research, 2025 ) ; and custom multi-agent systems (Cemri et al. , 2025 ) . The taxonomy is modality-agnostic, with several worked examples drawn from multimodal settings.

This paper makes three contributions.

• First, we introduce an interaction-centric taxonomy of 41 agent failure modes, assigning each to an interaction edge and a fault side (Figure 2 ). Most modes are model-side, partly because our attribution rule assigns fault to the model when a more capable model could have avoided or recovered from the failure under the same conditions.

• Second, we ground the taxonomy in worked examples drawn from public benchmarks, model system cards, published reports, and logged agent trajectories, covering almost all of the failure modes.

• Third, we evaluate whether independent reasoning agents can consistently recover the human-assigned categories, providing evidence that the taxonomy captures a reproducible structure.

2 Related Work

Existing taxonomies typically focus on one part of the agent interaction surface. Some are tied to a particular benchmark (Deng et al. , 2025 ; Zhu et al. , 2026a ) , while others address a specific setting, such as coordination in multi-agent systems (Cemri et al. , 2025 ; Lin et al. , 2025 ) , or are presented as a flat list of failure modes (Vinay, 2025 ) . These approaches are valuable within their intended scope. But none of them indicates which component is at fault, and therefore which kind of intervention a failure calls for. A general framework should map each failure to the intervention it needs, such as model post-training, harness engineering, or environment redesign. A coarse label such as Execution Failure , for example, can conflate an unrecoverable external-service failure with a model giving up after a transient error that it could have retried or routed around. The visible outcome may be identical, but the former requires repairing the external system, whereas the latter requires improving the model’s recovery policy.

Work defining failure taxonomies specifically for agent systems comes closest to ours. Cemri et al. ( 2025 ) analyze a large set of multi-agent traces and, derive a taxonomy comprising system-design failures, inter-agent misalignment, and task-verification failures. Their inter-agent category distinguishes mechanisms such as withholding a message, ignoring a message, and losing shared context. Our representation is complementary: these failures can occupy the same interaction edge while differing in which endpoint is responsible. Zhu et al. ( 2025a ) divide a single agent into memory, reflection, planning, action, and system-level operations and classify errors according to the affected module. In our framework, planning, reflection, and action selection remain part of the LLM policy. Persistent memory stores, tool interfaces, graders, users, and environments are instead represented as separate components of the agent system. The security literature instead organizes failures by threats and consequences (Microsoft AI Red Team, 2025 ) . Shah et al. ( 2026 ) distinguish fault types, symptoms, and root causes in open-source agent systems and frequently identify causes at producer–consumer boundaries. Our edge and fault-side representation makes the two endpoints of such a boundary explicit.

Figure 1: Radial interaction map . The model is the hub, the User, Harness, and Environment families form the inner ring, and their components appear on the outer ring. Each failure is represented by an edge between two components. Since a model interacts with another model through its own harness, model–model interactions are grouped under Harness alongside Context, Memory, and Tool. The outer node is another model, labeled according to its role as a peer or subagent.

A complementary line of work studies failure localization in agent execution traces. Barke et al. ( 2026 ) identify the critical failure as the first unrecoverable event and reconstruct its causal relationship to the terminal outcome. Qiao et al. ( 2026 ) verify candidate failure hypotheses against the full interaction trace before attributing responsibility. These approaches address which event in a trajectory should be treated as causal. We adopt this root-cause view to determine which event receives a taxonomic label. Our taxonomy then addresses a separate question by identifying the interaction on which that failure occurred and the component responsible for it.

Within individual benchmarks, failure analyses are necessarily scoped to the tasks, interfaces, and evaluation procedures under study. This specialization is valuable because it reveals domain-specific failure mechanisms, and provides actionable guidance for improving agents in a particular setting. A coding benchmark may distinguish failures visible through tests and diffs, such as an incorrect patch or a missed file (Jimenez et al. , 2024 ; Deng et al. , 2025 ) , while a tool-use benchmark may characterize malformed calls and failures to recover from tool errors (Kokane et al. , 2024 ; Bandi et al. , 2026 ) . Yet no individual analysis captures the full failure surface of contemporary agents, which increasingly interact with users, context-management systems, persistent memory, tools, graders, local and external environments, and other agents. As this interaction surface expands, practitioners need a rigorous shared taxonomy that complements task-specific analyses and supports consistent diagnosis across systems.

Prior work primarily identifies what behavior occurred, which internal module was affected, or which trajectory event was decisive. Our framework is orthogonal: it identifies the causal event, localizes it to interaction between components, and determines where the intervention should be applied. This distinction helps separate failures that call for model post-training from those requiring harness engineering or closer scrutiny of the evaluation setup.

3 The Mechanism Axis

We represent each failure as an interaction edge paired with a fault side. The edge identifies the two components involved, while the fault side identifies the component responsible.

Components.

We model an agent as a set of interacting components, defined in Table 1 . Most of the component boundaries are straightforward, but the distinctions between the owner and grader, and between third parties and the external environment, need further clarification. We treat the grader as separate from the owner because the model can fail in its interaction with the grader independently of whether it followed the owner’s instructions. For example, in the Specification Gaming case E12 , an agent instructed to win against a chess engine edited the board state until the opposing engine resigned. The grader recorded a win even though the agent had bypassed the intended game. In the case of third-party interactions, the key distinction is whether the failure arises from the actor or from the external system through which the interaction occurs. The external environment is the delivery channel, whereas the third party is the actor behind the interaction. A system failure or stale response belongs to the external environment, whereas a failure caused by an actor attempting to influence or manipulate the model belongs to the third party. We group the components into three families: User, Harness, and Environment. Each family captures interactions between the focal model and the surrounding components listed in Table 1 . In a multi-agent interaction, the other endpoint is also a model. We therefore represent such interactions on the model — model edge and specify the role of the other model, as in model — model (role: peer ) or model — model (role: subagent ) . We treat peer and subagent as roles rather than components because the component at either endpoint remains a model. The role only specifies how that model participates in the interaction. Figure 1 visualizes this structure as a radial map, with the model at the hub, the three families on the inner ring, and their components on the outer ring.

Localizing a failure.

We write a failure as

comp 1 ​ — ​ comp 2 ⏟ edge ⋅ fault: ​ side ⏟ component at fault \underbrace{\textsc{comp}{1};\text{\textemdash};\textsc{comp}{2}}{\text{edge}};\cdot;\underbrace{\text{fault:}~\textsc{side}}{\text{component at fault}}

where the edge comp 1 ​ — ​ comp 2 \textsc{comp}{1},\text{\textemdash},\textsc{comp}{2} is the interaction between the two components and side is the component at fault. For example, tool — model ⋅ \cdot fault: model assigns the failure to the model side of the interaction between the model and the tool.

When several errors contribute to the final outcome, we use a fixed attribution rule. Starting from the observed system-level failure, the preceding events are traced backward to identify the earliest failure from which execution does not recover. Later errors are treated as consequences, and the taxonomy label is assigned to the interaction in which the earliest unrecovered failure occurred.

Model LLM policy User Harness Environment Owner Grader Third party Context Memory Tool Model (peer/subagent) External Local 1 Over-initiative , Trinh et al. ( 2026 ) 2 Under-initiative , Röttger et al. ( 2024 ) 3 Satisficing , Anthropic ( 2026c ) 4 Instruction-Following Failure , Zhou et al. ( 2023 ) 5 Reasoning Failure , Mirzadeh et al. ( 2025 ) 6 Unauthorized Irreversible Action , Ruan et al. ( 2024 ) 7 Sycophancy , Perez et al. ( 2023 ) 8 Domain Knowledge Deficit , Huang et al. ( 2025 ) 9 Value Misalignment , Chiu et al. ( 2025 ) 10 Instruction–Grader Mismatch , Bercovich ( 2026 ) ; Zhu et al. ( 2025b ) 1 Specification Gaming , Skalse et al. ( 2022 ) 2 Evaluation Awareness , Needham et al. ( 2025 ) 1 Indirect Prompt Injection , Greshake et al. ( 2023 ) 2 Contextual Sycophancy , Sharma et al. ( 2024 ) 1 State Tracking Failure , Cemri et al. ( 2025 ) 2 Goal Drift , Arike et al. ( 2025 ) 3 Context Rationale Erosion , Li et al. ( 2026 ) 1 Missed Write , Garg et al. ( 2026 ) 2 State Staleness , Chao et al. ( 2026 ) 3 Overgeneralization , Lam et al. ( 2026 ) 4 Memory Rationale Erosion , Garg et al. ( 2026 ) 5 Pollution , Xiong et al. ( 2025b ) 6 Redundancy , Kim et al. ( 2026 ) 7 Missed Read , Garg et al. ( 2026 ) 8 Memory Following Failure , Garg et al. ( 2026 ) 1 Incorrect Tool Selection , Huang et al. ( 2024 ) 2 Tool Hallucination , Patil et al. ( 2024 ) 3 Tool Feedback Neglect , Zhou et al. ( 2024 ) 4 Tool Recovery Failure , Kokane et al. ( 2024 ) 5 Malformed Arguments , Li et al. ( 2023 ) 6 Suboptimal Arguments , Xiong et al. ( 2025a ) 7 Mistranslation , Taraghi et al. ( 2026 ) 1 Delegation Failure , Cemri et al. ( 2025 ) ; Xiong et al. ( 2026 ) 2 Communication Failure , Khatua et al. ( 2026 ) ; Ruan et al. ( 2026 ) 1 Recovery Failure , Zhu et al. ( 2026a ) 2 Service Failure , Liu et al. ( 2026 ) ; Kara et al. ( 2025 ) 3 Stale State Delivery , Mazumder and others ( 2026 ) 1 Observation Failure , Zhou et al. ( 2024 ) 2 Recovery Failure , Arora et al. ( 2025 ) Fault side ∙ \bullet model ▲ \blacktriangle other component Family User Harness Environment Figure 2: Interaction-centric taxonomy of 41 failure modes. Failures are organized by the family of the component interacting with the model: User, Harness, or Environment, and then by the specific component within that family. Each branch represents an interaction edge between the model and that component. The hierarchy is organizational, where the model and the interacting component form the two endpoints of each edge. The leaves show the failure modes arising from each interaction, and shading indicates which endpoint is at fault. Of the 41 role-specific failure modes, 36 are assigned to a model and five to surrounding components.

4 Categorization Methodology

We developed the taxonomy iteratively while reviewing failures from public benchmarks, model system cards, published reports, and logged agent trajectories. As new cases exposed overlaps or unclear boundaries, we refined the component definitions and failure modes. Once these definitions had stabilized, we froze the taxonomy and used that version for all reported labels and for the validation in § 6 . The final definitions are reproduced verbatim in Appendix B .

To assign labels consistently, we applied the root-cause principle of § 3 . For each example, we reviewed all available evidence in the trace or report and identified the observed system-level failure. We then traced the causal chain backward and selected the earliest failure from which execution did not recover. This procedure follows Barke et al. ( 2026 ) , which defines the critical failure as the first unrecoverable failure and reconstructs its causal relationship to the system-level outcome. After identifying the root-cause failure, we assigned it to the interaction edge on which it occurred, identified the fault side, and selected the corresponding failure mode. The supporting rationale for each label is provided in Appendix C . For failures with a clear safety or security impact, we add a separate impact annotation alongside the taxonomy label. We assign the most salient applicable category, drawing primarily from the OWASP Top 10 for LLM and Agentic Applications (OWASP Foundation, 2025 ; OWASP Gen AI Security Project, 2025 ) . Appendix C provides the complete mapping.

We selected examples that illustrate the taxonomy across a range of interaction edges and failure modes. The set is illustrative rather than exhaustive and should not be used to estimate the prevalence of individual failure modes. These examples also form the evaluation set in § 6 , where we test whether independent reasoning agents can recover the human-assigned labels from the frozen taxonomy definitions and the original source material.

5 Failure Families

Figure 2 presents the full taxonomy as a hierarchy. The model appears at the root, followed by the four component families, the component associated with each interaction edge, and the failure modes assigned to each component. Within each component, model-attributable failures are listed first, and shading indicates which side of the interaction is at fault. Multi-agent failures follow the same model-centered structure. The tree is read from the perspective of one focal model, and the peer or subagent at the other endpoint enters as a model in that role. The following subsections mirror this hierarchy, with one subsection devoted to each family. Appendix B provides the verbatim definition of every failure mode.

5.1 Users

model — owner .

This edge captures the relationship between an agent and its owner. The owner-side failure mode is Instruction–Grader Mismatch (Bercovich, 2026 ; Zhu et al. , 2025b ) , where the stated instruction differs from the owner’s actual intent, as reflected by a test suite, evaluator, or unstated expectation. The agent follows the instruction but is judged against that intent, due to which it’s the owner’s fault. The model-side failure modes are Over-initiative (Trinh et al. , 2026 ) , acting beyond the scope it was given, whether by taking a consequential action where a clarifying question was due, or by volunteering information it was never asked for; Under-initiative (Röttger et al. , 2024 ) , withholding action or demanding confirmation it does not need; Satisficing (Anthropic, 2026c ) , cutting scope to finish sooner and declaring a task done while real work remains; Instruction-Following Failure (Zhou et al. , 2023 ) ; Reasoning Failure (Mirzadeh et al. , 2025 ) ; Unauthorized Irreversible Action (Ruan et al. , 2024 ) ; Sycophancy (Perez et al. , 2023 ) ; Domain Knowledge Deficit (Huang et al. , 2025 ) ; and Value Misalignment (Chiu et al. , 2025 ) , where a sound conclusion is reached through misaligned deliberation.

model — grader .

This edge captures failures in the model’s interaction with the evaluator rather than with the task itself. Both modes are model-attributable. Specification Gaming involves exploiting the evaluation or reward channel (Krakovna et al. , 2020 ; Amodei et al. , 2016 ; Skalse et al. , 2022 ; Mahmoud et al. , 2026 ) , whereas Evaluation Awareness (Needham et al. , 2025 ) occurs when the model behaves differently after inferring that it is being evaluated.

model — third party .

This edge covers failures in how the model interprets or responds to third-party content. Indirect Prompt Injection (Greshake et al. , 2023 ) occurs when the model treats directives embedded in third-party content as owner-authorized instructions. Contextual Sycophancy (Sharma et al. , 2024 ) occurs when the model aligns its response with a third party’s views or preferences instead of maintaining an independent judgment.

5.2 Harness

model — context .

This edge covers failures in how the active context is preserved and used. Goal Drift (Arike et al. , 2025 ) occurs when recent context displaces the original instruction, especially in long contexts where models may use information differently depending on its position (Liu et al. , 2023 ) . State Tracking Failure (Cemri et al. , 2025 ) occurs when the model repeats a subtask or action without recognizing that it is no longer making progress. Both are model-side failures because the relevant information remains available but is not used correctly. Context Rationale Erosion occurs when compaction removes information needed later, such as an important constraint. We attribute this failure to the harness when compaction is harness-driven, and to the model when compaction is model-driven (Li et al. , 2026 ) .

model — memory .

This edge covers failures in how the model stores information in persistent memory and uses it later (Packer et al. , 2023 ; Zhang et al. , 2025b ) . Memory Write Failure includes Missed Write (Garg et al. , 2026 ) , when relevant information is not stored; State Staleness (Chao et al. , 2026 ) , when stored information is no longer valid; Overgeneralization (Lam et al. , 2026 ) , when a specific observation is stored as a broader rule; Memory Rationale Erosion (Garg et al. , 2026 ) , when the reasoning behind stored information is lost; Pollution (Xiong et al. , 2025b ) , when incorrect or irrelevant information enters memory; and Redundancy (Kim et al. , 2026 ) , when the same information is stored repeatedly. Memory Read Failure includes Missed Read , when relevant memory is not retrieved, and Memory Following Failure , when retrieved information is not used correctly (Garg et al. , 2026 ) .

model — tool .

This edge covers failures in selecting a tool, constructing a tool call, and using its response. Tool-invocation failures include Malformed Arguments (Li et al. , 2023 ) , where the call does not follow the required format; Suboptimal Arguments (Xiong et al. , 2025a ) , where the arguments are valid but poorly chosen; Incorrect Tool Selection (Huang et al. , 2024 ) , where the model chooses the wrong available tool; and Tool Hallucination (Patil et al. , 2024 ) , where it attempts to use a tool that is unavailable. Response-handling failures include Tool Feedback Neglect (Zhou et al. , 2024 ) , where the model overlooks the returned information, and Tool Recovery Failure (Kokane et al. , 2024 ) , where it fails to adapt after an unsuccessful call. The tool-side failure mode is Mistranslation (Taraghi et al. , 2026 ) , where the integration layer incorrectly conveys an otherwise correct observation or action across the model–environment boundary.

model — model (role: peer/subagent ) .

Each model — model interaction is labeled by the role of the non-focal model. A peer is another agent that is part of the same workflow but is not invoked or directed by the focal model. For example, a solver and a verifier may be assigned their roles by the workflow and exchange outputs without either assigning work to the other. A subagent, by contrast, receives its role or task from the focal model, which defines the workflow and acts as the orchestrator.

• Peer. In peer interactions, Delegation Failure (Cemri et al. , 2025 ) occurs when a peer treats its assigned work as independent despite dependencies on another peer’s work, while Communication Failure (Cemri et al. , 2025 ; Khatua et al. , 2026 ) occurs when a model fails to share information needed by a peer.

• Subagent. In hierarchical systems, Delegation Failure (Xiong et al. , 2026 ) occurs when the orchestrator assigns work with incorrect scope or dependencies, while Communication Failure (Ruan et al. , 2026 ) occurs when the orchestrator omits necessary context or fails to use the subagent’s output, or when the subagent fails to report relevant results or constraints.

model — external environment .

This edge covers failures involving external services and the model’s response to them. Service Failure (Liu et al. , 2026 ; Kara et al. , 2025 ) occurs when an external service cannot complete a request. Stale State Delivery (Mazumder and others, 2026 ) occurs when a service reports a healthy status but returns outdated data. Both lie on the environment side because the problem originates in the service. Recovery Failure (Zhu et al. , 2026a ) is model-side when recovery remains possible but the model does not retry, diagnose the problem, or use an alternative route. If recovery is not possible, we attribute the failure to the external environment.

model — local environment .

This edge covers failures in how the model observes and responds to its local execution environment. Observation Failure (Zhou et al. , 2024 ) occurs when the model overlooks a cue available in its observation space. Recovery Failure (Arora et al. , 2025 ) occurs when the model fails to resolve a fixable local condition, such as a missing file or broken state.

6 Validating the Taxonomy with an Agent-as-a-Judge

We test whether independent reasoning agents can apply the taxonomy consistently to the same evidence. Each judge agent attempts to recover the human-assigned labels for the worked examples using only the taxonomy definitions and the original source material.

Task.

For each worked example, the judge receives the taxonomy definitions and a reference to the original failure source, but not the human-assigned label. The source may be a GitHub issue, blog post, model system-card section, arXiv paper, or logged agent trajectory hosted on platforms such as Hugging Face or Docent (Transluce, 2025 ) . The judge independently reviews the source, identifies the earliest failure from which execution does not recover, and predicts:

1. the interaction category, comp1 — comp2 ⋅ \cdot fault: fault ; and

2. the complete failure-mode label, comp1 — comp2 ⋅ \cdot fault: fault ⋅ \cdot Failure Mode .

   We run four frontier models as separate judges: GPT-5.5 and Claude Opus 4.6, 4.7, and 4.8. Full inference and harness configurations are provided in Appendix A .

   Pipeline.

   Unlike conventional LLM-as-a-judge systems, which evaluate candidate outputs by placing them directly in the evaluator’s context (Zheng et al. , 2023 ) , we use the agent-as-a-judge setup of Zhuge et al. ( 2024 ) . Each judgment is produced in three turns within a single session:

3. Evidence reconstruction. Given a reference to the original failure source, the judge retrieves the relevant evidence and organizes it into a neutral, chronological account.

4. Failure classification. Using the reconstructed account and the frozen taxonomy definitions, the judge identifies the earliest failure from which execution does not recover and assigns the interaction edge, fault side, and failure mode.

5. Reflection and disambiguation. The judge checks its proposed label against the predefined disambiguation rules and either confirms or revises it. The final label is used for evaluation.

   Evaluation metrics.

   We compare each judge’s predictions with the human-assigned labels using exact-match accuracy, macro-averaged F 1 , and Cohen’s κ \kappa . Category-level evaluation requires the correct interaction edge and fault side. Failure-mode evaluation additionally requires the correct named failure mode.

   Category Failure mode

   Model Acc F 1 F{1} Acc F 1 F{1}

   GPT-5.5 0.80 0.69 0.72 0.64

   Claude-Opus-4.6 0.75 0.61 0.70 0.57

   Claude-Opus-4.7 0.75 0.63 0.62 0.53

   Claude-Opus-4.8 0.75 0.62 0.68 0.58

   Table 2: Agreement of each judge with the human labels on the 40 worked examples. Category scores require the correct interaction edge and fault side. Failure-mode scores additionally require the correct named failure. Acc denotes exact-match accuracy and F 1 is macro-averaged.

   Agreement with human labels.

   Figure 3 shows pairwise Cohen’s κ \kappa between the human annotator and the four judges, with category agreement on the left and complete failure-mode agreement on the right. For category labels, GPT-5.5 has the highest agreement with the human annotations at κ = 0.76 \kappa=0.76 . Claude Opus 4.6 and 4.7 each reach κ = 0.71 \kappa=0.71 , followed by Claude Opus 4.8 at κ = 0.70 \kappa=0.70 . Agreement among the judges is comparable, with the highest pairwise value of κ = 0.84 \kappa=0.84 between Claude Opus 4.6 and 4.8. Agreement on the complete failure-mode label is lower across all pairs.

   Sources of disagreement.

   The remaining disagreement has two main sources. First, the source material is heterogeneous. Each judge receives only a reference to the original source, which may be a complete execution trace, GitHub issue, blog post, arXiv paper, or system-card section. Some of these sources do not provide enough evidence to identify a unique root cause. For example, in E4 , a public incident report attributes the agent’s deletion of more than 200 emails to context compaction dropping the owner’s instruction not to act, but does not provide the full trajectory. From the source alone, the case could be interpreted as either a context-side failure or a model-side unauthorized action. Second, root-cause attribution remains difficult even when the relevant evidence is available. In the case study in Appendix A.2 , the agent correctly completes the initial task, but a scripted reply email required for the follow-up never arrives because of a bug in the evaluation environment. The judge, however, interprets the incomplete follow-up as the model failing to check for the reply, rather than tracing the failure back to the undelivered email. OpenRCA 2.0, a root-cause analysis benchmark, identifies the same path-level bottleneck: frontier models often fail to reconstruct a verified causal propagation path from the initiating fault to the observed symptom, resulting in what the authors term an ungrounded diagnosis (Fang et al. , 2026 ) .

   Predicted cat. Gold cat.

   Model Acc F 1 F{1} Acc F 1 F{1}

   GPT-5.5 0.72 0.64 0.72 0.62

   Claude-Opus-4.6 0.70 0.57 0.80 0.70

   Claude-Opus-4.7 0.62 0.53 0.70 0.58

   Claude-Opus-4.8 0.68 0.58 0.78 0.69

   Table 3: Failure-mode agreement for the four judges on the 40 worked examples. Under Predicted cat., the judge predicts both the category and failure mode; under Gold cat., it selects the failure mode given the human-assigned category. Acc is exact-match accuracy, and F 1 is macro-averaged.

   Failure-mode prediction introduces an additional challenge because the set of possible labels is larger and several failure modes can produce similar visible symptoms. The prediction also depends on selecting the correct category first, so a category error can lead to an incorrect failure-mode label. When given the gold category, accuracy improves for the Opus models (Table 3 ), indicating that some failure-mode errors originate at the category stage rather than from confusion among the modes within the correct category.

   Category Failure mode

   Agreement Cov P R F 1 F{1} P R F 1 F{1}

   ≥ \geq 2 of 4 1.00 0.78 0.78 0.78 0.70 0.70 0.70

   ≥ \geq 3 of 4 0.90 0.83 0.75 0.79 0.75 0.68 0.71

   4 of 4 0.68 0.96 0.65 0.78 0.89 0.60 0.72

   Table 4: Selective-voting ensemble of the four judges at increasing category-agreement thresholds. At each threshold, the ensemble assigns a category label only when the required number of judges agree and abstains otherwise. Coverage is the proportion of all examples that receive a label. Precision is computed over labeled examples, whereas recall is computed over the full evaluation set. After selecting a category, the ensemble assigns the failure mode by majority vote among only the judges that predicted that category.

   Selective voting.

   We use selective voting to retain a subset of predictions with higher precision rather than assigning a label to every example (Verga et al. , 2024 ) . A category is assigned only when at least k k of the four judges agree, and the system abstains otherwise. For retained examples, the failure mode is selected by majority vote among the judges supporting that category. Increasing k k trades coverage for precision (Table 4 ). Agreement among three judges yields 0.83 category precision at 90% coverage, while unanimity raises precision to 0.96 at 68% coverage. Appendix A provides the full prompts and implementation details.

   Figure 3: Pairwise Cohen’s κ \kappa among the four LLM judges and the human annotator, with category labels on the left and complete failure-mode labels on the right. The bold outline marks agreement with the human annotator.

   7 Discussion

   The proposed taxonomy shows why fault localization matters in practice: the same observed failure may require a different intervention depending on the fault side. Model-side failures identify targets for post-training, harness-side failures point to changes in the agent scaffolding, and faults in the environment or evaluation setup require interventions outside the model. It also reveals how responsibility is distributed across the system. As shown in Figure 2 , most failure modes are assigned to the model side. This imbalance partly reflects our attribution rule: a failure is model-side when a more capable model could have prevented it or recovered from it. The remaining non-model failures identify cases that model improvement alone cannot resolve.

   The agent-as-a-judge experiments in § 6 test whether these distinctions can be applied consistently. Independent judges often recover the human-assigned labels from the same definitions and evidence, and their agreement with one another is comparable to their agreement with the human annotations. These results provide evidence that the labels capture shared structure rather than annotator-specific labeling preferences.

   Limitations

   The taxonomy is descriptive rather than quantitative: it organizes failures and assigns responsibility but does not estimate their relative frequency. It is derived from the cases we reviewed and may need to expand as agent architectures and harnesses evolve. The taxonomy labels also depend on the available evidence, and brief reports or model system cards may omit details needed to identify a unique root cause.

   Moreover, the agent-as-a-judge framework used to validate the taxonomy may be difficult to deploy in production because judge accuracy remains limited, especially for failure-mode labels. We attempted to mitigate this through ensembling, but the gain in precision comes at the cost of lower coverage. The system may therefore abstain on cases where fault attribution is most uncertain.