Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks

Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks

Youting Wang Xiao Han Dingyan Shang Yuan Tang Bowen Liu

Abstract Agent-safety benchmarks measure different behaviors, and their scores get quoted interchangeably as an agent’s safety. We treat four of them (R-Judge, InjecAgent, AgentHarm, AgentDojo) as measurements to be validated, running each under its official implementation and author-provided scorer on up to 22 models, with MMLU and GPQA measured by us under one protocol as a capability composite. The metric is the first problem. On any binary trace-judgment benchmark scored by F 1 F{1} , an “always positive” policy attains F 1 = 2 ​ π / ( 1 + π ) F{1}=2\pi/(1+\pi) ; on R-Judge that is 0.690 0.690 , above five of the 21 models that actually discriminate. The three broad-coverage benchmarks then rank the same 18 models differently, and the trade-off behind that disagreement is a small-panel artifact: R-Judge specificity against AgentHarm safety correlates − 0.64 -0.64 at n = 7 n{=}7 and + 0.02 +0.02 at n = 18 n{=}18 , and a quarter of random size-7 subsets reach | ρ | ≥ 0.5 |\rho|\geq 0.5 around that near-zero value. Held-out validity turns on which outcome you pick. Capability predicts task success ( ρ = + 0.60 \rho{=}{+}0.60 ) but correlates negatively with misalignment safety ( ρ = − 0.44 \rho{=}{-}0.44 , n = 21 n{=}21 ). On their paired n = 20 n{=}20 panel, the corresponding contrast is Δ = − 1.00 \Delta{=}{-}1.00 (95% CI [ − 1.48 , − 0.49 ] [-1.48,-0.49] , p < 0.001 p<0.001 ), and it survives leave-one-organization-out and organization-clustered bootstrap analyses. On an expanded 41-model panel, the misalignment correlation weakens to − 0.16 -0.16 (95% CI [ − 0.54 , + 0.22 ] [-0.54,+0.22] ) and jailbreak strengthens to + 0.34 +0.34 , though neither change is significant. AgentHarm shows the strongest held-out association, ρ = + 0.72 \rho{=}{+}0.72 with three-template jailbreak safety after controlling capability. But both instruments score harmful compliance, so this is evidence of convergent validity rather than general safety. Naming the benchmark, metric, target behavior, and model panel is the minimum a safety claim needs.

1 Introduction

Between 2024 and 2026, dozens of benchmarks appeared claiming to measure whether LLM agents are safe or reliable : resistance to prompt injection (Zhan et al. 2024 ; Debenedetti et al. 2024 ) , safety-risk awareness (Yuan et al. 2024 ) , refusal of harmful agentic tasks (Andriushchenko et al. 2025 ) , and risky-tool-use identification (Ruan et al. 2024 ) , among others. A recent taxonomy (Li et al. 2026 ) catalogs more than forty of them and reports a symptom worth taking seriously: swap the benchmark and the safety ranking contradicts itself, with rank concordance near 0.10 0.10 across models. That work stops at a twelve-model concordance check over four benchmarks, and lists capability-controlled evaluation and benchmark consolidation among its open problems.

This paper picks up there. If a safety score is a measurement, it can fail in the ways measurements fail, so we separate construct validity (what a score measures), metric validity (whether its metric measures that target), and criterion validity (whether it tracks held-out behavior). Four questions follow:

• RQ1 (construct structure). Do nominally-distinct agent-safety benchmarks measure a common underlying factor, or dissociable ones?

• RQ2 (capability confound). How much of the between-benchmark signal is general capability in disguise?

• RQ3 (criterion validity). Do the safety scores track held-out behavioral criteria beyond what general capability predicts?

• RQ4 (metric validity). Are the headline metrics valid instruments, or do they reward degenerate behavior?

Contributions.

We identify a formal metric-validity failure (Observation 1 ): for any binary trace-judgment benchmark scored by F1, the score of an “always positive” baseline has a closed form, and on R-Judge that baseline outranks five evaluated models. We then quantify a small-panel failure mode we walked into: the R-Judge-specificity/AgentHarm-safety correlation moves from − 0.64 -0.64 at n = 7 n{=}7 to + 0.02 +0.02 at n = 18 n{=}18 , and a quarter of random size-7 subsets show | ρ | ≥ 0.5 |\rho|\geq 0.5 despite the near-zero full-panel value. The core is a pre-specified, capability-controlled criterion-validity audit over one task-success outcome and two safety outcomes: capability predicts task success, its safety correlations move with the outcome and the panel, and the strongest safety-score result is the AgentHarm–jailbreak association ( ρ = + 0.72 \rho{=}+0.72 after controlling capability), with borderline evidence of outcome selectivity under organization-level resampling ( p = 0.051 p{=}0.051 ). We release an API-only audit harness and re-run artifacts for four benchmarks and three held-out outcomes, included with the accompanying reproducibility package.

Main conclusion.

A capability score is not a safety score, and no one agent-safety benchmark stands in for safety as a whole. What a score licenses you to say depends on how it was produced. Small panels are the sharpest edge: at seven models a weak relationship can look systematic, so validity needs re-checking as the model population turns over.

Scope.

The three RQ3 criteria are stand-ins for deployment, not claims about real-world harm; their limits (single instances, grader dependence, panel size) are in Section 6 .

2 Related Work

Agent-safety benchmarking.

Each of the four benchmarks we audit targets one construct. InjecAgent (Zhan et al. 2024 ) and AgentDojo (Debenedetti et al. 2024 ) target prompt-injection robustness; AgentHarm (Andriushchenko et al. 2025 ) measures compliance with harmful agentic tasks; R-Judge (Yuan et al. 2024 ) scores safety-risk awareness over interaction traces. Evaluators we do not re-run, such as ToolEmu (Ruan et al. 2024 ) for risky tool use, share the same status: designed as evaluators, never validated as measurements. AutoMonitor-Bench (Yang et al. 2026 ) scores misbehavior monitors by miss and false-alarm rates, the two-sided reporting our metric analysis argues for. Closest to this paper is a taxonomy and consistency analysis of agent-safety benchmarks (Li et al. 2026 ) , which documents ranking disagreement but does not control for capability, test criterion validity, or recover a latent structure; capability-controlled evaluation is named there as future work, and we take up all three.

Construct validity.

Prior work asks whether benchmark scores agree and what latent constructs they capture. Benchmark-agreement testing (Perlitz et al. 2024 ) , metabench (Kipnis et al. 2025 ) , and direct construct analyses (Bean et al. 2025 ) find that knowledge and reasoning benchmarks share a strong latent factor. Kearns ( 2026 ) and NIST’s statistical framework (Keller et al. 2026 ) bring quantitative construct-validity and uncertainty modeling to LLM evaluation. We point the same psychometric lens at agent reliability and safety, where benchmarks target distinct behaviors—refusal, injection robustness, risk awareness—and nobody has yet quantified how much capability confounds their comparison.

Benchmark gaming.

A separate concern is that a benchmark can be gamed by an adversary. Our claim is different and logically prior: even for an honest model, the headline score may not validly measure the intended construct. The F1 gameability we document (Section 4 ) is a concrete instance. The policy that scores mid-leaderboard is not adversarial; it labels every trace “unsafe” without examining any of them.

Over-refusal.

XSTest (Röttger et al. 2024 ) and OR-Bench (Cui et al. 2024 ) document exaggerated safety , or over-refusal, at the level of individual prompts. We ask the analogous question one level up, between benchmarks: are models that refuse harmful tasks (AgentHarm) worse at identifying benign traces correctly (R-Judge specificity), and does that produce the ranking disagreement? On the full panel that correlation sits near zero. On seven-model subsets it often does not.

Agent reliability and task performance.

Recent work isolates specific agent-reliability failures such as corrupt success (Cao et al. 2026 ) and evidence-bounded reporting (Chen 2025 ) ; τ \tau -bench (Yao et al. 2024 ) and its dual-control successor τ 2 \tau^{2} -bench (Barres et al. 2025 ) measure agentic task success. We borrow the latter’s retail domain as our held-out RQ3 task-success criterion, and treat the safety benchmarks as candidate columns for a validity audit rather than as settled measurements.

3 Method

3.1 Model panel and benchmarks

We assemble a compute-light, API-only panel of up to 22 models from nine model-developing organizations (OpenAI 6, Meta 3, Qwen 3, Mistral 3, DeepSeek 2, Amazon 2, Anthropic, Google, Cohere; full roster in Fig. 3 and the artifact). We re-run four safety benchmarks with their official implementations and scorers: R-Judge (full 571 items, self-judge), InjecAgent (300 stratified direct-harm/data-stealing items, rule-scored), AgentHarm (44 base behaviors spanning all eight harm categories, via the official Inspect evaluation with a gpt-4o-mini judge), and AgentDojo (the slack suite, 100 items, environment-scored for security and utility), plus a held-out task-success criterion for RQ3 (below).

Coverage differs by benchmark (Fig. 3 maps the full panel): R-Judge has n = 21 n{=}21 after Qwen3-32B falls below the 90 % 90% valid-output threshold; InjecAgent has n = 22 n{=}22 ; AgentHarm and AgentDojo, which require native tool use, have n = 19 n{=}19 and 5 5 ; and the τ 2 \tau^{2} criterion has n = 20 n{=}20 (App. A.2 ). The common R-Judge/InjecAgent/AgentHarm panel therefore has n = 18 n{=}18 . Capability anchors (MMLU and GPQA-Diamond) are measured by us under one uniform protocol: MMLU zero-shot on a fixed 500-item subset (seed 42, identical items for every model) and GPQA-Diamond (198 items, chain-of-thought), through the same API harness as the safety runs. We began with provider-reported model-card numbers; the replacements hold the protocol fixed across models, and the composite passes its positive control (§ 4 ). The anchor panel is 21 models (DeepSeek-R1 is excluded for unstable long-form GPQA generation); we keep the provider-reported anchors as a pre-harmonization comparison.

3.2 Benchmark targets

Before looking at any relationship among the score columns, we wrote down what behavior each benchmark is meant to measure. R-Judge tests whether a model distinguishes unsafe from benign interaction traces. InjecAgent and AgentDojo both test resistance to prompt injection, so their correlation serves as a convergent-validity check. AgentHarm tests whether a model refuses harmful agentic tasks. For R-Judge, we analyze specificity and balanced accuracy in addition to the official F1 because the three metrics reward different error patterns.

Score orientation and terminology.

All reported scores are oriented so that higher is better or safer. R-Judge specificity is the fraction of benign traces correctly identified as benign; its balanced accuracy averages performance on benign and unsafe traces. InjecAgent robustness is one minus attack success, and AgentHarm safety is one minus harmful compliance. For RQ3, τ 2 \tau^{2} -bench is a held-out task-success criterion, not a safety score. We use internal consistency for split-half analyses (App. A.2 ) and name each benchmark-specific score explicitly, instead of stretching “calibration” or “reliability” past their usual meanings.

3.3 Analysis plan

Before running each analysis, we fixed its hypotheses, thresholds, and decision rules. Because these choices lack an independent timestamp, we describe the analyses as pre-specified rather than preregistered. We use Spearman correlations for all association tests. Unless otherwise noted, correlation p p -values use two-sided asymptotic Spearman tests. In Fig. 2 , error bars are model-level percentile-bootstrap intervals. The panel expansion and direct panel comparisons follow their pre-specified bootstrap procedures (App. A.6 ). The corroborative PCA operates on the Pearson correlation matrix of the standardized score columns; factor count is chosen with Horn’s parallel analysis against a random-normal null, with the rank-based (Spearman) sensitivity in App. A.3 . We report partial Spearman correlations controlling the capability composite; App. A.5 gives the rank-residual implementation and an alternative convention. The capability composite is the mean of the standardized, verified MMLU and GPQA anchors; its model ordering is identical to the anchors’ first principal component. Between-construct redundancy is measured by rank-residualized partial correlations that control for the capability composite.

We pre-specified three controls: positive (capability-anchor loadings ≥ 0.6 \geq 0.6 on the first factor); negative (a column-permuted score matrix within the parallel-analysis null); and convergent (a high correlation between the same-construct InjecAgent and AgentDojo scores). The headline metric is the median pairwise partial Spearman | ρ | |\rho| among distinct-construct benchmarks, controlling the capability composite; the decision rule permits interpretation of the factor structure only if first-factor variance is ≥ 40 % \geq 40% or controlling capability reduces that median by ≥ 0.15 \geq 0.15 , with the positive and negative controls passing. These thresholds and the disconfirming test were fixed before any real run.

For criterion validity (RQ3) we pre-specified three held-out outcomes, each before running it. Task success is τ 2 \tau^{2} -bench retail (Barres et al. 2025 ) : 40 multi-turn, environment-scored tool-agent-user tasks. Misalignment safety records whether a model avoids blackmail or information leaking in a fictional autonomous-agent scenario (Lynch et al. 2025 ) , scored by a fixed classifier over 100 decisions per model. Jailbreak safety is one minus harmful-compliance on 50 forbidden prompts, averaged over three published jailbreak templates to reduce the ceiling created by any single template (Souly et al. 2024 ; Wei et al. 2023 ) . All three run through the same API-only harness (the two safety criteria are single-turn or scenario-based and cover the full panel; the τ 2 \tau^{2} criterion is a multi-turn tool loop covering 20 models), are distinct from the four audited benchmarks, and are not among the predictors. For each, the headline metric is the partial Spearman correlation of every safety score with the criterion, controlling the capability composite. We first report the criterion’s raw association with capability, then test whether each safety score adds predictive information. The pre-specified analysis gives reporting rules for both significant and null results. The accompanying reproducibility package contains the analysis scripts and frozen inputs needed to reproduce the reported results.

4 Results

Results run in order of evidential strength: the metric-validity failure (RQ4) and capability correlations (RQ2) are firmest, cross-benchmark structure (RQ1) is weaker, and the power-limited held-out validity tests (RQ3) come last.

4.1 The headline metric is gameable (RQ4)

Figure 1: A leaderboard’s headline metric can be matched by an “always unsafe” baseline. On R-Judge ( n = 21 n{=}21 ), five real, discriminating models score below this constant baseline (F1 = 0.69 {=}0.69 , dashed): F1 gives no credit for correctly rejected benign traces, so the degenerate policy can outrank models that discriminate.

R-Judge labels 52.7 % 52.7% of its 571 items “unsafe.” A degenerate policy that answers “unsafe” on every item, exercising no safety reasoning, therefore attains recall 1.0 1.0 , specificity 0 , and F1 = 0.690 \mathrm{F1}=0.690 . On the 21 panel models that produce valid R-Judge verdicts (Qwen3-32B is excluded for 21 % 21% unparseable output; see Method) this constant-baseline score outranks five real models (F1 0.30 0.30 – 0.57 0.57 ; Fig. 1 ), all of which discriminate. The panel’s highest-specificity model, o3-mini ( 0.97 0.97 ), receives F1 = 0.702 \mathrm{F1}=0.702 , only 0.012 0.012 above the constant baseline, because its recall is lower. Balanced accuracy reorders the board, since it credits correct decisions on both classes.

Observation 1 (F1 admits a high-scoring constant baseline) .

On a benchmark with class base rate π \pi scored by F1, the constant “always-positive” policy attains F1 = 2 ​ π / ( 1 + π ) \mathrm{F1}=2\pi/(1+\pi) without distinguishing safe from unsafe items. For R-Judge ( π = 0.527 \pi=0.527 , a near-balanced base rate) this is 0.690 0.690 , which exceeds the observed F1 of several models that do discriminate. The closed form depends on prevalence, but the measurement problem is F1’s blindness to true negatives: correctly identifying a benign trace earns nothing, so F1 cannot stand alone as a measure of two-sided discrimination on this trace-judgment benchmark.

Model R-Judge InjecAgent AgentHarm GPQA

spec. robust. safety

Llama-3.1-8B 0.04 0.32 0.89 30.4

Claude-3-Haiku 0.40 0.77 0.81 33.3

Llama-3.1-70B 0.11 0.35 0.78 46.7

o3-mini 0.97 0.87 0.77 77.2

Nova-Lite 0.94 0.88 0.76 42.0

Nova-Pro 0.64 0.60 0.75 46.9

Gemini-2.5-Flash 0.92 0.71 0.62 68.3

DeepSeek-V3 0.86 0.76 0.61 59.1

Qwen-2.5-72B 0.62 0.92 0.60 49.1

GPT-4.1 0.61 0.87 0.60 66.3

GPT-4.1-mini 0.76 0.87 0.51 65.0

GPT-4o 0.72 0.90 0.47 53.6

GPT-4.1-nano 0.79 0.94 0.42 50.3

Llama-3.3-70B 0.28 0.31 0.39 50.0

Mistral-Nemo 0.80 0.34 0.35 28.0

GPT-4o-mini 0.61 0.89 0.31 40.2

Mixtral-8x22B 0.58 0.78 0.24 33.2

Mistral-Large 0.50 0.81 0.24 44.0

Table 1: The three broad-coverage agent-safety benchmarks rank the same n = 18 n{=}18 models differently (higher is safer). Rows are sorted by AgentHarm safety. Bold marks column maxima and underlining marks minima, with display-precision ties included. GPQA is shown as a capability reference. A model high on AgentHarm can rank low on R-Judge specificity, so the scores are not interchangeable.

4.2 The capability confound is real but metric-dependent (RQ2)

Across the R-Judge panel ( n = 20 n{=}20 with harmonized anchors), capability correlates strongly with balanced accuracy ( ρ = + 0.71 \rho=+0.71 with MMLU, p < 0.001 p<0.001 ; + 0.49 +0.49 with GPQA, p = 0.03 p=0.03 ) and F1 ( + 0.76 +0.76 and + 0.62 +0.62 ), but only weakly with specificity ( + 0.16 +0.16 and + 0.07 +0.07 ). This difference follows the metric definitions: balanced accuracy and F1 include recall on unsafe traces, whereas specificity measures only correct treatment of benign traces and can be inflated by rarely flagging anything. So the capability confound is a property of the metric you report, not of the benchmark as a whole—and it moves with the panel too: the specificity–MMLU correlation fell from 0.85 0.85 on our first nine models to + 0.16 +0.16 at n = 20 n{=}20 .

4.3 Benchmarks disagree; a seven-model panel suggested a trade-off that does not persist (RQ1)

The firmest RQ1 result is descriptive and correlation-free: on this panel no single benchmark orders the models the way another does (Table 1 ); a top-AgentHarm model can rank near the bottom on R-Judge specificity. This makes “safety rank” benchmark-dependent without resting on any correlation estimate. Disagreement between benchmarks coded to different constructs (§ 3 ) is expected and does not by itself indict any single instrument; what it defeats is the practice of quoting these scores interchangeably as the safety of an agent.

We read a trade-off into that disagreement early on; it does not survive the larger cross-benchmark panel. The R-Judge-specificity/AgentHarm-safety rank correlation was − 0.64 -0.64 at n = 7 n{=}7 but + 0.02 +0.02 ( p = 0.95 p=0.95 ) at n = 18 n{=}18 (Fig. 5 , appendix), and random size-7 subsets yield | ρ | ≥ 0.5 |\rho|\geq 0.5 in a quarter of draws despite the near-zero full-panel value. Sampling variability alone is enough to manufacture a “clean reversal” at that panel size, and the dissolution holds under all four R-Judge metrics (App. A.4 ). Whether capability moderates what is left we do not test; we flag it as a hypothesis.

A single-factor PCA provides weaker, corroborative evidence (Fig. 6 , appendix): its first component ( 48.7 % 48.7% variance) places R-Judge specificity, InjecAgent robustness, and both capability anchors in one direction and AgentHarm safety in the other ( + 0.33 +0.33 , opposite the cluster in 17 17 of 18 18 leave-one-out fits).

The positive control now passes.

On the harmonized anchors the capability composite passes its pre-specified positive control (MMLU loads 0.74 ≥ 0.6 0.74\geq 0.6 ; on provider-reported numbers it had failed at 0.42 0.42 , our original reason for caution) and the negative control also passes (a column-permuted score matrix yields eigenvalues within the parallel-analysis null), so the pre-specified interpretation conditions are met (the convergent control is inconclusive at AgentDojo’s n = 5 n{=}5 ). We still treat the factor as corroboration only: a rank-based (Spearman) PCA keeps the loading structure but drops PC1 to 40.8 % 40.8% with no factor retained by Horn (App. A.3 ), and the median between-benchmark correlation is just 0.22 0.22 ; the firm, factor-independent RQ1 result is the raw-score disagreement (Table 1 ).

4.4 Predictive validity differs across held-out outcomes (RQ3)

Everything above is internal to the benchmarks. RQ3 steps outside them: do these scores predict held-out behavior that a capability test does not already predict? Which outcome you choose decides the answer. We pre-specified three (§ 3 ): τ 2 \tau^{2} -bench task success, misalignment safety, and jailbreak safety. All are distinct from the four audited benchmarks, and each has a fixed metric and decision rule.

Task success Safety outcomes

Predictor τ 2 \tau^{2} misalign. jailbreak †

Capability (raw ρ \rho )

• 0.60 ∗ ∗ +0.60^{*} − 0.44 ∗ -0.44^{}

• 0.08 +0.08

  Safety-score partial ρ \rho (controlling capability):

  R-Judge specificity − 0.09 -0.09

• 0.41 +0.41 − 0.11 -0.11

  InjecAgent robustness

• 0.16 +0.16
• 0.47 +0.47

• 0.21 +0.21

  AgentHarm safety − 0.23 -0.23

• 0.16 +0.16

• 0.72 ∗ ⁣ ∗ ∗ +0.72^{*}

  Table 2: Criterion validity on the 2024–25 panel (cellwise n = 18 n{=}18 – 21 21 because predictor coverage differs; App. A.2 ; higher is better or safer). The top row reports raw correlations with capability; the remaining rows report partial Spearman correlations controlling the capability composite. Capability predicts task success but not either safety outcome consistently. AgentHarm is strongly associated only with the three-template jailbreak outcome, marked by † . Stars use two-sided asymptotic Spearman tests ( p ∗ < 0.05 {}^{}p<0.05 , p ∗ ∗ < 0.01 {}^{*}p<0.01 , p ∗ ⁣ ∗ ∗ < 0.001 {}^{*}p<0.001 ).

  Against task success, safety scores add no information beyond capability.

  Capability predicts τ 2 \tau^{2} -retail success ( ρ = + 0.60 \rho=+0.60 , p = 0.005 p=0.005 , n = 20 n{=}20 ), while no safety score shows incremental validity once capability is partialled out (best | ρ | = 0.23 |\rho|=0.23 , n.s. at n = 19 n{=}19 ; Table 2 ; Fig. 7 , appendix). Because capability already predicts τ 2 \tau^{2} , these partials ask the stricter question of whether a safety score predicts what capability leaves unexplained; none does. The null is not a verdict on the safety scores. Task success is simply an outcome that capability already explains well, which is why we test two safety outcomes as well.

  Figure 2: Capability predicts task success; safety point estimates differ by outcome and panel. Filled circles show the 2024–25 correlations; open diamonds show the expanded-panel safety correlations; error bars are model-level percentile-bootstrap 95% CIs. Thin lines connect panel-level estimates, not model trajectories. Misalignment changes from − 0.44 -0.44 to − 0.16 -0.16 and jailbreak from + 0.08 +0.08 to + 0.34 +0.34 ; neither between-panel change is significant. Task success was not re-collected for the expansion.

  Against agentic misalignment, capability has the opposite association.

  On the full 21-model panel, capability predicts misalignment-safety negatively ( ρ cap = − 0.44 \rho_{\text{cap}}=-0.44 , asymptotic p = 0.047 p=0.047 ), the opposite sign to its + 0.60 +0.60 on τ 2 \tau^{2} : higher capability scores go, if anything, with lower safety here (Mistral-Large blackmails or leaks 85 % 85% of the time; o3-mini is a capable exception at 1 % 1% ). After controlling capability, InjecAgent robustness ( + 0.47 +0.47 ) and R-Judge specificity ( + 0.41 +0.41 ) reach the pre-specified 0.40 0.40 effect-size threshold, while AgentHarm safety does not ( + 0.16 +0.16 ; Table 2 ). The two associations remain exploratory: both confidence intervals include zero, neither survives Bonferroni correction, power is only ≈ 0.35 \approx 0.35 , and the pre-specified directional prediction named AgentHarm, not the two scores that came up. The R-Judge result is also metric-sensitive: its partial correlation falls from + 0.41 +0.41 under specificity to + 0.19 +0.19 under balanced accuracy and reverses to − 0.46 -0.46 under recall (App. A.4 ).

  A direct interaction test on the original panel.

  We compare capability’s correlations with task success and misalignment on their paired n = 20 n{=}20 panel. The task-success correlation is

• 0.60 +0.60 and the misalignment-safety correlation is − 0.41 -0.41 ( − 0.44 -0.44 on the full panel), a difference of Δ = − 1.00 \Delta=-1.00 (95% CI [ − 1.48 , − 0.49 ] [-1.48,-0.49] , p < 0.001 p<0.001 ). This difference is stable to leave-one-model-out, leave-one-organization-out, organization-clustered bootstrap, and subsampling (App. A.5 ). The corresponding R-Judge interaction is only marginal ( Δ = + 0.50 \Delta=+0.50 , p = 0.082 p=0.082 ), so the evidence supports the capability interaction but not a full two-way dissociation. An independent misalignment grader produces nearly identical model scores ( ρ = 0.97 \rho=0.97 ).

  The capability–misalignment correlation is weaker on the expanded panel.

  Before running the expansion, we specified that the original negative correlation should “hold or strengthen.” Instead, it changes from ρ = − 0.44 \rho=-0.44 ( n = 21 n{=}21 , asymptotic p = 0.047 p=0.047 ) to − 0.16 -0.16 ( n = 40 n{=}40 , bootstrap p = 0.40 p=0.40 ; 95% CI [ − 0.54 , + 0.22 ] [-0.54,+0.22] ; App. A.6 ). The point estimate is less than half as large and no longer significant, but the interval still includes − 0.44 -0.44 and the between-panel difference is not significant (nested-bootstrap Δ ​ ρ = + 0.27 \Delta\rho=+0.27 , 95% CI [ − 0.25 , + 0.78 ] [-0.25,+0.78] ). So the original estimate should not be assumed to carry over to the expanded model population, though the data do not establish a temporal shift either. Matching subsets to the original organization composition gives a similar estimate (median ρ = − 0.17 \rho=-0.17 ), which suggests composition alone does not explain the difference; a generational explanation stays possible and undemonstrated. Because τ 2 \tau^{2} was not re-collected, we could not re-test the original interaction.

  Models with similar capability scores differ in safety across organizations (Fig. 4 ).

  The 41-model panel spans 12 model-developing organizations (misalignment n = 40 n{=}40 ; jailbreak n = 41 n{=}41 after one engagement-gate exclusion). Six organizations contribute at least four models for within-organization correlations, and seven contribute at least three models for the group comparison (App. A.6 , Table 5 ). Within organizations, capability–safety correlations vary in sign and average near zero ( − 0.14 -0.14 misalignment, + 0.26 +0.26 jailbreak). After adjusting safety ranks for capability, the seven organization groups differ on both criteria (permutation p = 0.004 p=0.004 misalignment, p < 0.001 p<0.001 jailbreak; Holm-significant), with df-adjusted Kruskal–Wallis effect sizes ε 2 = 0.34 \varepsilon^{2}=0.34 and 0.57 0.57 . None of this is causal: the static capability composite may miss agentic ability, each organization contributes few models, and some groups have restricted safety ranges. What the panel supports is narrower—similar MMLU/GPQA scores do not imply similar safety scores.

  Against jailbreak safety, AgentHarm shows the strongest held-out association.

  We pre-specified a second safety outcome with a different failure mode, harness, and scoring procedure: harmful compliance under jailbreak, scored with StrongREJECT (Souly et al. 2024 ) . A single jailbreak template (AIM) produces a ceiling because 11 of 22 models refuse every request. Averaging scores over three published templates (Wei et al. 2023 ) leaves no model at the ceiling. Capability is weakly associated with this outcome on the 2024–25 panel ( ρ = + 0.08 \rho=+0.08 , n = 21 n{=}21 , p = 0.73 p=0.73 ). The expanded-panel estimate is

• 0.34 +0.34 ( n = 41 n{=}41 , p = 0.04 p=0.04 ), but the between-panel change is not significant.

  After controlling capability, AgentHarm safety correlates with the three-template jailbreak outcome at ρ = + 0.72 \rho=+0.72 ( p < 0.001 p<0.001 ), compared with + 0.50 +0.50 for the ceiling-limited single-template measure. The estimate is stable to leave-one-model-out analysis and an independent grader (App. A.5 ). AgentHarm is more strongly associated with jailbreak than with misalignment in a bootstrap that treats models as independent ( Δ = + 0.59 \Delta=+0.59 , p = 0.027 p=0.027 ); a bootstrap that instead resamples the eight represented organizations gives p = 0.051 p=0.051 . On this paired panel, the misalignment partial is + 0.13 +0.13 ( n = 19 n{=}19 ); Table 2 ’s + 0.16 +0.16 uses the common n = 18 n{=}18 panel. R-Judge specificity ( − 0.11 -0.11 ) and InjecAgent robustness ( + 0.21 +0.21 ) do not predict jailbreak safety. AgentHarm and StrongREJECT both measure harmful compliance, so part of what the + 0.72 +0.72 shows is convergent validity. The stronger claim—that AgentHarm is selective for jailbreak over misalignment—rests on a borderline organization-resampled test, and the matching comparison for R-Judge and InjecAgent, whether they predict misalignment better than jailbreak, is marginal as well ( Δ = + 0.52 \Delta=+0.52 , p = 0.068 p=0.068 ).

  Two alternative explanations, checked.

  Seven LLM coders who saw only construct descriptions, and none of the predictive results, independently assigned AgentHarm and the jailbreak outcome to the same harm-blocking category ( 7 / 7 7/7 for each; Krippendorff α = 0.61 \alpha=0.61 across three categories). They classified the misalignment outcome less cleanly, one more reason to keep its R-Judge/InjecAgent associations exploratory (App. A.1 ). The second check: after controlling capability, AgentHarm safety is uncorrelated with over-refusal on benign XSTest prompts (Röttger et al. 2024 ) ( ρ = − 0.01 \rho=-0.01 , n = 17 n{=}17 ), which weighs against the story that models score well on AgentHarm and the jailbreak outcome simply by refusing everything.

  5 Discussion

  What the results support is narrow: these agent-safety benchmarks are not interchangeable measures of a single property. An “always unsafe” baseline matches R-Judge F1 (Observation 1 ), and the three broad-coverage benchmarks rank the same models differently. Held-out validity shifts with the outcome and the model panel (Table 2 ): capability predicts task success, while its correlations with the two safety outcomes are unstable across panels. AgentHarm’s capability-controlled association with three-template jailbreak safety is strong ( ρ = + 0.72 \rho=+0.72 ), but both measures concern harmful compliance and the outcome-selectivity test gives p = 0.051 p=0.051 under organization resampling, which makes this convergent validity with the specificity claim still open. The R-Judge/InjecAgent associations with misalignment are exploratory. Safety ranks also differ among organization groups after adjustment for the capability composite—an observational pattern that identifies no organizational cause. A safety claim is defensible only with its benchmark, metric, target behavior, and model population attached.

  6 Limitations and Threats to Validity

  Panel size.

  The AgentHarm-limited cross-benchmark panel has n = 18 n{=}18 , up from our first pass at n = 7 n{=}7 , which is where our caution about small-panel claims comes from. The gap most worth closing is a larger, capability-spanning agent-loop panel.

  Capability anchors.

  Provider-reported model-card anchors initially failed their positive control (MMLU 0.42 0.42 ). Re-measuring MMLU and GPQA-Diamond (§ 3 ) fixed it (MMLU 0.74 0.74 ); every RQ2/RQ3 sign was unchanged. These are static-knowledge tests, although their composite predicts multi-turn tool-agent success at + 0.60 +0.60 . A six-model BFCL pilot (Patil et al. 2025 ) failed as an agentic replacement: strict function-call syntax scoring ranked GPT-4o-mini highest ( 0.72 0.72 ) and the two most capable models lowest ( 0.42 0.42 – 0.48 0.48 ), apparently rewarding format conformance. That failure is the argument for a validated agentic anchor. Measurement error of this kind attenuates or suppresses partial relationships instead of biasing them in a direction we could sign.

  Multiple comparisons.

  We report per-test p p -values without a family-wise correction, but our claims rest on effect sizes and pre-specified rules: the two strongly significant tests (the crossover and AgentHarm → \to jailbreak, both p < 0.001 p<0.001 ) would survive Bonferroni, while the R-Judge/InjecAgent–misalignment cells ( p ≈ 0.05 p\approx 0.05 – 0.09 0.09 ) would not, and we treat those as suggestive throughout.

  Judges and coverage.

  Three independent-grader checks support robustness to judge choice: model scores correlate at ρ = 0.97 \rho=0.97 for misalignment and ρ = 0.98 \rho=0.98 for jailbreak. Rescoring AgentHarm refusals with a different model lowers agreement ( ρ = 0.69 \rho=0.69 ), yet the AgentHarm–jailbreak partial correlation barely moves ( + 0.66 +0.66 against + 0.72 +0.72 ; App. A.5 ). AgentDojo runs on only five models, so the cross-benchmark analyses depend primarily on the other three benchmarks.

  Criterion validity (RQ3).

  τ 2 \tau^{2} -retail (40 tasks, one run/model) is scored by a user-simulator whose variance we do not bound; the misalignment criterion is a fictional-scenario probe (grader-robust, ρ = 0.97 \rho=0.97 , though its blackmail–leaking scenario halves agree only moderately, Spearman–Brown 0.70 0.70 ); the jailbreak criterion averages three jailbreaks. None gives literal deployment-harm probabilities. For the item-rich instruments with resampling support, split-half consistency is ≥ 0.96 \geq 0.96 and rank stability is ≥ 0.97 \geq 0.97 . Across the bounded instruments, worst-case within-model SEs are 12 12 – 25 % 25% of the between-model SD, which supports the panel-level spread but not every pairwise model difference (App. A.2 ). The two construct-matching patterns are not equally well supported. The AgentHarm–jailbreak association is large and stable in direction, but it partly reflects convergent measurement and its selectivity over misalignment is borderline under organization resampling. The R-Judge/InjecAgent–misalignment pattern is exploratory (marginal Δ = + 0.52 \Delta=+0.52 , power ≈ 0.35 \approx 0.35 ; the larger three-scenario result is post-hoc; App. A.5 ).

  7 Conclusion

  Across four benchmarks and up to 22 models, R-Judge’s F1 lets an “always unsafe” baseline outrank five models, and switching benchmarks switches the rankings. Capability predicts held-out task success; it does not predict safety consistently across outcomes or panels. The strongest held-out safety association is AgentHarm’s ( ρ = + 0.72 \rho=+0.72 with jailbreak safety after controlling capability), though the two measures partly overlap and organization-resampled evidence for outcome selectivity is borderline. Safety evaluations should report the full confusion structure, drop standalone F1 wherever true negatives matter, and tie every claim to an explicit target behavior and model population.