The Asymmetric Effects of Knowledge Distillation on Bias in Small Language Models

Plawan Kumar Rath Meta plawan@meta.com This work was conducted in the author’s personal capacity. The views expressed in this paper are those of the authors and do not reflect the views of Meta.

Abstract We show that knowledge distillation (KD) in small instruction-tuned language models has asymmetric effects on bias, and that measuring those effects correctly requires accounting for where refusal mass moves and for what the answer parser can legitimately score. On unambiguous tasks (BBQ-disambig), response-based distillation from a Mistral-7B teacher genuinely improves context-following for the most context-biased baseline (SmolLM2-1.7B-Instruct): among committed (non-abstaining) answers, the rate of overriding correct context with a stereotype falls from 44.5 % 44.5% to 37.2 % 37.2% , with accuracy rising from 0.55 0.55 to 0.61 0.61 . On ambiguous tasks (BBQ-ambig), the same distillation degrades conditional refusal: 15 % 15% of the cases where the baseline correctly abstained instead receive stereotype answers (silence-loss), and the distilled refusal pattern only weakly preserves the baseline’s (Spearman ρ = 0.44 \rho{=}0.44 ). The ambiguous-side harm reproduces in aggravated form on a second student family (OLMo-2-1B-Instruct): silence-loss reaches 49 % 49% and filled-silence accounts for 95 % 95% of new bias. We further show that two apparently stronger results are artifacts: an unconditioned override metric reports a 44 % → 23 % 44%{\to}23% improvement under a Gemma-2-9B teacher that shrinks to 44.5 % → 39.8 % 44.5%{\to}39.8% once conditioned on committed answers (the model abstains on 43 % 43% of items and its accuracy collapses from 0.55 0.55 to 0.35 0.35 ), and an apparent cross-condition independence reverses to a positive correlation ( ρ = 0.58 \rho{=}0.58 , p < 0.01 p{<}0.01 ) on the valid 19 19 -configuration grid once parser-invalid logit-KD configurations are excluded and a corrected answer parser is applied. Aggregate stereotype metrics (CrowS-Pairs, overall BBQ Stereotype Reliance Score) average over both effects and conceal the per-item harm. Consistent with a data-side account, an audit of four training corpora finds < 0.5 % <{}0.5% refusal-as-answer-shape, though a per-item association test is heterogeneous across teachers, so we present the corpus account as a hypothesis rather than an established mechanism. Supervised fine-tuning (SFT) with refusal injection either breaks parsing or over-corrects into a trivial-refuser regime (refusal rate 99.8 % 99.8% , committing on 0.03 % 0.03% of disambiguated items) that unconditioned metrics would call perfectly calibrated. We propose Per-Condition Calibration Diagnosis (PCCD) , a three-step protocol that evaluates refusal-pattern preservation, committed-answer context-following, and capability preservation. No configuration in our grid passes all three steps at the scales where all three are informative.

The Asymmetric Effects of Knowledge Distillation on Bias in Small Language Models

Plawan Kumar Rath † † thanks: This work was conducted in the author’s personal capacity. The views expressed in this paper are those of the authors and do not reflect the views of Meta. Meta plawan@meta.com

1 Introduction

Small instruction-tuned language models (LMs) are increasingly deployed on resource-constrained edge devices (Maliakkal et al. , 2026 ) , where knowledge distillation (KD) is a primary technique for compressing larger teachers into deployable students. As these compressed models reach end users, their fairness properties become consequential. A mainstream story about KD and bias holds that teacher bias transfers to student bias : a biased teacher induces a biased student, and the cure is debiasing the teacher (Hooker et al. , 2020 ; Mohammadshahi et al. , 2022 ) . We show this framing misidentifies the locus of the problem for small instruction-tuned language models.

Distillation is asymmetric, and measuring the asymmetry requires care with refusals. On disambiguated contexts (where the correct answer is determinable from context), response-based distillation can make small students genuinely better: the SmolLM2-1.7B-Instruct baseline overrides correct context with a stereotype on 44.5 % 44.5% of its committed answers to anti-stereotype items, and response-Mistral distillation reduces this to 37.2 % 37.2% while raising accuracy from 0.55 0.55 to 0.61 0.61 . On ambiguous contexts (where “unknown” is the correct answer), the same distillation introduces a new harm: 15 % 15% of the cases the baseline correctly refused now receive stereotype answers, and the refusal pattern is only weakly preserved ( ρ = 0.44 \rho{=}0.44 ) even as the marginal refusal rate rises ( 4.6 % → 13.5 % 4.6%{\rightarrow}13.5% ).

Refusal mass is also where bias measurement breaks. The apparently strongest improvement in our grid, a 44 % → 23 % 44%{\to}23% drop in unconditioned context-override under a Gemma-2-9B teacher (the headline of an earlier version of this paper), mostly dissolves under committed-answer renormalization ( 44.5 % → 39.8 % 44.5%{\to}39.8% ): that model “improves” mainly by abstaining on 43 % 43% of disambiguated items while its accuracy collapses from 0.55 0.55 to 0.35 0.35 , so the residual reduction is not a context-following gain. The same correction reverses a cross-condition independence result: across the 19 19 valid distilled configurations, silence-loss and renormalized context-override are positively correlated ( ρ = 0.58 \rho{=}0.58 , p = 0.009 p{=}0.009 ), driven by configurations that degrade on both sides at once. Aggregate metrics (CrowS-Pairs whole-sentence stereotype preference, overall BBQ stereotype rate) conceal all of this: the same response-Gemma model looks beneficially distilled in aggregate.

We evaluate a data-side account of the harm. An audit of four training corpora used in our pipeline (Alpaca-cleaned plus three teacher response sets) finds opener-refusal rates of 0.05 % − 0.33 % 0.05%{-}0.33% : the corpora contain almost no refusal-as-answer-shape. A per-item association test is directionally supportive but heterogeneous: in 3 3 of 19 19 valid configurations (Holm-corrected, all with the Mistral teacher), items with at least one refusal-shaped training neighbor are 9 − 21 9{-}21 percentage points more likely to preserve refusal post-distillation, while Gemma-teacher configurations show null or negative associations. The distilled student appears to learn the marginal frequency of refusal more readily than which prompts deserve it, but our evidence for the corpus account is correlational; the discriminating experiment is paired corpus injection (§ 7 ).

Contributions.

1. We document the asymmetric effect of distillation on bias using refusal-aware metrics: at 1.7B scale, response-KD can genuinely improve committed-answer context-following on disambiguated items while introducing silence-loss and refusal-pattern remapping on ambiguous items; the ambiguous-side harm reproduces across two student families (SmolLM2 and OLMo-2) (§ 4 ).

2. We show that both aggregate and unconditioned conditional metrics mislead when distillation moves refusal mass (§ 5 ). Configurations that pass CrowS-Pairs and overall BBQ SRS checks fail at the per-item level, an unconditioned override metric misreports abstention as the grid’s strongest fairness improvement, and a previously reported cross-condition independence reverses under the corrected metric.

3. We provide a cell decomposition of newly-introduced stereotype answers into Inherited / Amplified-vs-Anti / Filled-Silence (§ 6 ). Direct contradiction of teacher labels is rare ( ≤ 13 % \leq 13% ); 50 − 95 % 50{-}95% of new bias arises in items where the teacher correctly abstained.

4. We identify a data-side signal : an audit of four training corpora finds < 0.5 % <{}0.5% refusal-as-answer (§ 7 ). A per-item refusal-neighbor association is positive and Holm-significant in 3 3 of 19 19 valid configurations but heterogeneous across teachers, so we state the corpus account as a working hypothesis with a designed discriminating experiment, not an established mechanism.

5. We propose Per-Condition Calibration Diagnosis (PCCD) (§ 8 ): a three-step protocol covering refusal-pattern preservation, committed-answer context-following, and capability preservation. It flags the ambiguous-side harm, the metric artifacts, and the trivial-refuser failure mode; no configuration in our grid passes all three steps at the scales where all three are informative.

   2 Related Work

   Distillation methods.

   We study three standard KD families: response-based distillation, in which the student is trained on teacher-generated completions (Hinton et al. , 2015 ) ; logit-based distillation, which minimizes Kullback–Leibler divergence between teacher and student output distributions (Sanh et al. , 2019 ; Gu et al. , 2024 ) ; and a combined response + + logit objective. We use SmolLM2 students (Allal et al. , 2025 ) distilled from three open instruction-tuned teachers: Gemma-2-9B-it (Gemma Team, 2024 ) , Mistral-7B-Instruct-v0.3 (Jiang et al. , 2023 ) , and Phi-3.5-mini-Instruct (Abdin and others, 2024 ) .

   KD and bias.

   A broad survey of bias in large language models (LLMs) is provided by Gallegos et al. ( 2024 ) . Prior work has linked distillation specifically to amplified bias in classification (Hooker et al. , 2020 ; Mohammadshahi et al. , 2022 ) and observed that smaller models can inherit teacher stereotypes (Ahn et al. , 2022 ; Silva et al. , 2021 ) . Bias-mitigation methods during distillation include counterfactual role reversal (Gupta et al. , 2022 ) and constrained distillation that explicitly penalises stereotypical association (Delobelle and Berendt, 2022 ) . Compression more broadly has been shown to amplify bias in instruction-tuned LMs: pruning produces a “smart pruning paradox” where perplexity-preserving methods yield the highest bias amplification (Rath and Maliakkal, 2026b ) , and aggressive quantization induces dose-response bias emergence with a 17 % 17% decline in unknown-selection rate that is invisible to perplexity (Rath and Maliakkal, 2026a ; Marcuzzi et al. , 2025 ) . Ramesh et al. ( 2023 ) compare compression techniques; we extend this line by analyzing per-item refusal behavior rather than aggregate stereotype rates.

   Bias benchmarks and methodological critiques.

   We use BBQ (Parrish et al. , 2022 ) , which separates ambiguous from disambiguated contexts (enabling our cross-condition decomposition), and CrowS-Pairs (Nangia et al. , 2020 ) as a secondary aggregate metric. Related benchmarks include StereoSet (Nadeem et al. , 2021 ) and WinoBias (Zhao et al. , 2018 ) . Blodgett et al. ( 2021 ) catalogue methodological pitfalls in pair-based fairness benchmarks, and Goldfarb-Tarrant et al. ( 2021 ) show that intrinsic bias metrics do not predict downstream task fairness. These critiques motivate our shift from aggregate stereotype scoring to per-item refusal analysis and committed-answer conditioning. We treat abstention as a first-class outcome rather than a parsing failure, following calibration work (Kadavath et al. , 2022 ) .

   3 Experimental Setup

   Models.

   Three SmolLM2-Instruct students ( 135 135 M, 360 360 M, 1.7 1.7 B) × \times three teachers (Gemma-2-9B-it, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-Instruct) × \times three KD methods (response, logit, combined; we refer to these as response-KD, logit-KD, and combined-KD) = 27 27 SmolLM2 distilled configurations. We additionally distill AI2 OLMo-2-1B-Instruct (OLMo Team et al. , 2025 ) from Gemma-2-9B-it using response-KD as a second-family generalization probe (§ 10 ), and train three SFT-only controls (one per SmolLM2 student), for 31 31 trained configurations in total. A parser-validity screen (Appendix A ) excludes all 9 9 logit-KD configurations: their generations are degenerate (validation perplexity 10 5 − 10 6 10^{5}{-}10^{6} ) and none of their records yields a parseable answer under the corrected parser cascade ( 100 % 100% failure). The resulting valid grid is 19 19 KD configurations ( 18 18 SmolLM2 response/combined + + OLMo response-Gemma) plus the 3 3 SFT controls; every cross-configuration statistic and figure in this paper uses the valid grid unless stated otherwise.

   Training data.

   51 , 760 51{,}760 Alpaca-cleaned prompts (Taori et al. , 2023 ) , derived from the Self-Instruct framework (Wang et al. , 2023 ) . Teacher response sets are generated by running each teacher on the same prompt set; instruction-tuning follows the supervised-fine-tuning recipe of Ouyang et al. ( 2022 ) . We use Low-Rank Adaptation (LoRA) (Hu et al. , 2022 ) adapters with rank 16 16 , α = 32 \alpha{=}32 , batch size 16 16 , learning rate 1 ​ e − 4 1\mathrm{e}{-}4 , trained for 3 3 epochs.

   Evaluation.

   BBQ-ambig and BBQ-disambig ( 12 , 148 12{,}148 items each); 5 5 seeds per item; temperature 0.3 0.3 , max ⁡ ​ tokens = 5 \max\\textrm{tokens}{=}5 (multiple-choice question, MCQ, letter answer). CrowS-Pairs whole-sentence pseudo log-likelihood scoring on 1 , 508 1{,}508 pairs. Total > 1.5 ​ M >1.5\mathrm{M} inferences. Answer letters are extracted with a deterministic four-branch parser cascade (exact letter, letter prefix, answer-keyword match, and unique content match against the option texts). An earlier version of this work additionally used a bare first-character scan as a last resort; Appendix A shows that branch manufactures position-biased letters for degenerate or truncated outputs, so it is removed and every number in this paper uses the corrected cascade. Per-configuration parse-branch shares and failure rates are reported in Appendix A . All analyses use the full 12 , 148 12{,}148 -item set per condition; no baseline-dependent item filtering is applied.

   Metrics.

   We track Stereotype Reliance Score (SRS, fraction of valid responses selecting the stereotypical answer; Rath and Maliakkal, 2026b ), unknown-selection rate (USR), and anti-stereotype rate on BBQ-ambig, plus CrowS-Pairs stereotype preference. On BBQ-disambig we track accuracy and the context-overriding rate in two forms: unconditioned (fraction of anti-stereotype-correct items answered with the stereotype) and renormalized to committed answers (the same fraction computed only over responses that select a substantive option rather than unknown). The renormalized form is primary throughout: distillation moves refusal mass, and the unconditioned form scores abstention as if it were fairness (§ 5 ). We define two per-item diagnostics. Silence-loss is the fraction of (item, seed) pairs where the baseline student selected unknown but the post-trained student selects the stereotype; this single definition is used everywhere in the paper. Refusal-pattern preservation ρ \rho (called per-item refusal calibration in an earlier version) is the Spearman correlation between baseline and distilled per-item USR; it measures preservation of the baseline student’s refusal pattern, not agreement with ground truth. Silence-loss, by contrast, is ground-truth-anchored: on BBQ-ambig the unknown option is correct for every item. We also report refusal discrimination , USR on ambiguous items minus USR on disambiguated items: a ground-truth-anchored summary of whether abstention concentrates where it is correct. Wilson confidence intervals, KD-vs-SFT χ 2 \chi^{2} tests, and Cohen’s h h effect sizes are reported throughout; headline percentages in the text are accompanied by 95 % 95% Wilson CIs in the appendix tables.

   Compute budget.

   All training and inference run on a single workstation with Apple Silicon (M-series) using Apple’s MLX framework with bfloat16 weights. Approximate wall-clock budgets: LoRA fine-tuning ≈ 270 \approx 270 hours total across all distilled configurations ( ≈ 10 \approx 10 hours per config × \times 27 27 configs plus 3 3 SFT controls); inference ≈ 210 \approx 210 hours total ( ≈ 0.5 \approx 0.5 s per generation × \times 1.5 ​ M 1.5\mathrm{M} generations); analysis and embedding-based predictive tests ≈ 8 \approx 8 hours. Total compute budget: ≈ 490 \approx 490 wall-clock hours on a single accelerator. No multi-node or cluster compute was used.

   Artifacts and licenses.

   We use the following datasets and models. BBQ (Parrish et al. , 2022 ) is released under CC-BY 4.0; we use the version distributed via HuggingFace ( Elfsong/BBQ ). CrowS-Pairs (Nangia et al. , 2020 ) is released under CC-SA 4.0. Alpaca-cleaned (Taori et al. , 2023 ) is released under CC-BY-NC 4.0 (research-only); we use it strictly for non-commercial research consistent with that license. Teacher models ( Gemma-2-9B-it (Gemma Team, 2024 ) , Mistral-7B-Instruct-v0.3 (Jiang et al. , 2023 ) , and Phi-3.5-mini-Instruct (Abdin and others, 2024 ) ) are used under their respective community-license terms (Gemma Terms of Use, Apache-2.0, and MIT). SmolLM2 (Allal et al. , 2025 ) , OLMo-2 , OpenELM , and Granite-3.1 are used under Apache-2.0. We release no new datasets; derivative model checkpoints will be released under Apache-2.0 with the original license terms inherited where applicable. Code and the evaluation pipeline are released under Apache-2.0 at https://github.com/plawanrath/knowledge-distilation-impact-analysis . Distilled model checkpoints will be released via HuggingFace.

   4 Distillation is Asymmetric

   Figure 1: Distillation has opposite effects on the two BBQ conditions, and the disambiguated-side gain must be measured on committed answers. For SmolLM2-1.7B-Instruct ( a ), dark bars condition on committed answers; light bars are the unconditioned metric. Response-Mistral genuinely improves committed-answer context-override ( 44.5 % → 37.2 % 44.5%{\to}37.2% , with accuracy rising); the large unconditioned improvement under response-Gemma ( 23 % 23% ) largely disappears among committed answers ( 39.8 % 39.8% ) because that model abstains on 43 % 43% of items and its accuracy collapses. ( b ) The same distillations introduce silence-loss on ambiguous items (baseline rate is 0 % 0% by definition).

   BBQ-ambig BBQ-disambig (anti-correct items)

   configuration SRS USR silence-loss [CI] ρ \rho override renorm. [CI] commit acc. discrim. LR + + fail

   SmolLM2-1.7B baseline 49.5 4.6 — — 43.9 44.5 [43.7, 45.3] 98.6 54.7 − 7.9 -7.9 0.0

   • • SFT (Alpaca) 39.3 28.4 2.0 [1.6, 2.6] 0.37 31.8 36.6 [35.8, 37.5] 86.8 55.0
   • 5.9 +5.9 0.0
   • • response-KD (Gemma) 22.3 59.2 1.2 [0.8, 1.7] 0.24 22.8 39.8 [38.8, 40.9] 57.3 34.5
   • 17.3 +17.3 25.2
   • • response-KD (Mistral) 47.5 13.5 14.6 [13.4, 16.0] 0.44 35.9 37.2 [36.4, 38.0] 96.5 60.6 − 2.3 -2.3 0.1
   • • response-KD (Phi) 48.7 8.9 23.8 [22.3, 25.4] 0.30 39.8 40.7 [39.9, 41.5] 97.9 58.1 − 4.8 -4.8 0.0

   OLMo-2-1B baseline 40.6 19.8 — — 42.1 45.7 [45.1, 46.4] 92.1 50.0

   • 2.8 +2.8 0.0
   • • response-KD (Gemma) 47.6 7.5 49.4 [48.4, 50.4] − 0.04 -0.04 43.2 48.4 [47.7, 49.1] 89.2 46.0 − 8.2 -8.2 87.0

   Table 1: Consolidated results for the 1.7B-scale configurations under the corrected parser cascade (sub-1.7B students are at chance on the disambiguated side and appear in the appendix; combined-KD in Appendix D ). SRS, USR, silence-loss (with 95 % 95% Wilson CI), and refusal-pattern preservation ρ \rho are computed on BBQ-ambig; unconditioned and renormalized (committed-answers-only, 95 % 95% Wilson CI) context-override, commitment rate, and accuracy are computed on anti-stereotype-correct BBQ-disambig items. discrim. is refusal discrimination (USR ambig minus USR disambig, in percentage points; positive means abstention concentrates where it is correct). LR + + fail is the share of records recovered by the keyword/content parser branches plus parse failures. All values are percentages except ρ \rho .

   The headline finding is summarized in Figure 1 . For the SmolLM2-1.7B-Instruct student (the most context-biased baseline in our grid), response-based distillation can improve the disambiguated condition while harming the ambiguous one. Establishing the improvement, however, requires conditioning on committed answers; we present the corrected metric first and the artifact it corrects second.

   On disambiguated items, KD can genuinely help.

   The 1.7B baseline commits to a substantive answer on 98.6 % 98.6% of anti-stereotype-correct disambiguated items and picks the stereotype on 44.5 % 44.5% of those committed answers. Response-Mistral distillation lowers committed-answer override to 37.2 % 37.2% (commitment 96.5 % 96.5% ) while raising disambig accuracy from 0.55 0.55 to 0.61 0.61 ; response-Phi reaches 40.7 % 40.7% with accuracy 0.58 0.58 ; SFT alone reaches 36.6 % 36.6% with accuracy 0.55 0.55 . These gains survive conditioning on committed answers and do not trade against accuracy: they are genuine context-following improvements. Table 1 consolidates all headline quantities for the 1.7B and OLMo configurations.

   The apparently strongest improvement is a refusal artifact.

   The unconditioned override metric tells a different story. Response-Gemma lowers unconditioned override from 43.9 % 43.9% to 22.8 % 22.8% , the largest apparent improvement in the grid and the number an earlier version of this paper headlined. But the same model commits on only 57.3 % 57.3% of these items, its committed-answer override is 39.8 % 39.8% (baseline: 44.5 % 44.5% ), and its disambig accuracy collapses from 0.55 0.55 to 0.35 0.35 : the residual committed-answer reduction comes from a model that has partly stopped tracking context altogether, not one that follows it better. The model did not learn context-following; it learned to abstain, and the unconditioned metric scores abstention as fairness. § 5 develops this failure mode.

   On ambiguous items, the same KD hurts.

   The 1.7B baseline correctly refuses (selects “unknown”) at USR = 4.6 % =4.6% . Under response-Mistral, marginal USR rises to 13.5 % 13.5% , yet 14.6 % 14.6% of the (item, seed) pairs where the baseline correctly refused now receive stereotype answers, and refusal-pattern preservation is ρ = 0.44 \rho{=}0.44 . We call the first quantity silence-loss : the loss of conditional refusal behavior. Response-Phi shows silence-loss of 23.8 % 23.8% . Response-Gemma moves the most mass into refusal (marginal USR rises to 59 % 59% ) and consequently shows silence-loss of only 1.2 % 1.2% , but its refusal pattern is heavily remapped ( ρ = 0.24 \rho{=}0.24 ): it refuses far more, on a largely different set of items. Base rates matter for interpretation: baseline USR is 4.6 % 4.6% , so silence-loss concerns a thin slice of all items (about 0.7 % 0.7% of (item, seed) pairs for response-Mistral) even as marginal USR moves by far more; we report both directions throughout.

   Cross-condition structure is a metric question.

   An earlier version of this work reported that silence-loss and context-override were uncorrelated across the grid ( ρ ≈ 0.19 \rho{\approx}0.19 , n.s.) and inferred two independent mechanisms. That computation had three flaws: it included parser-invalid logit-KD configurations, it scored several configurations with a contaminated parser branch (Appendix A ), and it used the unconditioned override metric. On the valid grid with renormalized override and the corrected parser (Figure 2 ), the two quantities are positively correlated: ρ = 0.58 \rho{=}0.58 ( p = 0.009 p{=}0.009 , n = 19 n{=}19 ), or ρ = 0.68 \rho{=}0.68 ( p = 0.002 p{=}0.002 ) on the 18 18 SmolLM2 configurations alone, driven by configurations that degrade on both conditions at once. The figure also shows a structural constraint: every sub-1.7B configuration sits at the 50 % 50% chance line on the renormalized axis because those students are at chance accuracy on disambiguated items, so cross-condition structure is interpretable only for 1.7B-scale students in our grid. We therefore state this result as a correction, the earlier independence claim does not survive metric and parser correction, rather than as an established positive coupling: the observed correlation is partly structural and its magnitude depends on grid composition.

   Figure 2: Cross-condition structure on the valid grid. Across 19 19 valid KD configurations (logit-KD excluded as parser-invalid; SFT controls excluded), silence-loss (BBQ-ambig) and committed-answer context-override (BBQ-disambig) are positively correlated ( ρ = 0.58 \rho{=}0.58 , p = 0.009 p{=}0.009 ). Sub-1.7B students sit at the 50 % 50% chance line because they are at chance accuracy on disambiguated items. An earlier version reported ρ ≈ 0.19 \rho{\approx}0.19 (n.s.) using the unconditioned override metric and parser-invalid configurations.

   The rest of the paper characterizes the refusal side of this picture. Sections 6 – 7 localize where newly-introduced bias arises and evaluate a data-side account of its source; § 8 proposes the diagnostic protocol (PCCD) that surfaces both the ambiguous-side harm and the metric artifacts.

   Second-family confirmation.

   The ambiguous-side pattern is not a SmolLM2 artifact; on OLMo-2-1B-Instruct distilled with response-Gemma it appears in aggravated form. OLMo’s baseline refuses often (USR 20 % 20% ), leaving far more silence to fill, and distillation fills it: silence-loss is 49 % 49% on the cases where the OLMo baseline correctly refused, marginal USR falls 0.20 → 0.07 0.20{\to}0.07 , SRS rises 0.41 → 0.48 0.41{\to}0.48 , and the refusal pattern decorrelates completely ( ρ = − 0.04 \rho{=}{-}0.04 ). 95 % 95% of newly-introduced stereotype answers fall on items where the teacher itself refused (filled-silence dominates, as for SmolLM2). The disambiguated side shows no improvement in any form: renormalized override rises 45.7 % → 48.4 % 45.7%{\to}48.4% and accuracy falls 0.50 → 0.46 0.50{\to}0.46 (§ 10 ).

   5 Bias Metrics Mislead When Refusal Mass Moves

   Three facts about the 1.7B response-Gemma model are simultaneously true: (i) on aggregate stereotype metrics it looks like a clear win; (ii) on the unconditioned conditional metric (context-override) it looks like the grid’s strongest improvement; (iii) at the per-item level its refusal pattern is decorrelated from the baseline and its disambig accuracy has collapsed. The first two are artifacts of moving refusal mass.

   CrowS-Pairs.

   Whole-sentence pseudo log-likelihood scoring on 1 , 508 1{,}508 pairs gives the 1.7B baseline a stereotype preference of 56.6 % 56.6% . After response-Gemma distillation it is 47.0 % 47.0% (response-Mistral: 48.9 % 48.9% ), reductions with no signal of harm. The metric averages over items and contexts and does not distinguish “refusal flipped to anti-stereotype” from “refusal flipped to stereotype.”

   Overall BBQ Stereotype Reliance Score (SRS).

   The 1.7B baseline scores SRS = 0.50 =0.50 on BBQ-ambig; response-Gemma reduces this to 0.22 0.22 . Again, the aggregate looks beneficial. Per-item analysis (Figure 4 , panel (b)) reveals that the 0.22 0.22 aggregate arises from a refusal set that overlaps the baseline-refusal set with ρ = 0.24 \rho{=}0.24 : the model is refusing on different items, not on the right items.

   Unconditioned conditional metrics fail the same way.

   Context-overriding conditions on the correct answer being the anti-stereotype, but not on the model committing to an answer. Because distillation moves refusal mass onto disambiguated items (response-Gemma commits on 57 % 57% of anti-stereotype-correct items vs. the baseline’s 99 % 99% ), the unconditioned rate credits abstention as if it were context-following. Renormalizing to committed answers (Figure 1 a) shrinks the apparent 44 % → 23 % 44%{\to}23% improvement to 44.5 % → 39.8 % 44.5%{\to}39.8% and reveals the accompanying accuracy collapse ( 0.55 → 0.35 0.55{\to}0.35 ). The same correction reverses the cross-condition independence result (§ 4 ); together with the parser correction it also eliminates OLMo’s apparent disambiguated-side improvement entirely (§ 10 ).

   Implication.

   A practitioner who screens distilled models on CrowS-Pairs, overall BBQ SRS, or unconditioned context-override will deploy a model whose refusal behavior has been redistributed and whose accuracy may have collapsed, properties that matter in safety-critical settings requiring abstention under genuine uncertainty. The methodological concern echoes prior critiques of pair-based fairness benchmarks: Blodgett et al. ( 2021 ) document inconsistencies in what these benchmarks claim to measure, and Goldfarb-Tarrant et al. ( 2021 ) show that intrinsic bias scores correlate poorly with downstream task fairness. Refusal-pattern preservation ρ \rho , silence-loss, and committed-answer renormalization are conditional analyses that surface what aggregate and unconditioned metrics average away.

   6 Cell Decomposition and Per-Item Calibration

   Figure 3: Decomposition of newly-introduced stereotype answers (response-KD only). For each (student × \times teacher) configuration we partition new-bias items into Inherited (teacher also stereo), Amplified-vs-Anti (teacher anti), and Filled-Silence (teacher refused). Direct contradiction is rare ( ≤ 13 % \leq 13% ); filled-silence dominates for Gemma and Phi teachers.

   To localize the mechanism, we classify each post-distillation stereotype answer by the teacher’s behavior on the same prompt (Figure 3 ).

   Direct contradiction of the teacher is rare.

   Amplified-vs-Anti (items where the teacher labeled the anti-stereotype but the student picks the stereotype) accounts for ≤ 13 % \leq 13% of newly-introduced bias across all 9 9 response-KD configurations. Distilled students do not amplify bias by overriding teacher anti-stereotype labels. (All decomposition numbers use the corrected parser cascade; Appendix A reports per-configuration parse quality.)

   Silence-filling dominates for clean teachers.

   For Gemma-distilled students, 93 − 95 % 93{-}95% of newly-introduced bias appears on items where the teacher refused ( Filled-Silence ). The cleanest teacher by USR ( 88 % 88% ) produces the highest silence-filling rate. For Mistral (the most engaged teacher, 54 % 54% USR), inheritance reaches 44 % 44% and silence-filling drops to 50 % 50% . The total transition rate (the share of baseline-non-stereotype answers that become stereotype answers post-distillation) ranges 13 − 35 % 13{-}35% across response configurations and is lowest exactly where the distilled model moved the most mass into refusal (1.7B response-Gemma, 13 % 13% ): fewer committed answers mechanically mean fewer new stereotype answers, the same refusal-mass effect that inflates the override metric. The routing tracks the teacher’s refusal profile. The cross-family OLMo-2-1B + response-Gemma config shows the same shape: 95 % 95% filled-silence with only 5 % 5% inheritance and 0.5 % 0.5% amplified-vs-anti, confirming that the silence-filling pathway is a property of distillation-into-small-instruct-LMs broadly, not of SmolLM2 specifically.

   Counter-intuitive consequence.

   Choosing a less-biased teacher does not yield a better-calibrated distilled student. It changes the route by which bias appears (more silence-filling, less inheritance), and where it does lower the total rate of newly-introduced bias, it does so by moving answers into indiscriminate refusal rather than by improving context-following.

   6.1 Refusal-Pattern Preservation

   Figure 4: ( a ) Refusal-pattern preservation ρ \rho across 22 22 valid configurations ( 18 18 SmolLM2 response/combined KD, 3 3 SFT controls in olive, OLMo response-Gemma in green; logit-KD excluded as parser-invalid, Appendix A ). Red bars are negative. Two configurations exceed the proposed τ calib = 0.5 \tau_{\mathrm{calib}}{=}0.5 threshold. ( b ) 2D histogram of per-item USR for SmolLM2-1.7B response-Gemma: x x -axis is the baseline student’s per-item refusal rate (over 5 5 seeds), y y -axis is the distilled student’s. The dashed line marks perfect preservation; mass scattered off-diagonal shows the distilled model refuses on largely different items than the baseline, despite similar aggregate refusal rates.

   We compute, for each (model, KD method, teacher) tuple, the Spearman correlation between baseline-student per-item USR and distilled-student per-item USR. Figure 4 (a) shows the distribution; panel (b) shows the per-item scatter for 1.7B response-Gemma.

   Headline numbers.

   Response-KD yields ρ = 0.12 \rho{=}0.12 to 0.60 0.60 across the 9 9 (student × \times teacher) configurations and combined-KD yields − 0.10 {-}0.10 to 0.54 0.54 ; the three SFT controls span 0.37 0.37 to 0.42 0.42 . Only two valid configurations (360M-Phi under both objectives) exceed 0.5 0.5 , twelve of the 19 19 valid KD configurations fall below the weakest SFT control, and the OLMo configuration is indistinguishable from zero ( ρ = − 0.04 \rho{=}{-}0.04 ). We no longer report logit-KD correlations: Appendix A shows those configurations produce degenerate generations whose apparently anti-correlated ρ \rho values are artifacts of the answer parser’s last-resort branch, consistent with documented difficulties of logit-based distillation across mismatched tokenizers (Boizard et al. , 2024 ) .

   Marginal vs. per-item.

   Crucially, the marginal USR can look unremarkable while the pattern remaps: under response-Gemma it rises for the 1.7B student ( 0.046 → 0.592 0.046{\to}0.592 ) but falls for the smaller students (360M 0.313 → 0.287 0.313{\to}0.287 ; 135M 0.319 → 0.179 0.319{\to}0.179 ), and in all three cases the per-item pattern is weakly preserved at best. A student that refuses at a similar or higher rate but on different items has not gained safety.

   Refusal discrimination.

   Because ρ \rho is baseline-anchored, Table 1 also reports refusal discrimination (USR on ambiguous minus USR on disambiguated items), which is anchored to ground truth. No configuration achieves well-calibrated abstention: the 1.7B baseline itself has negative discrimination ( − 7.9 -7.9 pp: it abstains more where context determines the answer than where it does not), response-Mistral and response-Phi stay slightly negative ( − 2.3 -2.3 and − 4.8 -4.8 pp), and the largest positive value (response-Gemma, + 17.3 +17.3 pp) comes from a global upward shift that still leaves the model refusing on 42 % 42% of disambiguated items. Distilled OLMo is the worst case at − 8.2 -8.2 pp, refusing twice as often where it should answer as where it should abstain.

   7 Mechanism: Training-Corpus Refusal Absence

   Figure 5: Opener-refusal rate (fraction of training responses that begin with a refusal phrase such as “I don’t know”, “Cannot determine”, etc.) is < 0.5 % <{}0.5% across all four training corpora.

   If silence-loss reflects lost conditional refusal, what is the source? We hypothesize that the training corpora themselves lack refusal-as-answer-shape examples, so the student can learn the marginal frequency of refusal but not the conditional structure.

   Audit.

   We audit four corpora ( n = 51 , 760 n{=}51{,}760 each): Alpaca-cleaned SFT and three teacher response sets. Opener-refusal rate (response begins with a refusal phrase) is 0.18 % 0.18% , 0.33 % 0.33% , 0.05 % 0.05% , 0.06 % 0.06% for Alpaca, Gemma, Mistral, Phi respectively (Figure 5 ). Even these counts are inflated: only ∼ 18 % \sim 18% of these opener-refusal hits are true refusal-as-answer-shape under manual inspection (Appendix F ); the rest are hedge-then-answer patterns or dialogue-context refusals.

   For the MCQ subset of each corpus (prompts containing A)/B)/C) options, n = 495 − 931 n{=}495{-}931 per corpus), abstention rate is 0 − 0.5 % 0{-}0.5% .

   Predictive association.

   We use sentence-transformers embeddings to count, for each BBQ item, the number of training-corpus responses within a cosine similarity threshold that contain a refusal opener, and compare refusal preservation between items with and without at least one such neighbor. An earlier version described this as a binary chi-squared test; the implemented tests are a one-sided Mann-Whitney U and a Welch t-test, and we report them as such, with Holm correction across the 19 19 valid configurations. The association is heterogeneous: the difference ranges from − 9.7 -9.7 to + 21.3 +21.3 percentage points, is positive in 12 12 of 19 19 configurations (mean + 1.8 +1.8 pp), and is significant after correction in 3 3 (from + 8.9 +8.9 to + 21.3 +21.3 pp, all Mistral-teacher configurations). Gemma-teacher configurations show null or negative associations, and refusal-neighbor groups are small in some configurations (as few as 19 19 items). This is a correlational, data-side signal, not an established mechanism.

   Mitigation attempt: SFT-time refusal injection.

   We tested whether SFT-time refusal-injection mitigations can restore conditional refusal calibration on the 1.7B student. Two formats are compared (Appendix B ): a text-format injection that asks the model to produce a natural-language refusal phrase, in the spirit of Gupta et al. ( 2022 ) ’s counterfactual role reversal but targeted at answer-shape rather than the gender axis; and a letter-format injection that asks the model to output the letter of the unknown option in BBQ format. Neither restores conditional refusal. The text-format variant collapses parsing at moderate injection rates (parse-fail is essentially 100 % 100% at both 5 % 5% and 7 % 7% injection under the corrected cascade) because the model emits literal refusal phrases instead of letter tokens. The letter-format variant at 5 % 5% injection over-corrects into a trivial-refuser regime: USR rises to 99.8 % 99.8% , silence-loss falls to 0 % 0% (the model never loses refusal because it never gives answers), but disambig accuracy collapses to 0.02 % 0.02% . The trivial-refuser model drives unconditioned context-override to 0.01 % 0.01% (it never picks a stereotype because it never picks anything), yet it commits on 0.03 % 0.03% of disambiguated items, so its renormalized override is not meaningfully estimable, and it fails Step 1 ( ρ = 0.015 \rho{=}0.015 ) and Step 3 ( 0.02 % 0.02% accuracy) of PCCD (§ 8 ). This is a SmolLM2-specific instance of the broader over-refusal phenomenon documented by Cui et al. ( 2025 ) , which finds a Spearman correlation of 0.89 0.89 between safety and over-refusal across major LLMs. This negative result rejects the simplest form of the data-side account, that refusal density is what the corpus is missing, and refines it: what is missing is refusal conditioned on ambiguity . Adding unconditional refusal density does not transfer the structure that distinguishes “refuse when ambiguous” from “refuse always”; the paired-injection experiment described in the Discussion is the direct test of the refined account.

   8 PCCD: A Diagnostic Protocol

   We propose Per-Condition Calibration Diagnosis (PCCD) : a minimal three-step protocol for distillation evaluation that exposes the asymmetric harm and the trivial-refuser failure mode that aggregate-only checks miss. PCCD assumes access to an evaluation set partitioned into (i) uncertainty-correct items, where abstention is the correct response, and (ii) context-determinable items, where a correct answer is fixed by the prompt. Any QA benchmark that exposes this split with an abstention option supports the protocol; we instantiate PCCD on BBQ in this paper, but the construction is benchmark-agnostic.

   Step 1 (refusal-pattern preservation).

   On the uncertainty-correct subset, compute refusal-pattern preservation ρ \rho (Spearman correlation between baseline-student and distilled per-item abstention rates). Pass if ρ ≥ τ calib \rho\geq\tau_{\mathrm{calib}} . This step is anchored to the baseline student, not to ground truth; it detects disruption of an existing refusal pattern.

   Step 2 (context-following).

   On the context-determinable subset, compute the context-overriding rate renormalized to committed answers (among responses that select a substantive option, the fraction that override a correct context with the stereotype), and report the commitment rate alongside it. Pass if the renormalized rate does not exceed the baseline-student’s renormalized rate. The renormalization is essential: the unconditioned rate can be driven to zero by refusing on every item (Appendix B ).

   Step 3 (capability preservation).

   On the context-determinable subset, compute overall accuracy. Pass if accuracy ≥ τ acc \geq\tau{\mathrm{acc}} . We recommend τ acc \tau{\mathrm{acc}} equal to baseline-student accuracy on this subset (a no-regression criterion). Step 3 catches trivial-refuser models that pass Steps 1–2 by abstaining on every item.

   We recommend τ calib = 0.5 \tau{\mathrm{calib}}{=}0.5 based on the observed distribution of ρ \rho (Figure 4 , range − 0.10 -0.10 to + 0.60 +0.60 over valid configurations). The threshold is heuristic: baseline per-item USR distributions differ across students (the 1.7B baseline refuses on 0 of 5 5 seeds for 93 % 93% of items, which mechanically attenuates ρ \rho through ties), so ρ \rho values are not directly comparable across students and we treat τ calib \tau{\mathrm{calib}} as a within-student screen (see Limitations). Step 1 catches refusal-pattern disruption invisible to marginal-rate or aggregate-pair metrics; Step 2 catches regressions in committed-answer context-following; Step 3 catches capability collapse from over-aggressive refusal injection (Appendix B ).

   Applying PCCD to our grid.

   Of the 19 19 valid KD configurations: 12 12 pass Step 2 (renormalized), 2 2 pass Step 1, and 13 13 pass Step 3. At the scales where the disambiguated side is informative (1.7B and OLMo-1B; the sub-1.7B students hover at chance on Steps 2 and 3, so their passes reflect chance-level movement), no configuration passes all three. One 360M configuration (response-Phi) passes all three nominally, but its Step 2 margin is 0.3 0.3 pp around the 50 % 50% chance level, so we do not read it as a demonstrated success. 1.7B response-Mistral and response-Phi pass Steps 2 and 3 and fail only Step 1 ( ρ = 0.44 \rho{=}0.44 and 0.30 0.30 against τ calib = 0.5 \tau_{\mathrm{calib}}{=}0.5 ); response-Mistral is closest overall, with accuracy improving. OLMo-2-1B response-Gemma fails all three steps outright ( ρ = − 0.04 \rho{=}{-}0.04 ; renormalized override 48.4 % 48.4% vs. baseline 45.7 % 45.7% ; accuracy 46 % 46% vs. baseline 50 % 50% ). The SFT-time mitigation variants (Appendix B ) show why Step 2 must be renormalized: the letter-format injection drives unconditioned override to 0.01 % 0.01% , far below baseline, yet commits on 0.03 % 0.03% of disambiguated items and scores 0.02 % 0.02% accuracy. PCCD outputs a (pass/fail, pass/fail, pass/fail) tuple that surfaces both failure modes.

   9 KD vs. SFT and Per-Group Effects

   KD vs. SFT.

   KD-vs-SFT chi-squared tests at the (model, category, teacher) level confirm KD-specific effects beyond SFT-general shifts: response-KD raises the stereotype rate above same-student SFT in 73 % 73% of cells, and 36 % 36% of all valid KD cells are individually significant with KD above SFT (Appendix D breaks the headline quantities out by KD method). The KD-specific increment also runs against a small-capacity explanation: the transition-rate increment over SFT is absent for the 135M student (mean − 0.4 -0.4 pp, range − 9 -9 to + 4 +4 ) and substantial for the 360M ( + 5 +5 to + 32 +32 pp) and 1.7B ( − 4 -4 to + 15 +15 pp) students. If limited capacity alone drove stereotype amplification, the smallest student should show the largest KD-specific increment; we observe the opposite.

   Per-group fairness.

   Per-group breakdowns (Appendix C ) show response-Gemma KD reduces the highest-baseline group (Muslim) most ( − 0.44 -0.44 ); other teachers do not.

   10 Generalization Across Model Families

   Baselines across four families.

   Baseline BBQ-ambig SRS on three additional small instruction-tuned LMs from outside our distillation grid (OLMo-2-1B-Instruct (OLMo Team et al. , 2025 ) , OpenELM-1.1B-Instruct (Mehta et al. , 2024 ) , Granite-3.1-2B-Instruct (Granite Team, IBM, 2024 ) ) ranges 0.22 − 0.50 0.22{-}0.50 across these four families plus SmolLM2 (Appendix E ); the extreme high-SRS pattern of SmolLM2-1.7B is partly tuning-recipe-specific.

   Cross-family distillation: OLMo-2-1B + response-Gemma.

   Distilling OLMo-2-1B-Instruct from Gemma-2-9B-it (response-KD, same corpus and hyperparameters as the SmolLM2 grid) reproduces the ambiguous-side harm in aggravated form. The distilled model answers where the baseline abstained: marginal USR falls 0.20 → 0.07 0.20{\to}0.07 , SRS rises 0.41 → 0.48 0.41{\to}0.48 , silence-loss reaches 49.4 % 49.4% (the highest in the grid), and refusal-pattern preservation is absent ( ρ = − 0.04 \rho{=}{-}0.04 ). The cell decomposition is 95 % / 5 % / 0.5 % 95%/5%/0.5% (filled-silence/inheritance/amplified-vs-anti), matching the SmolLM2-Gemma shape. The disambiguated side shows no improvement in any form: unconditioned override rises 42.1 % → 43.2 % 42.1%{\to}43.2% , renormalized override rises 45.7 % → 48.4 % 45.7%{\to}48.4% , and accuracy on anti-stereotype-correct items falls 50.0 % → 46.0 % 50.0%{\to}46.0% . An earlier version of this work reported different OLMo numbers (silence-loss 8.3 % 8.3% , ρ = 0.65 \rho{=}0.65 , an apparent unconditioned override improvement); those were artifacts of a parser branch that mis-scored the distilled model’s verbose answer format, and Appendix A documents the correction.

   The ambiguous-side harm reproduces, amplified, on a second student family. PCCD verdict for OLMo response-Gemma: (FAIL, FAIL, FAIL) . We do not distill OpenELM or Granite; this is a stated limitation.

   11 Discussion

   Why does this happen?

   A plausible account is data-side: the student’s parametric prior for “what does an answer look like?” is shaped almost entirely by the training corpus, which contains almost no conditional refusal. On this account, distillation transfers marginal refusal frequency from teacher to student (when the teacher refuses often), but the conditional structure that maps prompts to refusals lives in a part of the corpus that does not exist. The evidence for this account in § 7 is correlational; the discriminating experiment is paired injection of (ambiguous, refuse) and (disambiguated, answer correctly) exemplars, which the account predicts should restore conditional refusal where density-only injection failed. This decoupling of bias from raw capability mirrors related compression studies on pruning (Rath and Maliakkal, 2026b ) and quantization (Rath and Maliakkal, 2026a ; Marcuzzi et al. , 2025 ) , where standard quality metrics fail to flag fairness-critical degradation.

   Alternative explanations.

   Four rival accounts deserve explicit statement. Training objective : SFT and response-KD share the same objective on the same prompt set, differing only in the response corpus, so the objective alone cannot explain KD-specific effects. Teacher quality : varying the teacher changes the route by which new bias appears (§ 6 ), but every teacher produces silence-filling; teacher choice modulates the effect without removing it. Student capacity : the KD-specific increment over SFT is smallest for the smallest student (§ 9 ), the opposite of what a capacity-limit account predicts. Decoding and format : temperature, token budget, and the MCQ template are held fixed across configurations, so they cannot explain between-configuration differences, though sensitivity to these choices is untested here (Limitations). The axis that co-varies with the harm is the response corpus’s refusal content, which is why paired injection is the discriminating experiment.

   What does this mean for practice?

   The mainstream prescription, debias the teacher, addresses only the inheritance pathway. Our cell decomposition shows inheritance is ≤ 44 % \leq 44% of newly-introduced bias and is often as low as 5 % 5% for clean teachers. The dominant pathway is silence-filling, and the locus of the fix is the training corpus, not the teacher. However, naive refusal-density injection over-corrects into a trivial-refuser regime (Appendix B ); corpus-side fixes require paired (ambiguous, refuse) and (disambiguated, answer) exemplars. Evaluation-side, conditional bias metrics must be computed on committed answers, and cross-configuration comparisons must first screen out configurations whose outputs the answer parser cannot legitimately score.

   12 Conclusion

   Distillation has asymmetric effects on bias, and refusal mass is central both to the harm and to its measurement. At 1.7B scale, response-KD can genuinely improve committed-answer context-following while degrading conditional refusal on ambiguous items. Apparently stronger improvements, and an apparent cross-condition independence, dissolve once metrics are conditioned on committed answers and parser-invalid configurations are excluded. Training corpora contain near-zero refusal-as-answer-shape and refusal-density injection over-corrects into trivial refusal, consistent with a data-side account whose causal test is paired injection of (ambiguous, refuse) and (disambiguated, answer) exemplars. PCCD surfaces both the ambiguous-side harm and the metric artifacts that aggregate and unconditioned evaluations conceal.

   13 Limitations

   Two student families, both ≤ 2 \leq 2 B parameters.

   We distill SmolLM2 ( 135 135 M, 360 360 M, 1.7 1.7 B) and OLMo-2-1B-Instruct (§ 10 ). Multi-family baselines (Appendix E ) extend to OpenELM-1.1B and Granite-3.1-2B but we do not distill those families. Findings should not be assumed to hold for substantially larger ( ≥ 7 \geq 7 B) or closed-weight models.

   MCQ format.

   We evaluate at max_tokens = 5 ={}5 on MCQ-format BBQ. Generative bias (free-form completions) is unstudied here.

   Sub-1.7B students are at chance on BBQ-disambig.

   The 135M and 360M students score near-chance accuracy on disambiguated items before training ( 34 % 34% ; chance is 33 % 33% ) and 33 − 43 % 33{-}43% after, and every sub-1.7B configuration’s renormalized override sits within 1.5 1.5 pp of the 50 % 50% two-way chance level. Disambiguated-condition claims in this paper are therefore restricted to 1.7B-scale students and OLMo-2-1B; for smaller students only the ambiguous-condition results are informative.

   Parser dependence.

   Answers are extracted by a deterministic parser cascade. The cascade used in an earlier version of this work included a bare first-character scan that manufactured position-biased letters for verbose or truncated outputs; all numbers here use a corrected cascade without it (Appendix A ). The correction leaves the exemplar configurations essentially unchanged (they parse > 99.9 % >99.9% by exact or prefix match) but materially revises configurations with non-letter output formats, above all the OLMo distilled arm, and up to 10 % 10% of records in the most affected retained configurations still fail to parse. Metrics for those configurations are computed over parseable records only.

   Refusal-pattern preservation is baseline-anchored.

   ρ \rho measures whether the distilled student’s per-item refusal pattern matches the baseline student’s , which is itself not a gold standard. Additionally, the 1.7B baseline refuses on 0 of 5 5 seeds for 93 % 93% of items, so its ρ \rho values are mechanically attenuated by ties, and cross-student comparisons of ρ \rho partly reflect baseline refusal variance. Silence-loss does not share this concern (on BBQ-ambig the unknown option is correct for every item, so it is anchored to ground truth), and we report the ground-truth-anchored refusal discrimination metric (USR on ambiguous minus USR on disambiguated items) alongside ρ \rho in Table 1 .

   Single training data source.

   The conditional-refusal absence we identify is a property of Alpaca-cleaned. Other SFT corpora may differ; we do not vary this axis.

   Logit-KD with mismatched tokenizers.

   Logit-KD across mismatched tokenizers fails (perplexity 10 5 − 10 6 10^{5}{-}10^{6} ; Appendix A ), so our grid cannot separate the logit objective’s effect from tokenizer degeneration; a matched-tokenizer teacher/student pair or a token-mapping bridge (Boizard et al. , 2024 ) would be required for that comparison. Our headline analysis uses response-based distillation as the clean condition.

   Inference hardware.

   We use Apple Silicon (MLX) inference with bfloat16 weights. 4-bit and 8-bit quantization variants are not separately evaluated; Rath and Maliakkal ( 2026a ) document bias emergence under quantization on a comparable set of small instruction-tuned LMs.

   No counterfactual generation.

   We do not evaluate counterfactual fairness or open-ended generation bias; our scope is multiple-choice abstention/SRS.

   English-only evaluation.

   BBQ and CrowS-Pairs are English-language benchmarks built against North American social context. Findings may not transfer to other languages or cultural contexts. Stereotype categories in BBQ (Religion, Race-Ethnicity, SES, etc.) reflect a US-centric framing.

   Definition of bias used.

   Following Parrish et al. ( 2022 ) , we operationalize “bias” as differential per-item preference for stereotype-aligned answers over the unknown-option in ambiguous contexts, and as context-override on disambiguated contexts. This is a narrow, behavioural definition; it does not capture representational harms, generative bias, or downstream-task harms in the sense of Blodgett et al. ( 2021 ) .

   Ethical Considerations

   Potential risks.

   The findings in this paper are most likely to be misused in two ways. (i) False assurance of safety from aggregate metrics. We show distilled small LMs can pass CrowS-Pairs and overall-SRS checks while exhibiting substantial per-item refusal miscalibration. Practitioners deploying compressed models to safety-sensitive settings (medical triage, content moderation, automated assistance for marginalized groups) should not interpret aggregate stereotype scores as evidence of calibrated abstention. We propose PCCD specifically to mitigate this risk. (ii) Gaming the PCCD diagnostic. A bad actor could fine-tune on the BBQ-ambig validation set used to set τ calib \tau_{\mathrm{calib}} , producing a model that passes the diagnostic without genuine calibration improvements. We recommend that deployments using PCCD hold out a fresh validation slice that is not used for model selection.

   Impact on vulnerable populations.

   BBQ contains items targeting a range of social categories (age, socio-economic status, nationality, etc.). Our cell decomposition and per-category breakdowns (§ 9 ) report findings across these categories. Distilled models that lose refusal calibration are most likely to introduce stereotype answers on items where the baseline refused, meaning the harm falls disproportionately on the group-stereotype combinations the baseline was correctly avoiding. Our results indicate caution in deploying compressed small LMs to user-facing settings without per-item validation.

   Dual-use of refusal-injection mitigation.

   The letter-format refusal-injection mitigation (Appendix B ) could in principle be repurposed to make models refuse selectively, e.g., to suppress legitimate queries about specific groups. We restrict our refusal exemplars to ambiguous BBQ-format items drawn from held-out categories; this is a narrow operating regime that does not generalize to arbitrary refusal injection. We do not release the augmented training data publicly.

   Over-generalization.

   The headline asymmetry is established for SmolLM2 students; one additional family (OLMo-2-1B) is included as a generalization probe (§ 10 ). The asymmetry should not be assumed to hold for: closed-weight commercial models, models substantially larger than 2 2 B parameters, non-English models, or models fine-tuned on substantially different instruction corpora.

   Use of AI assistants.

   Generative-AI assistants (a large language model coding assistant) were used during code authoring and figure generation, and for grammar and clarity polishing of the manuscript text. The authors retain full responsibility for all scientific claims, experimental decisions, statistical analyses, and the final text. No AI-generated content was included without authorial review.

   Acknowledgments

   Compute for distillation, inference, and analysis was provided by personal Apple Silicon hardware. We thank the authors of the SmolLM2, Gemma-2, Mistral, and Phi-3.5 model families for open releases that made this study possible, and the authors of the BBQ and CrowS-Pairs benchmarks for releasing their datasets under permissive licenses.