事実関係
arXiv:2608.07528v1 Announce Type: new Abstract: Linear probes detect corrupted context in language models with near-perfect accuracy, yet this does not translate into reliable failure prediction. The result is a dissoci
解説と影響
来自 arXiv 的最新论文提出了“Knowing-Saying Gap”（知行差距）这一概念。研究者发现，当语言模型的上下文窗口被注入错误或误导性信息时，通过在其内部表征上训练简单的线性分类器（即线性探针），可以几乎毫无差错地识别出模型正处于“被污染”的状态。这表明模型内部确实“知道”输入存在问题。然而，这种深层的察觉与模型最终输出的行为表现出了显著的脱节。模型并不会因为内部检测到了异常而降低其回答的置信度，或者拒绝生成不可靠的内容。
这种差距指向了当前模型在安全性与可靠性上的一个根本性挑战。线性探针作为一种事后解释工具，能够读取模型内部表征中蕴含的丰富信息，但这部分信息并没有被模型在生成回答的推理过程中有效利用。换句话说，模型在“思考”时捕捉到了危险信号，但在“表达”时却忽略了它，依然可能以高置信度生成基于错误前提的结论。原文摘要指出，这导致了一种分离现象：极高的内部检测精度无法转化为有效的故障预测能力。
该研究对于提升高风险场景下的 AI 安全性具有启示意义。如果模型无法自发地将内部状态监测与输出控制联动起来，那么单纯依赖模型自身的置信度分数来判断其回答是否可靠将存在巨大隐患。未来的研究可能需要探索新的训练范式或架构设计，以弥合这种“知”与“行”之间的鸿沟，让模型不仅能“意识”到问题，还能据此调整自己的行为。
参考資料
出典原文
arXiv:2608.07528v1 Announce Type: new Abstract: Linear probes detect corrupted context in language models with near-perfect accuracy, yet this does not translate into reliable failure prediction. The result is a dissociation with direct implications for deployment monitoring. Across multi-hop arithmetic chains, probes that detect corruption turn out to be uninformative about final answer correctness; models forced into structured confidence formats collapse to two values with indistinguishable error rates; and probe persistence across hops fails to separate correct from incorrect outcomes, refuting our pre-registered "persistence beats peak" hypothesis. This pattern of knowing but not saying generalises across model families including reasoning models. As a real-time monitor, probe-based interventions are sharply model and error-type dependent: branch-and-pick is net-positive across models and uniquely non-breaking on Llama-3.1-8B (4 rescued, 0 broken), while reprompt and replace-prior break correct traces at roughly the rate they rescue wrong ones. Probe-based monitoring is a necessary complement to verbalised confidence, but no single intervention dominates, and the deployable answer is model-aware, error-type-aware routing.