What happened
arXiv:2608.06398v1 Announce Type: new Abstract: Recent byte-level large language models (LLMs) have made tokenizer-free modeling increasingly competitive by grouping bytes into dynamically sized patches. However, existi
Analysis and impact
正文
近年来，字节级大语言模型通过将原始字节动态组合成可变长度的“块”（patches），绕过了传统分词器的限制，展现出与基于分词器的模型相抗衡的潜力。然而，这种灵活性也带来了新挑战：不同信息密度的字节块被送入混合专家（MoE）架构时，固定的路由策略难以有效区分哪些块需要激活哪些专家。EntropyMoE 的核心思路正是利用信息熵作为路由决策的内在信号，为无分词器模型提供一种更适配的稀疏激活方案。
从技术实现来看，EntropyMoE 在路由模块中引入了一个熵感知机制。模型会先计算每个输入字节块的熵值，这一数值反映了该块内部字节分布的均匀程度或信息丰富度。随后，路由网络依据熵值动态调整专家选择的门控权重，使得高熵块（通常包含更多样或关键的信息）能够被分配给更擅长处理复杂模式的专家，而低熵块则路由至更轻量或专门的专家。这种设计无需改变 MoE 的基础架构，仅通过修改路由逻辑，就实现了对无分词器模型输入特性的适配。
这一方法的直接价值体现在推理效率与模型性能的平衡上。传统的 MoE 路由对所有输入“一视同仁”，可能导致部分专家过载而其他专家闲置，尤其在处理字节级输入这种粒度差异极大的数据时更为明显。EntropyMoE 的熵感知路由有助于实现更均衡的专家负载，在保持模型表达能力的同时，降低计算浪费。对于追求端到端、无需分词器的多语言或跨模态大模型而言，这项技术提供了一条优化推理成本的可行路径。
值得注意的是，EntropyMoE 的研究与同期关于 MoE 架构可解释性的探索形成了呼应。例如，相关素材中《Beyond Routing Weights》一文聚焦于 MoE 奖励模型的可信解释，试图揭示专家决策背后的驱动因素。EntropyMoE 虽侧重于效率，但其基于熵的路由逻辑本身也提供了一种直观的决策依据：专家分配与输入的信息量直接挂钩，这在一定程度上增强了模型行为的可理解性，为未来构建更透明、更高效的稀疏模型提供了新思路。
References
Original source text
arXiv:2608.06398v1 Announce Type: new Abstract: Recent byte-level large language models (LLMs) have made tokenizer-free modeling increasingly competitive by grouping bytes into dynamically sized patches. However, existing byte-patch architectures still apply the same dense feed-forward computation to every patch. This uniform computation cannot adapt model capacity to variations in patch semantics and granularity. We address this limitation with EntropyMoE, a Mixture-of-Experts (MoE) architecture designed for dynamic byte patches. EntropyMoE replaces the dense feed-forward modules in the global patch Transformer with Top-K expert layers. Each dynamic patch serves as the basic unit of expert routing, and its byte coverage determines its contribution to workload accounting. The router selects experts directly from patch entropy, using the same granularity signal that underlies dynamic patch construction to organize sparse computation. Patch entropy and length jointly define the feature space for regulating expert specialization. Experiments show that EntropyMoE achieves the lowest held-out bits-per-byte among matched dense and sparse baselines while maintaining comparable downstream accuracy. These results establish patch entropy as an effective routing coordinate for sparse conditional computation and extend Mixture-of-Experts modeling beyond tokenizer-based representations.