Declarative Attention（DA）是一种新的语言模型注意力控制协议，旨在减少模型在处理长文本时的计算成本。该协议通过让模型自主声明需要关注的上下文部分，将生成过程分为三种模式，从而有效减少对整个上下文的全局扫描。
要点：
1. 传统语言模型在处理长文本时，大部分注意力集中在少量上下文上，但仍然需要读取整个KV缓存来找到重要的token。
2. DA协议通过让模型自主声明关注点，将生成过程分为全上下文、特定区域和最近输出三种模式，减少对KV缓存的读取。
3. 在15个长文本任务上的零样本评估中，DA在Gemma-4-31B和Qwen-3.6-27B等现成模型上显著减少了解码过程中的总关注token数量，分别降低了52.0%和31.1%。
4. 尽管准确性有所下降（1.27pp和2.75pp），但随着模型规模的增加，这种下降幅度减小。
5. DA为稀疏注意力开辟了新的方向，未来可以通过基于训练的方法进一步探索其潜力。
Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: (full context), (a specific region), and (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.
原始发布方：HuggingFace Daily Papers（社区热门论文）
原文时间：2026-09-02 08:00:00 +08:00
