Attention Sink
PremiumWhy the first token absorbs most attention: the mechanism and cost of this phenomenon, and why eliminating it is deferred to Gated Attention
Get code accessWhat Remains After GQA
In the previous chapter we saw that GQA compresses the KV cache to 1/4 of MHA, at the cost of a slight quality drop on some tasks. But GQA changes the number of KV heads; it does not touch the mathematical structure of attention itself: it is still softmax, still causal mask, still .
If we take a different angle, bypass the KV cache, and look only at the attention weights, we find a phenomenon that has nothing to do with the number of heads or the number of groups: during inference, the vast majority of attention flows to the first token.
This is Attention Sink. In this chapter we trace it from start to finish on Qwen3-0.6B; and mechanistically, as long as attention is still the combination of softmax + causal, it will appear.
Each row is a query, each column a key; under the causal mask a query only sees itself and the keys to its left (dashed cells are masked out). Notice column 0: almost every query puts most of its attention weight there, forming a vertical bright stripe across all queries. That stripe is the attention sink.
This chapter answers three questions:
- What does this phenomenon look like, and how extreme is it in the data?
- Why does it inevitably appear?
- What cost does it leave for the system?
Log in to continue reading
This is premium content. Please log in to access the full article.