LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
FundamentalsModel ArchitectureAttention Mechanisms

Attention Sink

Premium

Why the first token absorbs most attention: the mechanism and cost of this phenomenon, and why eliminating it is deferred to Gated Attention

Get code access

What Remains After GQA

In the previous chapter we saw that GQA compresses the KV cache to 1/4 of MHA, at the cost of a slight quality drop on some tasks. But GQA changes the number of KV heads; it does not touch the mathematical structure of attention itself: it is still softmax, still causal mask, still QKT/dQK^T / \sqrt{d}QKT/d​.

If we take a different angle, bypass the KV cache, and look only at the attention weights, we find a phenomenon that has nothing to do with the number of heads or the number of groups: during inference, the vast majority of attention flows to the first token.

This is Attention Sink. In this chapter we trace it from start to finish on Qwen3-0.6B; and mechanistically, as long as attention is still the combination of softmax + causal, it will appear.

Schematic: what an attention sink looks like
key j (attended position) →query t (attending) ↓012345678012345678sinkattention weighthighlow

Each row is a query, each column a key; under the causal mask a query only sees itself and the keys to its left (dashed cells are masked out). Notice column 0: almost every query puts most of its attention weight there, forming a vertical bright stripe across all queries. That stripe is the attention sink.

This chapter answers three questions:

  1. What does this phenomenon look like, and how extreme is it in the data?
  2. Why does it inevitably appear?
  3. What cost does it leave for the system?

Log in to continue reading

This is premium content. Please log in to access the full article.

From Self-Attention to GQA

Starting from Self-Attention, unpack the design trade-offs of Multi-Head, Causal Masking, and GQA / MQA in turn

Rotary Position Embedding

From position encoding basics to RoPE math, implementation, and length extrapolation

Table of Contents

What Remains After GQA
The Phenomenon: What the First Token Absorbs
A Direct Observation
Unpacking One More Layer: Does Sink Come from Magnitude or Angle?
Unpacking More Thoroughly: Where Does Massive Activation Land?
Why Sink Appears
The "Weights Must Be Fully Spent" Constraint of Softmax
The "Globally Visible" Privilege of the First Token
The Cost: The KV Cache Cannot Drop the First Token
Summary