Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

FundamentalsModel Architecture Attention Mechanisms

Attention Sink

Premium

Why the first token absorbs most attention: the mechanism and cost of this phenomenon, and why eliminating it is deferred to Gated Attention

Get code access

Log in to continue reading

This is premium content. Please log in to access the full article.

From Self-Attention to GQA

Starting from Self-Attention, unpack the design trade-offs of Multi-Head, Causal Masking, and GQA / MQA in turn

Rotary Position Embedding

From position encoding basics to RoPE math, implementation, and length extrapolation

Table of Contents

What Remains After GQA

The Phenomenon: What the First Token Absorbs

A Direct Observation

Unpacking One More Layer: Does Sink Come from Magnitude or Angle?

Unpacking More Thoroughly: Where Does Massive Activation Land?

Why Sink Appears

The "Weights Must Be Fully Spent" Constraint of Softmax

The "Globally Visible" Privilege of the First Token

The Cost: The KV Cache Cannot Drop the First Token