LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
FundamentalsModel Architecture

Attention Mechanisms

Premium

From MHA / Causal / GQA to Attention Sink and Gated Attention, understand the design, flaws, and evolution of attention

Overview

Attention is the core of the Transformer. This series starts from the most basic Self-Attention. We first make clear the three forms you must master in practice: MHA, Causal Attention, and GQA. Then we reveal a counterintuitive phenomenon shared by all modern softmax-based LLMs, Attention Sink: why the first token absorbs the vast majority of attention, along with the mechanism, cost, and removal paths of this phenomenon. Finally we move into Gated Attention, an architectural change proposed by the Qwen team in 2025 that uses a single head-wise sigmoid gate to simultaneously address sink, low-rank collapse, and training stability.

After finishing this series, you should be able to answer:

Log in to continue reading

This is premium content. Please log in to access the full article.

  • What problem each of MHA / Causal / GQA solves, and what cost each leaves behind
  • Why softmax + causal inevitably produces attention sink, and what cost this phenomenon brings to the system
  • Why GQA suffers "low-rank collapse" along the Wv→WoW_v \to W_oWv​→Wo​ path, and why this point ultimately becomes the leverage point for Gated Attention
  • What exactly a head-specific sigmoid gate added after the SDPA output is improving

Embedding and LM Head

Understand how token ids enter a continuous vector space, and how hidden states are projected back to vocabulary logits

From Self-Attention to GQA

Starting from Self-Attention, unpack the design trade-offs of Multi-Head, Causal Masking, and GQA / MQA in turn

Table of Contents

Overview
Chapters
References