Skip to main content
CookLLM logoCookLLM Docs
Fundamentals

Architecture (Model Architecture)

From the Transformer LM backbone to Attention, RoPE, and modern components: understand language model architecture

Overview

The Architecture module starts from a complete decoder-only Transformer LM, then progressively breaks down core components such as Attention, RoPE, Normalization, MLP, and the residual stream.

We recommend starting with the Transformer LM overview to build the overall data flow from token ids to next-token logits, then moving on to topics like Attention and RoPE.

Chapters

Learning Path

StageContentGoal
OverallTransformer LMMaster the input/output contract, shape changes, and forward pass
CoreAttention mechanismsMaster Self-Attention, Multi-Head, Causal Masking
CorePosition encoding and RoPEUnderstand the evolution of position encoding; master RoPE principles and implementation
OptimizationGQA/MQAUnderstand KV cache optimization and memory efficiency
AdvancedLength extrapolationMaster long-sequence methods like NTK-aware and YaRN

References