LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
Fundamentals

Architecture (Model Architecture)

From the Transformer LM backbone to Attention, RoPE, and modern components: understand language model architecture

Overview

The Architecture module starts from a complete decoder-only Transformer LM, then progressively breaks down core components such as Attention, RoPE, Normalization, MLP, and the residual stream.

We recommend starting with the Transformer LM overview to build the overall data flow from token ids to next-token logits, then moving on to topics like Attention and RoPE.

Chapters

Transformer LM

From token ids to next-token logits: build a mental model of the decoder-only Transformer forward pass

Attention Mechanisms

MHA, Causal Attention, GQA / MQA, and the design and evolution of Gated Attention

Position Encoding and RoPE

From sinusoidal PE to rotary position embeddings: understand the math, implementation, and length extrapolation of RoPE

Learning Path

StageContentGoal
OverallTransformer LMMaster the input/output contract, shape changes, and forward pass
CoreAttention mechanismsMaster Self-Attention, Multi-Head, Causal Masking
CorePosition encoding and RoPEUnderstand the evolution of position encoding; master RoPE principles and implementation
OptimizationGQA/MQAUnderstand KV cache optimization and memory efficiency
AdvancedLength extrapolationMaster long-sequence methods like NTK-aware and YaRN

References

  • CS336 Assignment 1
  • Attention Is All You Need
  • RoFormer: Enhanced Transformer with Rotary Position Embedding
  • YaRN: Efficient Context Window Extension of Large Language Models
  • GQA: Training Generalized Multi-Query Transformer
  • Fast Transformer Decoding: One Write-Head is All You Need

BPE Training Engineering

From toy data to real corpora: memory optimization, parallel pre-tokenization, incremental updates, and time-space tradeoffs

Transformer LM

From token ids to next-token logits — building an overall mental model of the Decoder-only Transformer

Table of Contents

Overview
Chapters
Learning Path
References