LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
SystemsDistributed Training

Pipeline Parallelism

Premium

The principles of the GPipe and 1F1B scheduling strategies and bubble analysis

Get code access
👨‍🍳

Content is cooking...

We're preparing high-quality content for you. Stay tuned!

Tensor Parallelism

The symmetric design of Column Parallel and Row Parallel

Multi-Dimensional Hybrid Parallelism

The ParallelContext coordinate system and industrial-grade combination of TP + DP + PP

Table of Contents

Layer-Level Sharding
Naive Pipeline: The Bubble Problem
GPipe: Micro-Batch Parallelism
1F1B: Interleaved Forward and Backward
PP's Communication Characteristics
GPipe vs 1F1B
Summary