Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel 张量并行流水线并行多维混合并行

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsDistributed Training

Data Parallelism

Premium

Understanding communication primitives and DDP's gradient synchronization mechanism

Log in to continue reading

This is premium content. Please log in to access the full article.

Distributed Training

From data parallelism to multi-dimensional hybrid parallelism — understanding the core parallel strategies of large model training

ZeRO Optimizer

Progressive de-redundancy: three-stage sharding from optimizer states to parameters

Table of Contents

Memory Composition of Single-GPU Training

Memory Requirements of Mixed-Precision Training

Communication Primitives

DataParallel: The Most Naive Multi-GPU Approach

Ring All-Reduce: Efficient Gradient Synchronization

Gradient Synchronization Mechanism

Limitations of DDP