Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel 张量并行流水线并行多维混合并行

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsDistributed Training

ZeRO Optimizer

Premium

Progressive de-redundancy: three-stage sharding from optimizer states to parameters

Log in to continue reading

This is premium content. Please log in to access the full article.

Data Parallelism

Understanding communication primitives and DDP's gradient synchronization mechanism

Fully Sharded Data Parallel

Understanding FSDP's Intra-Tensor sharding and All-Gather/Reduce-Scatter communication patterns

Table of Contents

Redundancy Analysis of Training State

ZeRO Stage 1: Sharding Optimizer States

Parameter Assignment Strategy

Gradient Synchronization

Training Loop Comparison

ZeRO Stage 2: Sharding Gradients

ZeRO Stage 3: Sharding Parameters

Parameter Sharding

Communication Pattern

Communication Overhead Comparison

ZeRO-3's Sharding Method: Inter-Tensor