Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

Attention Mechanisms

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Hands-on Training

ZeRO 优化器

Premium

渐进式去冗余，从优化器状态到参数的三级分片

Log in to continue reading

This is premium content. Please log in to access the full article.

Table of Contents

训练状态的冗余分析

ZeRO Stage 1：分片优化器状态

参数分配策略

训练循环对比

ZeRO Stage 2：分片梯度

ZeRO Stage 3：分片参数

通信开销对比

ZeRO-3 的分片方式：Inter-Tensor