Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

Attention Mechanisms

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Hands-on Training

数据并行

Premium

理解通信原语和 DDP 的梯度同步机制

Log in to continue reading

This is premium content. Please log in to access the full article.

Table of Contents

单卡训练的内存组成

混合精度训练的内存需求

DataParallel：最朴素的多卡方案

DDP 的工作原理

Ring All-Reduce：高效的梯度同步

梯度同步机制