Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

Attention Mechanisms

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Hands-on Training

全分片数据并行

Premium

理解 FSDP 的 Intra-Tensor 分片与 All-Gather/Reduce-Scatter 通信模式

👨‍🍳

Content is cooking...

We're preparing high-quality content for you. Stay tuned!

Table of Contents

两种分片方式

前向传播：All-Gather

反向传播：Reduce-Scatter

通信量对比

何时用 ZeRO-3，何时用 FSDP