Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel 张量并行流水线并行多维混合并行

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsDistributed Training

Fully Sharded Data Parallel

Premium

Understanding FSDP's Intra-Tensor sharding and All-Gather/Reduce-Scatter communication patterns

👨‍🍳

Content is cooking...

We're preparing high-quality content for you. Stay tuned!

ZeRO Optimizer

Progressive de-redundancy: three-stage sharding from optimizer states to parameters

张量并行

Column Parallel 和 Row Parallel 的对称设计

Table of Contents

Two Sharding Methods

Parameter Sharding

Forward Pass: All-Gather

Backward Pass: Reduce-Scatter

Communication-Volume Comparison

When to Use ZeRO-3, When to Use FSDP