Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel 张量并行流水线并行多维混合并行

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsDistributed Training

流水线并行

Premium

GPipe 和 1F1B 调度策略的原理与气泡分析

👨‍🍳

Content is cooking...

We're preparing high-quality content for you. Stay tuned!

张量并行

Column Parallel 和 Row Parallel 的对称设计

多维混合并行

ParallelContext 坐标系统与 TP+DP+PP 的工业级组合

Table of Contents

朴素流水线：气泡问题

GPipe：微批次并行

1F1B：交错前向反向

PP 的通信特点