Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsDistributed Training

Fully Sharded Data Parallel

Premium

Understanding FSDP's Intra-Tensor sharding and All-Gather/Reduce-Scatter communication patterns

Get code access

👨‍🍳

Content is cooking...

We're preparing high-quality content for you. Stay tuned!

ZeRO Optimizer

Progressive de-redundancy: three-stage sharding from optimizer states to parameters

Tensor Parallelism

The symmetric design of Column Parallel and Row Parallel

Table of Contents

Two Sharding Methods

Parameter Sharding

Forward Pass: All-Gather

Backward Pass: Reduce-Scatter

Communication-Volume Comparison

When to Use ZeRO-3, When to Use FSDP