Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsDistributed Training

Tensor Parallelism

Premium

The symmetric design of Column Parallel and Row Parallel

Get code access

👨‍🍳

Content is cooking...

We're preparing high-quality content for you. Stay tuned!

Fully Sharded Data Parallel

Understanding FSDP's Intra-Tensor sharding and All-Gather/Reduce-Scatter communication patterns

Pipeline Parallelism

The principles of the GPipe and 1F1B scheduling strategies and bubble analysis

Table of Contents

Why We Need Tensor Parallelism

Column Parallel Linear

Row Parallel Linear

Combining Column + Row