Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel 张量并行流水线并行多维混合并行

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsFlashAttention

Block Pointers and Multi-Dim Support

Premium

Scale from single sequence to Batch/Head parallelism and simplify pointer math with block pointers.

Log in to continue reading

This is premium content. Please log in to access the full article.

From Naive to Auto-Tuning

Write your first Flash Attention kernel and optimize it with auto-tune.

Causal Masking Optimization

Implement causal attention for autoregressive models and skip upper-triangular compute for ~2x speedup.

Table of Contents

From Single Sequence to Batch/Head Parallelism

4D Tensor Memory Layout

3D Grid Parallelism

Manual Pointer Offsets

Block Pointers: The Elegant Solution

Pointer Advances in the Loop

Full Comparison