Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel 张量并行流水线并行多维混合并行

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsFlashAttention

Causal Masking Optimization

Premium

Implement causal attention for autoregressive models and skip upper-triangular compute for ~2x speedup.

Log in to continue reading

This is premium content. Please log in to access the full article.

Block Pointers and Multi-Dim Support

Scale from single sequence to Batch/Head parallelism and simplify pointer math with block pointers.

Grouped Query Attention

Add GQA/MQA support so multiple query heads share KV, reducing KV cache memory.

Table of Contents

Quick Review: Causal Attention

Performance Opportunity

Half the Compute

Visual: Skipped Blocks

Change 1: Coarse Skip via Loop Bound

Change 2: Fine-Grained Mask Inside Blocks

Why Both Masks?

Performance and Correctness

Speedup vs Sequence Length

Implementation Tips

Compile-Time Causal Flag