Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsFlashAttention

Block Pointers and Multi-Dim Support

Premium

Scale from single sequence to Batch/Head parallelism and simplify pointer math with block pointers.

Get code access

Log in to continue reading

This is premium content. Please log in to access the full article.

From Naive Implementation to Auto-Tuning

Write your first Flash Attention kernel and use Auto-Tune for performance optimization.

Causal Masking Optimization

Implement causal attention for autoregressive models, achieving ~2x speedup by skipping the upper-triangular computation.

Table of Contents

From Single Sequence to Batch/Head Parallelism

4D Tensor Memory Layout

3D Grid Parallelism

Manual Pointer Offsets

Block Pointers: The Elegant Solution

Pointer Advances in the Loop

Full Comparison