Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

Attention Mechanisms

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Hands-on Training

SystemsFlashAttention

Causal Masking Optimization

Premium

Implement causal attention for autoregressive models and skip upper-triangular compute for ~2x speedup.

Log in to continue reading

This is premium content. Please log in to access the full article.

Block Pointers and Multi-Dim Support

Scale from single sequence to Batch/Head parallelism and simplify pointer math with block pointers.

Grouped Query Attention

Add GQA/MQA support so multiple query heads share KV, reducing KV cache memory.

Table of Contents

Quick Review: Causal Attention

Performance Opportunity

Half the Compute

Visual: Skipped Blocks

Change 1: Coarse Skip via Loop Bound

Change 2: Fine-Grained Mask Inside Blocks

Why Both Masks?

Performance and Correctness

Speedup vs Sequence Length

Implementation Tips

Compile-Time Causal Flag