Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsFlashAttention

Causal Masking Optimization

Premium

Implement causal attention for autoregressive models, achieving ~2x speedup by skipping the upper-triangular computation.

Get code access

Log in to continue reading

This is premium content. Please log in to access the full article.

Block Pointers and Multi-Dim Support

Scale from single sequence to Batch/Head parallelism and simplify pointer math with block pointers.

Grouped Query Attention

Add GQA/MQA support so multiple query heads share KV, reducing KV cache memory.

Table of Contents

Quick Review of Causal Attention

The Performance Opportunity of Causal Masking

The Mathematical Principle of Halving the Compute

Visualization: The Skipped Computation Region

Code Implementation Walkthrough

Change 1: Coarse-Grained Skip — Loop Bound Optimization

Change 2: Fine-Grained Mask — Boundary Handling Inside a Block

The Necessity of Two Layers of Masking

Performance Comparison and Validation

Numerical Correctness Validation

Performance Comparison

Speedup Across Different Sequence Lengths

Summary of Implementation Techniques

The Clever Use of Triton Compile-Time Constants