Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

Attention Mechanisms

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Hands-on Training

SystemsFlashAttention

From Naive to Auto-Tuning

Premium

Write your first Flash Attention kernel and optimize it with auto-tune.

Log in to continue reading

This is premium content. Please log in to access the full article.

Flash Attention Principles

Use interactive visuals to understand Flash Attention’s core ideas: memory bottlenecks, online softmax, and tiled matmul.

Block Pointers and Multi-Dim Support

Scale from single sequence to Batch/Head parallelism and simplify pointer math with block pointers.

Table of Contents

Core Loop Structure

Why tl.constexpr Is Required

Pointer Arithmetic Intuition

Interaction Guide

Verify Numerical Correctness

Auto-Tuning for Best Config

Using @triton.autotune

Pipeline Parallelism