Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

Attention Mechanisms

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Hands-on Training

SystemsFlashAttention

Flash Attention Principles

Premium

Use interactive visuals to understand Flash Attention’s core ideas: memory bottlenecks, online softmax, and tiled matmul.

Log in to continue reading

This is premium content. Please log in to access the full article.

Flash Attention

Deeply understand Flash Attention principles and Triton implementation

From Naive to Auto-Tuning

Write your first Flash Attention kernel and optimize it with auto-tune.

Table of Contents

The Memory Bottleneck in Standard Attention

GPU Memory Hierarchy: SRAM vs HBM

The Logical Trap in the Standard Implementation

Bandwidth Gap: SRAM vs HBM

IO-bound Bottleneck

SRAM Capacity Limits

Cost and Density

Capacity Limits

Core Idea: Optimize IO Complexity

Avoid Storing Intermediates

Limits of Offline Softmax

Online Algorithm and Dynamic Correction

Correction Formula

Numeric Example: [3, 2, 5, 1]

Why This Fits Flash Attention

Tiled Matrix Multiplication (Tiling)

Visual Demo: Tiled Compute

What to Observe

Tiling + Attention

Loop Strategy: V1 vs V2

Visualization Guide

Softmax Correction in Tiled Attention

Naive Local Softmax Fails

Solution: Online Rescaling

One Iteration Over a K Block

Final Normalization

Full Pseudocode

Algorithm in Perspective