LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Attention Mechanisms
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass

Hands-on Training

X (Twitter)
SystemsFlashAttention

Block Pointers and Multi-Dim Support

Premium

Scale from single sequence to Batch/Head parallelism and simplify pointer math with block pointers.

Companion Code

Log in to continue reading

This is premium content. Please log in to access the full article.

From Naive to Auto-Tuning

Write your first Flash Attention kernel and optimize it with auto-tune.

Causal Masking Optimization

Implement causal attention for autoregressive models and skip upper-triangular compute for ~2x speedup.

Table of Contents

From Single Sequence to Batch/Head Parallelism
4D Tensor Memory Layout
3D Grid Parallelism
Manual Pointer Offsets
Block Pointers: The Elegant Solution
Core API
Pointer Advances in the Loop
Full Comparison
Summary