Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsFlashAttention

Grouped Query Attention

Premium

Add GQA/MQA support so multiple query heads share KV, reducing KV cache memory.

Get code access

Log in to continue reading

This is premium content. Please log in to access the full article.

Causal Masking Optimization

Implement causal attention for autoregressive models, achieving ~2x speedup by skipping the upper-triangular computation.

Backward Pass Implementation

Implement Flash Attention gradient computation, achieving memory-efficient training through recomputation.

Table of Contents

Quick GQA Recap

Problem with Standard PyTorch GQA

Efficient Flash Attention Implementation

Core Idea: Pointer Indexing, Not Data Copy

Concrete Example

Unified Support: MHA/GQA/MQA

Full Implementation

Performance Validation

Autotune Best Config

Tradeoffs and Recommendations

Quality vs Memory

Key Implementation Summary