Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

Attention Mechanisms

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Hands-on Training

FundamentalsModel Architecture

Rotary Position Embedding

Premium

From position encoding basics to RoPE math, implementation, and length extrapolation

Log in to continue reading

This is premium content. Please log in to access the full article.

Attention Mechanisms

Deeply understand Attention in Transformers, including MHA, Causal Attention, GQA, and MQA.

Position Encoding Basics

Why Transformers need position information, and the methods and limits of absolute position encoding

Table of Contents