Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

FundamentalsModel Architecture Position Encoding

RoPE Implementation

Premium

Inverse frequency computation, cos/sin caching, and a vectorized apply_rotary_pos_emb

Get code access

Log in to continue reading

This is premium content. Please log in to access the full article.

RoPE Math Derivation

From complex rotations to higher-dimensional generalization, understand the core math of rotary position embeddings

Length Extrapolation

NTK-aware Scaling, YaRN, and other methods to let RoPE handle longer sequences

Table of Contents

Inverse Frequency Precomputation

Two Implementation Styles

Interleaved Style (Original Paper)

Pairing and Rotation Matrix

Complex-Multiply Vectorization

Split-Halves Style (HuggingFace Transformers)

Pairing and Rotation Matrix

rotate_half Vectorization

Vectorized apply_rotary_pos_emb

Equivalence of the Two Styles

A Full RoPE Module

Integrating RoPE Into Attention

Working With KV Cache