Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

Attention Mechanisms

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Hands-on Training

FundamentalsModel Architecture Position Encoding

RoPE Implementation

Premium

Inverse frequency computation, cos/sin caching, and a vectorized apply_rotary_pos_emb

Log in to continue reading

This is premium content. Please log in to access the full article.

RoPE Math Derivation

From complex rotations to higher-dimensional generalization, understand the core math of rotary position embeddings

Length Extrapolation

NTK-aware Scaling, YaRN, and other methods to let RoPE handle longer sequences

Table of Contents

Inverse Frequency Precomputation

Two Implementation Styles

Interleaved Style (Original Paper)

Pairing and Rotation Matrix

Complex-Multiply Vectorization

Split-Halves Style (HuggingFace Transformers)

Pairing and Rotation Matrix

rotate_half Vectorization

Vectorized apply_rotary_pos_emb

Equivalence of the Two Styles

A Full RoPE Module

Integrating RoPE Into Attention

Working With KV Cache