Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

FundamentalsModel Architecture Position Encoding

RoPE Math Derivation

Premium

From complex rotations to higher-dimensional generalization, understand the core math of rotary position embeddings

Get code access

Start From the Requirement

Log in to continue reading

This is premium content. Please log in to access the full article.

In the previous chapter we stated a core requirement: we want the dot product of $Q_m$ and $K_n$ to depend only on content and relative position $m - n$ :

\langle f(X_m, m), f(X_n, n) \rangle = g(X_m, X_n, m - n)

Here $f$ is the position-encoding function we design; it takes a token content vector and position index and outputs a position-aware vector.

The key insight of RoPE is: encode position by rotation. We start with the simplest 2D case.

Position Encoding Basics

Why Transformers need position information, and the methods and limits of absolute position encoding

RoPE Implementation

Inverse frequency computation, cos/sin caching, and a vectorized apply_rotary_pos_emb

Table of Contents

Start From the Requirement

2D Case: Complex View

Treat a Vector as a Complex Number

Encode Position With Rotation

Expand to Real Form

Generalize to Higher Dimensions

Grouped Rotations

Frequency Design

Efficient Implementation of Rotation

Avoid Explicit Matrix Multiplication

Verify the Relative Position Property

Comparison With Other Position Encodings

g

(

Xm

,

Xn

,

m

−

n)