LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
FundamentalsModel ArchitecturePosition Encoding

RoPE Math Derivation

Premium

From complex rotations to higher-dimensional generalization, understand the core math of rotary position embeddings

Get code access

Start From the Requirement

In the previous chapter we stated a core requirement: we want the dot product of QmQ_mQm​ and KnK_nKn​ to depend only on content and relative position m−nm - nm−n:

⟨f(Xm,m),f(Xn,n)⟩=g(Xm,Xn,m−n)\langle f(X_m, m), f(X_n, n) \rangle = g(X_m, X_n, m - n)⟨f(Xm​,m),f(Xn​,n)⟩=g(Xm​,Xn​,m−n)

Here fff is the position-encoding function we design; it takes a token content vector and position index and outputs a position-aware vector.

The key insight of RoPE is: encode position by rotation. We start with the simplest 2D case.

Log in to continue reading

This is premium content. Please log in to access the full article.

Position Encoding Basics

Why Transformers need position information, and the methods and limits of absolute position encoding

RoPE Implementation

Inverse frequency computation, cos/sin caching, and a vectorized apply_rotary_pos_emb

Table of Contents

Start From the Requirement
2D Case: Complex View
Treat a Vector as a Complex Number
Encode Position With Rotation
Expand to Real Form
Generalize to Higher Dimensions
Grouped Rotations
Frequency Design
Efficient Implementation of Rotation
Avoid Explicit Matrix Multiplication
Verify the Relative Position Property
Comparison With Other Position Encodings
Summary