LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
FundamentalsModel Architecture

Rotary Position Embedding

Premium

From position encoding basics to RoPE math, implementation, and length extrapolation

Overview

Transformers themselves do not encode position. If you shuffle the input order, the output does not change. Position Encoding is introduced to solve this problem.

This series starts from basic absolute position encoding and dives into the rotary position embedding (RoPE) used in most modern LLMs, including math derivations, implementation details, and how to handle sequences longer than the training context.

This series assumes you are familiar with the fundamentals of Attention. If you are not comfortable with Self-Attention and Multi-Head Attention, we recommend reading Attention Mechanisms first.

Log in to continue reading

This is premium content. Please log in to access the full article.

Attention Sink

Why the first token absorbs most attention: the mechanism and cost of this phenomenon, and why eliminating it is deferred to Gated Attention

Position Encoding Basics

Why Transformers need position information, and the methods and limits of absolute position encoding

Table of Contents

Overview
Chapters
References