LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
FundamentalsTokenization

BPE Algorithm

Premium

Deep dive into Byte Pair Encoding, with manual training, encoding, and decoding

Get code access

Core Idea of BPE

In the previous chapter we saw the issues with character-level tokenization: sequences are too long and structure is lost. Is there a method that supports all characters (via UTF-8 bytes) and compresses sequence length?

Byte Pair Encoding (BPE) is the answer.

The core idea is simple:

Iteratively merge the most frequent byte pair

Start from the UTF-8 byte sequence. Repeatedly find the most common adjacent byte pair and merge it into a new token. Continue until you reach the target vocabulary size.

This way, common words and phrases become single tokens, while rare combinations remain multiple bytes.

Log in to continue reading

This is premium content. Please log in to access the full article.

Tokenization Basics

Why tokenization? From character-level to subword-level, with Unicode and UTF-8

GPT Tokenizers

GPT-2/GPT-4 tokenization, regex pre-tokenization, and the tiktoken library

Table of Contents

Core Idea of BPE
BPE Algorithm Steps
Round 1: Find Most Frequent Pair
Round 2: Continue Merging
Iteration
Implementing BPE: Core Functions
1. Count Pair Frequencies
2. Merge a Pair
3. Train BPE
Encoding and Decoding
Encoding: Text → Tokens
Decoding: Tokens → Text
Full Example
Advantages of BPE
Summary