LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
FundamentalsTokenization

Tokenization Basics

Premium

Why tokenization? From character-level to subword-level, with Unicode and UTF-8

Why Tokenization

The core of an LLM is the Transformer, which processes sequences of numbers, not text. So the question is: how do we convert text into numbers the model can understand?

Tokenization solves this. It converts text strings into integer sequences (tokens). These integers index into an embedding table to obtain vector representations, which are then fed into the Transformer.

Tokenization Pipeline

Tokenization looks simple, but it is the root of many “weird” LLM behaviors. For example:

  • Why does GPT sometimes struggle with character-by-character spelling?
  • Why is performance worse in some languages than English?
  • Why is the model unusually sensitive to certain strings?

These issues can often be traced back to tokenization design.

Log in to continue reading

This is premium content. Please log in to access the full article.

Tokenization

Deeply understand LLM tokenization, from BPE to GPT implementations

BPE Algorithm

Deep dive into Byte Pair Encoding, with manual training, encoding, and decoding

Table of Contents

Why Tokenization
Start With Character-Level Tokenization
Character-Level Workflow
Limitations of Character-Level Tokenization
Unicode and UTF-8: Multi-language Support
What Is Unicode
UTF-8 Encoding
Why Not Use UTF-8 Bytes Directly?
Summary