LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
FundamentalsTokenization

GPT Tokenizers

Premium

GPT-2/GPT-4 tokenization, regex pre-tokenization, and the tiktoken library

Get code access

GPT-2 Tokenization

In the previous chapter we built a basic BPE. OpenAI did not use naive BPE directly in GPT-2; they made an important improvement: regex pre-tokenization.

Problems With Naive BPE

Consider a word like “dog” that appears with different punctuation:

dog.
dog!
dog?
dog,

Naive BPE may merge each into separate tokens:

Log in to continue reading

This is premium content. Please log in to access the full article.

token_123 = "dog."
token_124 = "dog!"
token_125 = "dog?"
token_126 = "dog,"

This causes two problems:

Vocabulary waste: the same word with different punctuation consumes multiple token slots, wasting limited vocab capacity.

Semantic entanglement: word meaning and punctuation meaning are mixed, forcing the model to learn their relationship.

Regex Pre-tokenization Solution

GPT-2’s approach: split text into chunks with regex before BPE.

Core idea:

Prevent cross-category merges: disallow BPE merges across boundaries such as:

  • letters and digits
  • letters and punctuation
  • spaces and non-space characters

GPT-2 regex (simplified):

basics/architecture/tokenizer/02_tiktoken_usage.py
import regex as re

# GPT-2 split pattern
gpt2_pattern = r"""'s|'t|'re|'ve|'m|'ll|'d

Intuitive explanation:

  • | means “or”, ? means “optional once”, + means “one or more”
  • \p{L} is Unicode letters, \p{N} is Unicode digits, \s is whitespace
  • [^...] means “not in this set”
  • The leading ? is an optional space; GPT-2 folds leading spaces into the token

Breakdown:

  • 's|'t|'re|...: match common contractions so they do not split into ' and s
  • ?\p{L}+: optional space + letters (a word)
  • ?\p{N}+: optional space + digits (a number chunk)
  • ?[^\s\p{L}\p{N}]+: optional space + punctuation/symbols
  • \s+(?!\S)|\s+: whitespace (including end-of-line)

Workflow

With regex pre-tokenization, BPE becomes:

  1. Split: use regex to split text into chunks
  2. Encode each chunk: apply BPE to each chunk independently
  3. Concatenate: join all chunk tokens

BPE Algorithm

Deep dive into Byte Pair Encoding, with manual training, encoding, and decoding

BPE Training Engineering

From toy data to real corpora: memory optimization, parallel pre-tokenization, incremental updates, and time-space tradeoffs

Table of Contents

GPT-2 Tokenization
Problems With Naive BPE
Regex Pre-tokenization Solution
Workflow
Interactive Demo: BPE Training
GPT-4 Improvements
Vocabulary Size Comparison
Using tiktoken
Install and Basic Usage
Compare Tokenizers
Inspect Token Byte Representation
Special Tokens
Common Special Tokens
Handling Special Tokens
Token Counting and Cost Estimation
Summary
|
?
\p
{L}
+|
?
\p
{N}
+|
?
[
^
\s
\p
{L}
\p
{N}]
+|
\s
+(?!
\S
)|
\s
+
"""
text = "Hello, world! It's 2024."
chunks = re.findall(gpt2_pattern, text)
print(chunks)
# Output: ['Hello', ',', ' world', '!', ' It', "'s", ' 2024', '.']