GPT Tokenizers

GPT-2 Tokenization

In the previous chapter we built a basic BPE. OpenAI did not use naive BPE directly in GPT-2; they made an important improvement: regex pre-tokenization.

Problems With Naive BPE

Consider a word like “dog” that appears with different punctuation:

dog.
dog!
dog?
dog,

Naive BPE may merge each into separate tokens:

token_123 = "dog."
token_124 = "dog!"
token_125 = "dog?"
token_126 = "dog,"

This causes two problems:

Regex Pre-tokenization Solution

GPT-2’s approach: split text into chunks with regex before BPE.

Core idea:

GPT-2 regex (simplified):

import regex as re

# GPT-2 split pattern
gpt2_pattern = r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"""

text = "Hello, world! It's 2024."
chunks = re.findall(gpt2_pattern, text)
print(chunks)
# Output: ['Hello', ',', ' world', '!', ' It', "'s", ' 2024', '.']

Intuitive explanation:

• | means “or”, ? means “optional once”, + means “one or more”
• \p{L} is Unicode letters, \p{N} is Unicode digits, \s is whitespace
• [^...] means “not in this set”
• The leading ? is an optional space; GPT-2 folds leading spaces into the token

Breakdown:

• 's|'t|'re|...: match common contractions so they do not split into ' and s
• ?\p{L}+: optional space + letters (a word)
• ?\p{N}+: optional space + digits (a number chunk)
• ?[^\s\p{L}\p{N}]+: optional space + punctuation/symbols
• \s+(?!\S)|\s+: whitespace (including end-of-line)

Workflow

With regex pre-tokenization, BPE becomes:

1. Split: use regex to split text into chunks
2. Encode each chunk: apply BPE to each chunk independently
3. Concatenate: join all chunk tokens