Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

FundamentalsTokenization

BPE Training Engineering

Premium

From toy data to real corpora: memory optimization, parallel pre-tokenization, incremental updates, and time-space tradeoffs

Get code access

Training on Real Data

In chapter 2 we implemented basic BPE; in chapter 3 we learned GPT-style pre-tokenization. Now we combine them to train a tokenizer on real data.

We will:

Build a baseline: combine BPE and GPT-2-style pre-tokenization, with file input
Test on TinyStories: 2GB data, 32K vocab, see if baseline holds up
Analyze bottlenecks: what breaks at scale?
Optimize step-by-step: chunked pre-tokenization, incremental updates, low-frequency pruning, checkpoints

Log in to continue reading

This is premium content. Please log in to access the full article.

GPT Tokenizers

GPT-2/GPT-4 tokenization, regex pre-tokenization, and the tiktoken library

Architecture (Model Architecture)

From the Transformer LM backbone to Attention, RoPE, and modern components: understand language model architecture

Table of Contents

Training on Real Data

1.1 Baseline Implementation

Data Structure Changes

Frequency Weighting

Baseline Training Function

1.2 Baseline Performance Test

1.3 The OOM Problem

Memory Bottleneck

Solution Directions

2.1 Pre-tokenization and Chunk Boundaries

Boundary Choice: Do Not Cut Arbitrarily

Parallel Pre-tokenization

2.2 Low-Frequency Pruning

Pruning Strategy

Impact of Pruning

2.3 Incremental Updates vs Full Recompute

Problem: Updating Counts After Each Merge

Option 1: Full Recompute

Option 2: Incremental Updates

What These Indices Do

Incremental Update Steps

Example Data Changes

2.4 Checkpointing

2.5 Performance Comparison

Key Optimization Impact