Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

Attention Mechanisms

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Hands-on Training

FundamentalsTokenization

GPT Tokenizers

Premium

GPT-2/GPT-4 tokenization, regex pre-tokenization, and the tiktoken library

Log in to continue reading

This is premium content. Please log in to access the full article.

BPE Algorithm

Deep dive into Byte Pair Encoding, with manual training, encoding, and decoding

BPE Training Engineering

From toy data to real corpora: memory optimization, parallel pre-tokenization, incremental updates, and time-space tradeoffs

Table of Contents

GPT-2 Tokenization

Problems With Naive BPE

Regex Pre-tokenization Solution

Interactive Demo: BPE Training

GPT-4 Improvements

Vocabulary Size Comparison

Install and Basic Usage

Compare Tokenizers

Inspect Token Byte Representation

Common Special Tokens

Handling Special Tokens

Token Counting and Cost Estimation