Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

Attention Mechanisms

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Hands-on Training

Tokenizer 训练

使用 RustBPE 训练 BPE tokenizer，并导出 tiktoken 编码

👨‍🍳

Content is cooking...

We're preparing high-quality content for you. Stay tuned!

Table of Contents

1. 输入和输出

2. 训练命令

3. 训练流程

4. 预分词规则

5. 特殊 token

6. 训练后的检查

7. 词表大小怎么选

8. 和训练配置的关系