LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
Pretraining

Tokenizer Training

Premium

Train a BPE tokenizer with RustBPE and export the tiktoken encoding

You must determine the tokenizer before pretraining. The model does not see raw strings, but sequences of token ids; the vocabulary size, special tokens, and pre-tokenization rules all affect the subsequent model configuration and the shape of the training data.

cookllm-bento already ships with a usable tokenizer by default:

tokens/
tokens/
├── tokenizer.pkl
├── token_bytes.pt
└── metadata.json

If you just want to run pretraining directly, you can use the default tokenizer first. Retraining the tokenizer is appropriate when you change the corpus, adjust the vocabulary size, or want to fully reproduce the training flow.

Log in to continue reading

This is premium content. Please log in to access the full article.

Training Data

Tokenizer training uses separate sampled data, not the pretraining shards directly. The corresponding option in the download script is:

OptionDataSizeDirectory
[2]Tokenizer 1.5% sampled data~1.2GBtokenizer_data

The 1.5% here refers to extracting a small portion of text from the 4_5 high-quality subset of Fineweb-Edu-Chinese-V2.1, specifically for training the tokenizer. It comes from the same high-quality data source as the 20% sampled shards used for pretraining earlier, but its purpose is different:

  • Tokenizer training only needs to learn how text is split into tokens; it does not need to cover the full pretraining scale.
  • Preparing tokenizer data separately makes BPE training faster, and avoids scanning the 14GB or 70GB pretraining shards every time you train the tokenizer.
  • This data is written as a single Parquet file, rather than multiple training shards.

If you want to regenerate this tokenizer data from the raw 4_5 data, you can use the same sampling script, just changing the sampling ratio to 0.015 and the output to a single file:

tools/sample_pretrain.py
python tools/sample_pretrain.py \
  --data ~/.cache/cookllm-bento/raw_data/fineweb_edu \
  --output ~/.cache/cookllm-bento/tokenizer_data/tokenizer_data.parquet \
  --ratio 0.015

Download command:

Download data
bash tasks/scripts/download/download_data.sh

After selecting [2], the default file location is:

Pretraining Data

Understand the Fineweb-Edu-Chinese data, sampled shards, and default directory

Model Architecture

Read BentoLM's structure and parameter size from bento_29m.yaml

Table of Contents

Training Data
Training Command
Output Files
Special Tokens