Tokenizer Training

You must determine the tokenizer before pretraining. The model does not see raw strings, but sequences of token ids; the vocabulary size, special tokens, and pre-tokenization rules all affect the subsequent model configuration and the shape of the training data.

cookllm-bento already ships with a usable tokenizer by default:

tokens/
├── tokenizer.pkl
├── token_bytes.pt
└── metadata.json

If you just want to run pretraining directly, you can use the default tokenizer first. Retraining the tokenizer is appropriate when you change the corpus, adjust the vocabulary size, or want to fully reproduce the training flow.

Training Data

Tokenizer training uses separate sampled data, not the pretraining shards directly. The corresponding option in the download script is:

The 1.5% here refers to extracting a small portion of text from the 4_5 high-quality subset of Fineweb-Edu-Chinese-V2.1, specifically for training the tokenizer. It comes from the same high-quality data source as the 20% sampled shards used for pretraining earlier, but its purpose is different:

• Tokenizer training only needs to learn how text is split into tokens; it does not need to cover the full pretraining scale.
• Preparing tokenizer data separately makes BPE training faster, and avoids scanning the 14GB or 70GB pretraining shards every time you train the tokenizer.
• This data is written as a single Parquet file, rather than multiple training shards.

If you want to regenerate this tokenizer data from the raw 4_5 data, you can use the same sampling script, just changing the sampling ratio to 0.015 and the output to a single file:

python tools/sample_pretrain.py \
  --data ~/.cache/cookllm-bento/raw_data/fineweb_edu \
  --output ~/.cache/cookllm-bento/tokenizer_data/tokenizer_data.parquet \
  --ratio 0.015

Download command:

bash tasks/scripts/download/download_data.sh

After selecting [2], the default file location is: