LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
Pretraining

Pretraining Data

Premium

Understand the Fineweb-Edu-Chinese data, sampled shards, and default directory

The goal of language model pretraining is straightforward: give the model a passage of text and let it learn to predict the next token. Data quality, data scale, and how the data is organized all directly affect training efficiency and final performance.

The pretraining example in cookllm-bento uses the OpenCSG Fineweb-Edu-Chinese-V2.1 data. To let teaching experiments run quickly on a single card, the project provides Parquet shards that have already been sampled and split.

This page first answers three questions: where the data comes from, why the project uses 20% sampled shards first, and which directory the training config ultimately reads data from.

Data Source

The pretraining data comes from OpenCSG's Fineweb-Edu-Chinese-V2.1. The corresponding dataset page is opencsg/Fineweb-Edu-Chinese-V2.1. It is a dataset oriented toward Chinese corpus cleaning and educational-quality filtering, making it a good starting point for small-model pretraining.

One easily overlooked point about V2.1 is that it is not a flat dataset, but is divided into different directories by educational quality score.

DirectoryData ScaleMeaning
4_5~70GB, ~46B tokens, 17,790,513 rowsHigh-quality educational text, more clearly and coherently written
3_4~800GB, ~530B tokens, 289,975,835 rowsStill suitable for training, but may have minor coherence or relevance issues
2_3~1.4TB, ~930B tokens, 649,842,063 rowsMay be useful, but noise and limitations are more obvious

The introductory pretraining in cookllm-bento does not directly use the full TB-scale V2.1 data; instead, it first uses the 4_5 high-quality subset. The raw data option in the download script actually matches 4_5/*.parquet, and the later tokenizer sampled data, pretrain sampled shards, and full pretrain shards are all organized around this part of the high-quality data.

"Fineweb-Edu-Chinese-V2.1" here needs to be understood together with the scripts: the dataset as a whole is very large, but cookllm-bento's introductory pretraining defaults to working around the 4_5 high-quality subset. This is done to strike a balance among data quality, download cost, and training cost that is more suitable for teaching.

Log in to continue reading

This is premium content. Please log in to access the full article.

Pretraining

Starting from sample data, complete the full pretraining loop for the 29M BentoLM

Tokenizer Training

Train a BPE tokenizer with RustBPE and export the tiktoken encoding

Table of Contents

Data Source
Data Examples
Paper Background
Re-sampling
Downloading Data
Parquet Shards