LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)

Pretraining

Premium

Starting from sample data, complete the full pretraining loop for the 29M BentoLM

Overview

Pretraining is the first main track in cookllm-bento. Here we start from a small 29M-parameter model and walk through all the key stages needed for language model pretraining: preparing text data, training the tokenizer, defining BentoLM, building the data pipeline, entering the Lightning training loop, and writing out checkpoints, TensorBoard/SwanLab logs, and sampled text.

This chapter starts with the data and tokenizer needed before training, then gradually moves into model architecture, the data pipeline, the training loop, and monitoring and validation. By the end, you should know which files make up a pretraining task, what each config file controls, and where to start looking when training speed, loss, sample outputs, or checkpoints go wrong.

This chapter focuses on "how to run it" in cookllm-bento. If you want to deeply understand the principles behind Attention, RoPE, RMSNorm, activation functions, or optimizers, you can read the corresponding chapters in "Fundamentals" alongside this one.

Log in to continue reading

This is premium content. Please log in to access the full article.

Overview

Introduction to the cookllm-bento training framework

Pretraining Data

Understand the Fineweb-Edu-Chinese data, sampled shards, and default directory

Table of Contents

Overview
Chapter Contents