LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
Pretraining

Model Architecture

Premium

Read BentoLM's structure and parameter size from bento_29m.yaml

BentoLM-29M is not built to be large, but to let a single card complete the full training pipeline. The benefit of a small model is fast feedback: a wrong config, an incorrect data path, abnormal loss, or a failed checkpoint save can all be exposed in a relatively short time.

The 29M figure is not arbitrary either. Parameter count comes from three places: the embedding (vocab_size × hidden_size), the attention and FFN weights in each layer, and the output head. With vocab_size at just 8192 and hidden_size at 512, the embedding accounts for only about 4M, which leaves the rest of the budget for 8 Transformer layers. That ratio matters: when the embedding dominates, the model spends its capacity memorizing a vocabulary instead of learning structure, which is the most common way to waste a small model.

Every field in the config below corresponds to one of these trade-offs.

Key Configuration

Log in to continue reading

This is premium content. Please log in to access the full article.

configs/model/bento_29m.yaml
model:
  model_config:
    vocab_size: 8192
    hidden_size: 512
    num_hidden_layers: 8
    max_position_embeddings: 512
    rms_norm_eps: 1e-5
    rope_theta: 10000.0
    tie_word_embeddings: true
    dropout: 0.0
  learning_rate: 3e-4

Note that head_dim is not written in the YAML; the code defaults to 128. Therefore:

Tokenizer Training

Train a BPE tokenizer with RustBPE and export the tiktoken encoding

Data Pipeline

Understand how Parquet shards become input_ids, labels, and attention_mask

Table of Contents

Key Configuration
Model Forward Flow
Modern Tricks Used in BentoLM
Why Start With 29M
weight_decay
:
0.1
betas: [0.8, 0.95]
warmup_ratio: 0.0
warmdown_ratio: 0.5