BentoLM
Training a bilingual Chinese and English small model from scratch, in the open.
- Experiments
- 14
- Spent so far
- ¥224
Updated 2026-10-26
Progress
- 0Infrastructure
- 1Pre-trainingIn progress
- 2Mid-training
- 3SFT
- 4Preference tuning and RL
- 5Multimodal
- 6Agentic training
- 7Inference and deployment
Pre-training: 4 of 5 steps done
- 1.1Data
zh/en mix of the tokenizer (E001, E003: zh50); zh/en mix of the training data (E002: 5:5); quality-sorted order (E010: not adopted for pre-training, moved to mid-training annealing); Ultra-FineWeb or Nemotron-CC for English (optional)
- 1.2Architecture
E004 to E009: adopt SwiGLU and the attention output gate; keep 4 heads of 128 dims, one KV group per head, global attention, rotation on all dimensions
- 1.3Optimization
E011, E012: adopt Muon (lr 0.02) and WSD, 2% warmup; batch size and learning-rate transfer across sizes folded into 1.4
- 1.4Scaling
E013, E014: fit L(N, D) on 5 sizes (15M to 148M) × 5 training lengths (5 to 80 tokens per parameter), 20 points, error 0.004; predicts zh bpb about 1.07 for 1B on 30B tokens; for the same compute a larger model is better, while the optimal ratio is still an extrapolation, so the training length follows the budget; Muon learning rate on 96M: 0.01 and 0.02 tie, 0.04 is worse, so the main model uses 0.01
- 1.5Final trainingDeferred
a ~1B or larger model, trained at scale once resources allow
The recipe
S model, 1B tokens, Chinese and English validation bpb on 4090s. Lower is better.
- StartLlama-style baseline
- Recipe v1SwiGLU, attention gating, 2% warmupChinese −0.067 · English −0.051
- Recipe v2Muon (learning rate 0.02)Chinese −0.025 · English −0.022
| Chinese | English | |
|---|---|---|
| Start | 1.452 | 1.171 |
| Recipe v1 | 1.386 | 1.120 |
| Recipe v2 | 1.361 | 1.097 |
Each experiment on its own
Experiment log
E014Learning-rate transfer
Finding- Question
- Muon's learning rate of 0.02 was tuned on S (57.3M parameters, E011). Is it still optimal for the larger ml (96.1M)? Is the growing annealing gain in E013 caused by this learning rate being too high for larger models?
- Finding
It carries over.
0.01 and 0.02 are a tie (zh differs by 0.004, en by 0.002); 0.04 is clearly worse (zh +0.021, en +0.016). The optimum lies between 0.01 and 0.02, and 0.01 is slightly better on knowledge answer bpb (by 0.023). The course's main model (about 300M parameters) uses 0.01, since a learning rate that is too high costs much more than one that is too low. The growing annealing gain in E013 is mainly not caused by the learning rate.
Cost¥17W&B runsE013Scaling law
Finding- Question
- Under recipe v2, how does validation loss change with model size N and training tokens D? Can we use it to predict the loss of a 1B model and decide how large the model should be and how many tokens to train on?
- Finding
L(N, D) = E + A/N^α + B/D^β fits well (20 points, error 0.004, below the run-to-run noise of 0.008).
For a 1B model (2048 × 18) trained on 30B tokens it predicts zh bpb about 1.07 and en about 0.86. After adding points at 5 and 80 tokens per parameter, which widens the data range from 4x to 16x, the exponent intervals shrank by nearly half, and for the same compute, a larger model beats more data is now better supported; the fitted optimal ratio (3 to 5 tokens per parameter) is still an extrapolation and is not used directly to set the 1B training length.
Cost¥149W&B runsE012Learning rate schedule
Kept- Question
- Under Muon (E011), which of three schedules is best: WSD, cosine, or constant learning rate plus checkpoint averaging (CMA)?
- Finding
Keep WSD.
Cosine and WSD are a tie (zh difference 0.005); WSD is slightly better, and its training length can be extended after the fact. CMA cannot catch up with learning rate decay: even after averaging the last 10 checkpoints it is still 0.032 worse than WSD.
E011AdamW learning rate and Muon
Adopted- Question
- Under recipe v1, is the AdamW learning rate of 3e-3 a good choice? How much better is Muon (Muon for the 2D matrices inside Transformer blocks, AdamW for all other parameters)? What learning rate should Muon use?
- Finding
Adopt Muon, learning rate 0.02.
Muon is 0.025 lower in zh bpb and 0.022 lower in en bpb than the best AdamW, at the same speed. Muon learning rates from 0.01 to 0.02 make almost no difference; 0.04 is clearly worse. AdamW's 3e-3 is itself close to optimal.
E010Quality-sorted data order
Rejected- Question
- For the same set of documents, does sorting them by Ultra-FineWeb quality score from low to high, so that the high-quality documents land in the learning rate decay phase (following Puro-2B), beat random order?
- Finding
Not adopted for pretraining.
After sorting, validation set bpb got worse (zh +0.017, en +0.013); only the answer bpb on knowledge tasks improved. Checkpoint averaging over the learning rate decay phase did not help either. The constant learning rate plus checkpoint averaging (CMA) pair is postponed to week 2, to be run together with the learning rate schedule experiments.
E009Attention output gate
Adopted- Question
- What happens if we multiply the attention output by an input-dependent sigmoid gate (one extra dim × dim projection per layer)?
- Finding
Adopt; the biggest improvement of week 1.
The zh bpb drops by 0.047 and the en bpb by 0.033, and answer bpb is better in every task group, far beyond run-to-run noise. The cost is 3.1M more parameters (+5.8%) and 5.5% slower training.
E008Partial RoPE
Kept- Question
- What happens if each head applies rotary position embedding to only part of its dimensions, and the remaining dimensions match on content only?
- Finding
Keep rotating all dimensions for now.
Rotating only a quarter (rope25) is clearly worse; rotating half (rope50) is a tie with the baseline. Partial RoPE may help long-context extrapolation, so test it again in the mid-training stage (context extension).
E007Local attention
Rejected- Question
- If 3 out of every 4 layers attend only to the most recent 256 tokens (sliding window) and the last layer attends globally, can we save compute without losing quality?
- Finding
Not adopted.
Quality is a tie, but training is actually 14% slower. The window-128 run was not done: window 256 already shows no benefit, and a smaller window can only be worse.
E006SwiGLU vs ReLU²
Adopted- Question
- What happens if the MLP switches from V1's ReLU² (hidden size 2048) to SwiGLU (hidden size 1344, about the same parameter count)?
- Finding
Adopt SwiGLU.
One run on each of the two GPU types: zh and en bpb both drop by about 0.01, and answer bpb is better in both. Each single gap is slightly below the 0.015 threshold, but the two independent runs agree in direction, and both are below all four typical baseline runs. Only 2% slower.
E005Attention head size and KV sharing
Kept- Question
- The S model (dim 512) uses 4 heads of 128 dims. How do quality and speed change if we switch to 8 heads of 64 dims, or let several query heads share KV (GQA, MQA)?
- Finding
Keep 4 heads of 128 dims, with one KV group per query head.
GQA (2 KV groups) is clearly worse; 64-dim heads are slightly worse and not faster. MQA was not run: GQA is already worse, and sharing a single KV group can only be worse.
E004Run-to-run noise for a fixed config
Finding- Question
- If the same config is trained repeatedly, how much does validation set bpb vary? Below what difference should an ablation result not be trusted?
- Finding
For the same config, zh bpb ranges from 1.433 to 1.455.
Four of five runs fall within 1.449 to 1.455, and one is clearly better. A single-run comparison with a gap within 0.015 cannot support a conclusion. Runs with the same seed diverge early in training, regardless of GPU model.
Cost¥6W&B runsE003Choosing the bilingual tokenizer
Adopted- Question
- The three tokenizers trained in E001 (training text 25% / 50% / 75% Chinese) differ in compression by less than 3%. Which one is better when actually training a model?
- Finding
zh50 is the most balanced and becomes the V2 tokenizer.
With less Chinese vocabulary (zh25), zh gets clearly worse and en does not get better; with more Chinese vocabulary (zh75), zh is only slightly better (0.008) while en gets worse by more (0.016).
Cost¥0.5W&B runsE002zh/en ratio of the pretraining data
Adopted- Question
- For bilingual pretraining of the S model (55M), what share of the tokens should be zh and what share en?
- Finding
5:5 is the most balanced and becomes the V2 default ratio.
Moving away from 5:5 in either direction gains less than it costs: raising zh from 50% to 70% lowers zh bpb by only 0.013 but raises en bpb by 0.047; lowering zh from 50% to 30% lowers en bpb by only 0.021 but raises zh bpb by 0.067.
Cost¥12W&B runsE001zh/en ratio for the bilingual tokenizer
Finding- Question
- For a bilingual BPE tokenizer with a 32768 vocabulary, what share of the training text should be Chinese?
- Finding
zh compression saturates quickly (raising the Chinese share from 25% to 75% improves zh by only 5%), while en keeps benefiting (it drops 12% over the same range).
The overall compression difference between the three ratios is within 3%. zh50 is the tentative default tokenizer (decent on both zh and en); the final choice will be validated with S-model bpb.
Cost¥0.4W&B runs