Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

Overview

Introduction to the cookllm-bento training framework

👨‍🍳

Content is cooking...

We're preparing high-quality content for you. Stay tuned!

Multi-Dimensional Hybrid Parallelism

The ParallelContext coordinate system and industrial-grade combination of TP + DP + PP

Pretraining

Starting from sample data, complete the full pretraining loop for the 29M BentoLM

Table of Contents

Runtime Environment

Choosing Cloud Compute

Staying Up to Date

Chapter Navigation