Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsDistributed Training

Multi-Dimensional Hybrid Parallelism

Premium

The ParallelContext coordinate system and industrial-grade combination of TP + DP + PP

Get code access

👨‍🍳

Content is cooking...

We're preparing high-quality content for you. Stay tuned!

Pipeline Parallelism

The principles of the GPipe and 1F1B scheduling strategies and bubble analysis

Overview

Introduction to the cookllm-bento training framework

Table of Contents

ParallelContext: A Multi-Dimensional Coordinate System

Building Communication Groups

A 2D Hybrid Example: TP + DP

3D Hybrid: TP + DP + PP

Configuration Recommendations