Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel 张量并行流水线并行多维混合并行

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsDistributed Training

多维混合并行

Premium

ParallelContext 坐标系统与 TP+DP+PP 的工业级组合

👨‍🍳

Content is cooking...

We're preparing high-quality content for you. Stay tuned!

流水线并行

GPipe 和 1F1B 调度策略的原理与气泡分析

Overview

Introduction to the cookllm-bento training framework

Table of Contents

为什么要混合

ParallelContext：多维坐标系统

通信组的构建

2D 混合示例：TP + DP

3D 混合：TP + DP + PP