Skip to main content

Home Jobs LLM Engineering Course

Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Hands-on Training

Hands-on Training

Training in public, coming soon

We are training a bilingual (Chinese and English) small language model, BentoLM, from scratch and in public. When training is done, this chapter will walk through the whole process: the question, setup, results and cost of every experiment, and the reasoning behind each choice.

Progress so far and every experiment's record are on the Build in Public page. Until then, start with the Principles part.

Multi-Dimensional Hybrid Parallelism

The ParallelContext coordinate system and industrial-grade combination of TP + DP + PP