LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data Parallel张量并行流水线并行多维混合并行

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
Systems

GPU Programming Basics

Learn CUDA and Triton, and write efficient GPU kernels

Overview

Before diving into advanced optimizations like Flash Attention, we need the fundamentals of GPU programming. This module takes you from scratch to understanding how GPUs work and how to write efficient kernels with Triton.

This module is a prerequisite for the Systems track. We recommend completing it before Flash Attention.

Chapters

GPU Architecture Basics

Understand SIMT, memory hierarchy, and hardware limits

Tensor Layout

Go deep into memory: stride, contiguous, and view mechanics

Triton Basics: Vector Add

Write your first Triton kernel from scratch

Why Learn This?

What You Want to DoWhat You Need
Understand Flash Attention implementationShared memory, tiling
Write your own attention kernelTriton programming
Optimize inference speedMemory layout, coalescing
Implement custom quantization kernelsCUDA/Triton fundamentals

References

  • CUDA C++ Programming Guide
  • Triton Documentation
  • GPU Architecture Explained

Length Extrapolation

NTK-aware Scaling, YaRN, and other methods to let RoPE handle longer sequences

GPU Architecture Basics

Understand GPU design philosophy, the SIMT model, and hardware hierarchy mapping to build parallel intuition.

Table of Contents

Overview
Chapters
Why Learn This?
References