LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
SystemsGPU Programming Basics

GPU Architecture Basics

Premium

Understand GPU design philosophy, the SIMT model, and hardware hierarchy mapping to build parallel intuition.

Get code access

The Core Tension: Latency vs Throughput

Before writing any CUDA code, we need to switch our mental model.

CPU and GPU are built for computation, but they solve very different physical problems. Think Ferrari vs bus:

  • CPU (Latency-oriented): designed for low latency. It has large caches and complex control logic (branch prediction, out-of-order execution). The goal is to finish a serial task as fast as possible.
    • Use cases: OS scheduling, complex business logic.
  • GPU (Throughput-oriented): designed for high throughput. It cuts control logic and caches, using transistors to build compute units (ALUs). The goal is to process massive data in parallel.
    • Use cases: rendering, matrix multiplication, deep learning training.

CPU vs GPU Heterogeneous Architecture

Heterogeneous Computing

A GPU is not a standalone compute platform; it is a coprocessor for the CPU.

When we say “GPU parallel computing,” we really mean a CPU + GPU heterogeneous architecture:

  • Host: CPU + system memory. Handles control, IO, orchestration.
  • Device: GPU + VRAM. Handles massively parallel compute.

Log in to continue reading

This is premium content. Please log in to access the full article.

GPU Programming Basics

Learn CUDA and Triton, and write efficient GPU kernels

Tensor Layout

Understand physical memory layout, strides, view vs reshape, and gradient tracking.

Table of Contents

The Core Tension: Latency vs Throughput
Heterogeneous Computing
Transistor Economics
Task Division
From Graphics to AI: The Compute Evolution
End of Moore’s Law and Parallelism
CUDA: The Key to General Compute
Tensor Cores: Built for AI
SIMT: Single Instruction, Multiple Threads
Goodbye to Loop Thinking
Why Bounds Checks?
Hardware Hierarchy: Grid, Block, Thread
Hierarchy Mapping
Query Hardware Limits
Scaling Up: Global Index Computation
Multi-dimensional Mapping: Toward Matrices
2D Indexing
Why This Matters
Summary
  • Bridge: connected via PCIe.

    • Bottleneck alert: PCIe bandwidth (tens of GB/s) is far below GPU VRAM bandwidth (TB/s). Frequent host-device transfers are the biggest performance killer. First rule of fast kernels: keep data on the GPU.

    Transistor Economics

    Intuition: If you need to move 100 bricks from A to B:

    • CPU is a Ferrari: extremely fast, carries 2 bricks per trip, makes 50 trips.
    • GPU is a swarm of ants (or a slow truck): slower, but carries 100 bricks at once, one trip.

    Task Division

    On the chip, CPU area is mostly Control and Cache; GPU area is mostly ALUs. This drives their division of labor:

    FeatureCPUGPU
    Core countFew (dozens)Many (thousands)
    Best atControl-heavy (branching, logic)Compute-heavy (data-parallel)
    ThreadsHeavyweight (expensive context switches)Lightweight (fast switching to hide latency)

    That means: GPUs are bad at complex control flow, but excellent at repeating the same compute over large data.

    Thus CPU+GPU is complementary:

    • CPU: orchestrates, runs complex serial logic.
    • GPU: runs data-parallel kernels at scale.

    From Graphics to AI: The Compute Evolution

    Why do GPUs dominate AI today? Not just because they have more cores.

    End of Moore’s Law and Parallelism

    As Moore’s Law hits physical limits, single-core performance gains slow down. Hennessy & Patterson argue that future gains will come from domain-specific architectures.

    GPU is the early example: if you cannot make one worker faster (latency limit), hire ten thousand workers (throughput win).

    CUDA: The Key to General Compute

    In 2007, NVIDIA released CUDA, enabling GPU programming in C-like code instead of graphics APIs. This began the GPGPU era.

    Tensor Cores: Built for AI

    Modern NVIDIA GPUs (Volta and later) add Tensor Cores alongside standard CUDA cores.

    • CUDA Core: general scalar ops (float add/mul)
    • Tensor Core: one job, extremely fast—matrix multiply-accumulate (D=A×B+CD = A \times B + CD=A×B+C), typically in mixed precision

    That is why LLM training and inference always emphasize “maxing out Tensor Cores.”