LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
FundamentalsModel Architecture

Transformer LM

From token ids to next-token logits — building an overall mental model of the Decoder-only Transformer

Overview

The Transformer LM series answers one concrete question: when we call model(input_ids), what actually happens inside the model?

This series first establishes the overall forward pass of a Decoder-only Transformer, then takes apart the key interfaces — embedding, LM head, decoder block, residual stream, and so on. Attention, RoPE, RMSNorm, and SwiGLU are expanded on in later topics; here we first put them back into the data flow of a complete language model.

Chapter Contents

From token ids to logits

Understand the input/output relationship of a Decoder-only Transformer

Embedding and LM Head

Understand how token ids enter a continuous vector space, and how hidden states are projected back to vocabulary logits

Learning Path

StageContentGoal
OverallFrom token ids to logitsSee clearly the inputs, outputs, shapes, and the training/inference fork point
Input/OutputEmbedding and LM HeadUnderstand how token ids enter the model and how hidden states become logits

References

  • Attention Is All You Need: the original paper on the Transformer architecture, residual connections, and the attention mechanism.
  • Language Models are Unsupervised Multitask Learners: the GPT-2 technical report, showcasing the language-modeling paradigm of Decoder-only LMs.
  • LLaMA: Open and Efficient Foundation Language Models: a reference for the RoPE, RMSNorm, SwiGLU, and pre-norm design of modern Decoder-only LMs.

Architecture (Model Architecture)

From the Transformer LM backbone to Attention, RoPE, and modern components: understand language model architecture

From token ids to logits

Understand how a Decoder-only Transformer turns token ids into next-token logits

Table of Contents

Overview
Chapter Contents
Learning Path
References