Transformer LM
From token ids to next-token logits — building an overall mental model of the Decoder-only Transformer
Overview
The Transformer LM series answers one concrete question: when we call model(input_ids), what actually happens inside the model?
This series first establishes the overall forward pass of a Decoder-only Transformer, then takes apart the key interfaces — embedding, LM head, decoder block, residual stream, and so on. Attention, RoPE, RMSNorm, and SwiGLU are expanded on in later topics; here we first put them back into the data flow of a complete language model.
Chapter Contents
From token ids to logits
Understand the input/output relationship of a Decoder-only Transformer
Embedding and LM Head
Understand how token ids enter a continuous vector space, and how hidden states are projected back to vocabulary logits
Learning Path
| Stage | Content | Goal |
|---|---|---|
| Overall | From token ids to logits | See clearly the inputs, outputs, shapes, and the training/inference fork point |
| Input/Output | Embedding and LM Head | Understand how token ids enter the model and how hidden states become logits |
References
- Attention Is All You Need: the original paper on the Transformer architecture, residual connections, and the attention mechanism.
- Language Models are Unsupervised Multitask Learners: the GPT-2 technical report, showcasing the language-modeling paradigm of Decoder-only LMs.
- LLaMA: Open and Efficient Foundation Language Models: a reference for the RoPE, RMSNorm, SwiGLU, and pre-norm design of modern Decoder-only LMs.