LogoCookLLM Docs
LogoCookLLM Docs
HomeCookLLM

Principles

Tokenization
Tokenization BasicsBPE AlgorithmGPT TokenizersBPE Training Engineering
Model Architecture
Transformer LM
From token ids to logitsEmbedding and LM Head
Attention Mechanisms
From Self-Attention to GQAAttention Sink
Position Encoding
Position Encoding BasicsRoPE Math DerivationRoPE ImplementationLength Extrapolation
GPU Programming Basics
GPU Architecture BasicsTensor LayoutTriton Basics: Vector Add
FlashAttention
Flash Attention PrinciplesFrom Naive Implementation to Auto-TuningBlock Pointers and Multi-Dim SupportCausal Masking OptimizationGrouped Query AttentionBackward Pass Implementation
Distributed Training
Data ParallelismZeRO OptimizerFully Sharded Data ParallelTensor ParallelismPipeline ParallelismMulti-Dimensional Hybrid Parallelism

Hands-on Training

Overview
Pretraining
Pretraining DataTokenizer TrainingModel ArchitectureData PipelineTraining LoopMonitoring and Validation
X (Twitter)
FundamentalsModel ArchitectureTransformer LM

Embedding and LM Head

Premium

Understand how token ids enter a continuous vector space, and how hidden states are projected back to vocabulary logits

Get code access

Why are the input/output layers worth looking at separately?

The heaviest computation in a Decoder-only Transformer happens in the blocks in the middle, but the input/output relationship of a language model is actually determined by the two ends:

token ids→embeddinghidden states→LM Headnext-token logits\text{token ids} \xrightarrow{\text{embedding}} \text{hidden states} \xrightarrow{\text{LM Head}} \text{next-token logits}token idsembedding​

Log in to continue reading

This is premium content. Please log in to access the full article.

From token ids to logits

Understand how a Decoder-only Transformer turns token ids into next-token logits

Attention Mechanisms

From MHA / Causal / GQA to Attention Sink and Gated Attention, understand the design, flaws, and evolution of attention

Table of Contents

Why are the input/output layers worth looking at separately?
Embedding
Looking up a table by token id
What does the embedding learn after training?
Adding new special tokens: the cleverness of padded slots
What did SFT change in the embedding
Reading SFT priorities from the special tokens
Reading SFT content from the ordinary BPE tokens
The assembled SFT recipe
LM Head
Geometric view: why having the same shape is not a coincidence
In code it is just one line of assignment
Cost: tying is not free
After wrapping a layer with LoRA, does tying still hold?
Summary
hidden statesLM Head​
next-token logits

If these two ends are not understood clearly, many later questions get tangled together: why the input is integers, why the output is not a token id, why the loss needs shifted targets, why the vocabulary size significantly affects the parameter count, and even why some models share the weights of the input embedding and the output LM Head.

The main thread of this chapter only covers the two ends, input and output: Embedding and LM Head.