Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

FundamentalsModel Architecture Position Encoding

Position Encoding Basics

Premium

Why Transformers need position information, and the methods and limits of absolute position encoding

Get code access

Permutation Invariance in Transformers

Let’s start with a key fact: standard Self-Attention does not care about input order.

Recall the Attention formula:

Log in to continue reading

This is premium content. Please log in to access the full article.

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V

Here $Q = XW_Q$ , $K = XW_K$ , $V = XW_V$ . If we shuffle the tokens in , the , , and vectors for each token do not change, only their positions in the sequence do. Attention computes pairwise dot products between all tokens, which is .

In other words, “I like cats” and “Cats like me” look the same to Attention. That is clearly unacceptable because language meaning depends on word order.

That is why we need position encoding: inject position information into the input so the model knows where each token is in the sequence.

Rotary Position Embedding

From position encoding basics to RoPE math, implementation, and length extrapolation

RoPE Math Derivation

From complex rotations to higher-dimensional generalization, understand the core math of rotary position embeddings

Table of Contents

Permutation Invariance in Transformers

Absolute Position Encoding

Sinusoidal Position Encoding

Learned Position Encoding

How Absolute Position Encoding Is Used

Hidden Properties of Sinusoidal PE

Dot Product Depends Only on Relative Position

Long-Range Decay

Limits of Absolute Position Encoding

QKT

)

V

V

=

XWV

X

Q

K

V

order-invariant