Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

Attention Mechanisms

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Hands-on Training

FundamentalsModel Architecture

Attention Mechanisms

Premium

Deeply understand Attention in Transformers, including MHA, Causal Attention, GQA, and MQA.

Log in to continue reading

This is premium content. Please log in to access the full article.

Architecture (Model Architecture)

Deeply understand LLM architecture design

Rotary Position Embedding

From position encoding basics to RoPE math, implementation, and length extrapolation

Table of Contents

What Is Attention

Standard Self-Attention Flow

PyTorch Reference Implementation

Multi-Head Attention (MHA)

Why Multiple Heads?

PyTorch Implementation

MHA Advantages and Challenges

Causal Attention

What Is Causal Attention?

Mathematical Form

Why Causal Masking?

PyTorch Implementation

Performance Opportunity in Causal Masking

Grouped Query Attention (GQA)

From MHA to GQA

MHA Memory Problem

Multi-Query Attention (MQA)

Grouped Query Attention (GQA)

PyTorch Implementation

Comparing the Three Mechanisms