Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

Attention Mechanisms

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass

Hands-on Training

从 token ids 到 logits

Premium

理解 Decoder-only Transformer 如何把 token ids 变成 next-token logits

Log in to continue reading

This is premium content. Please log in to access the full article.

Table of Contents

从 token ids 开始

为什么每个位置都有一组 logits？

一次 forward pass 经过哪些形状？

中间为什么一直保持

[B,T,D]

训练和生成在哪里分开？