Principles

Tokenization Basics BPE Algorithm GPT Tokenizers BPE Training Engineering

Model Architecture

From token ids to logits Embedding and LM Head

Attention Mechanisms

From Self-Attention to GQA Attention Sink

Position Encoding

Position Encoding Basics RoPE Math Derivation RoPE Implementation Length Extrapolation

GPU Programming Basics

GPU Architecture Basics Tensor Layout Triton Basics: Vector Add

Flash Attention Principles From Naive Implementation to Auto-Tuning Block Pointers and Multi-Dim Support Causal Masking Optimization Grouped Query Attention Backward Pass Implementation

Distributed Training

Data Parallelism ZeRO Optimizer Fully Sharded Data Parallel Tensor Parallelism Pipeline Parallelism Multi-Dimensional Hybrid Parallelism

Hands-on Training

Pretraining Data Tokenizer Training Model Architecture Data Pipeline Training Loop Monitoring and Validation

SystemsGPU Programming Basics

Tensor Layout

Premium

Understand physical memory layout, strides, view vs reshape, and gradient tracking.

What Is a Tensor?

If you have seen this error, it is confusing at first:

Log in to continue reading

This is premium content. Please log in to access the full article.

RuntimeError: view size is not compatible with input tensor's size and stride

Or you may have wondered why PyTorch code often uses detach().clone() instead of just one of them.

These issues mix three different concepts: memory layout, view operations, and gradient tracking. The starting point is:

A PyTorch tensor = a flat memory buffer + metadata that interprets it.

The metadata includes:

Shape: dimensions and sizes
Stride: how many memory steps to move along each dimension
Storage offset: where the tensor starts in the underlying storage

Let’s verify:

import torch

x = torch.tensor([[1, 2, 3],
                  [4, 5, 6]])

print(x)
print(

Although we see a 2×3 matrix, PyTorch stores data as [1, 2, 3, 4, 5, 6] and interprets it with shape=(2,3) and stride=(3,1).

GPU Architecture Basics

Understand GPU design philosophy, the SIMT model, and hardware hierarchy mapping to build parallel intuition.

Triton Basics: Vector Add

Learn Triton’s programming model through a simple vector add example.

Table of Contents

What Is a Tensor?

Key Concept: Strides

Vector Example (1D)

Matrix Example (2D)

Contiguity Explained

What Breaks Contiguity?

Why Non-contiguous?

View vs Reshape: A Performance Pivot

view(): Zero-copy, But Restricted

reshape(): Smarter, Safer

Gradient Tracking: Clone, Detach, and Their Combination

clone(): Copy Data, Keep Grad History

detach(): Cut Grad, Share Memory

detach().clone(): Common Pattern

Layout Types: Row-major vs Column-major

Test Your Understanding

f

"Shape:

{

x.shape

}

"

)

# torch.Size([2, 3])

print(f"Stride: {x.stride()}") # (3, 1)