Transformer primer

What a primer is for

You do not need to derive attention. You need the vocabulary to read a kernel and a model config without translating in your head: block, head, head dimension, sequence length, KV cache, pre-norm, grouped-query attention, int8 weights.

The block, with shapes

A decoder block is layer norm, multi-head attention, residual, layer norm, feed-forward with a four-times expansion, residual. With batch B, sequence S, width d and h heads, the attention tensors are [B, h, S, d/h]. Attention is softmax(QK^T / sqrt(d/h)) V, with a causal mask during training and prefill. Those four lines are most of what an interview asks you to reproduce.

Decode is a different regime

At decode you process one token against a cache of past keys and values. The cache is 2 * layers * S * d * bytes per sequence, and it is read at every step, so decode moves bytes in proportion to parameters and cache rather than doing arithmetic per byte. That is why decode is bandwidth-bound, why quantising weights helps it, and why the roofline node applies here with different constants.

Quantisation vocabulary

Which tensor is quantised, per-tensor or per-channel, symmetric or not, and what stays in higher precision. Weights and the cache tolerate int8 or fp8; softmax and layer norm usually do not.

What to read

Read the annotated implementation for the block, with the paper beside it for the shapes. Read the survey of variants for the cache and the efficiency tricks. Skip positional-encoding derivations on a first pass; they matter for a long-context claim, not for reading a kernel.

Afterwards, take a model config and compute cache bytes and decode flops per token, then say whether decode is bandwidth- or compute-bound on your device.

Things that break

Memorising the block diagram without the shapes. Every performance claim about attention is a claim about S, d/h and batch, and a diagram without dimensions cannot tell you why prefill and decode behave differently. The tell is being unable to say what the cache costs for a given config.