Attention benchmark

Three builds, one table

Build the materialised form, the QK-plus-softmax fusion, and the full fused block. Time all three on the same shapes, then attribute the difference to the boundary each build removes. What you cannot do is read stage times off the fused kernel: there are no stages left in it to time.

Ablate instead of instrument

To find what dominates, remove one thing at a time and check that what remains is still a legal program. Replace the softmax with a copy, drop the PV contraction, use an all-zero mask. Each ablation must consume its output, or dead code elimination deletes the work and the time collapses to a launch.

That is also the answer to the question this node exists for: after fusion, the dominant stage is the one whose ablation saves the most time, not the one a profiler labelled.

The sweep

Sequence lengths 512, 1024, 2048 and 4096; head dimensions 64 and 128; causal and non-causal in separate rows; one dtype per table. Expect a speedup that rises with sequence length while the fused kernel’s traffic stays flat, then a plateau where the exponential cost takes over. Record the fused kernel’s error against the dense reference at every shape, and the peak allocation of each build, because the allocation difference is the IO argument stated in bytes.

Things that break

Attributing stage times inside a fused kernel, which is a measurement of nothing. An ablation whose output nobody reads. A ratio that mixes causal and non-causal rows, so the number describes the mask rather than the fusion. A table where the baseline changed between rows and no note says so.