Rigorous benchmarking

The protocol

Two kernels are comparable only when everything except the kernel is equal: the same buffers, shape, seed and alignment; a reference result computed once by ref/matmul_ref.c and checked before any timing; a warmup whose timings are discarded; at least twenty samples; one rig, meaning one binary set, one QEMU CPU string, and nothing else running.

Interleaving is not fussiness. Host frequency and thermal state drift over minutes, so a block of A followed by a block of B measures the drift as much as it measures the kernels. Alternating makes both paths meet the same machine.

The output

matmul_scalar  512x512x512  <median>  <spread>  20
matmul_rvv     512x512x512  <median>  <spread>  20

<speedup> is a ratio and means nothing without both bands. When the bands overlap, the honest sentence is that the two kernels are not distinguishable at this sample count, and the next move is more samples or a quieter rig, not a smaller font. No figure in this node is a measurement. The harness prints yours.

One favourable run

A single run is a sample, and the one you kept is the one that looked best. Once runs are interleaved you can do better than a ratio of medians: pair sample i of A with sample i of B, take the ratio within each pair, and report the median ratio. It is a different estimator, it uses the pairing you paid for, and it usually comes with a tighter band.

Things that break

The band is wider than the gap between medians and the table is published anyway. The numbers are not wrong; the conclusion is unsupported, and no reader can tell from the table alone. The other failure is comparability. One path recompiled at -O3, or two runs under different vlen, produce a ratio that describes the change in the rig, and the results file records neither unless the script wrote the flags down.