Performance regression guard

A guard is a comparison

A correctness test asks whether the output is right. A codegen regression produces the right output more slowly, so the guard compares a candidate against a baseline under the same conditions and fails when the difference exceeds what the rig noise can explain.

for i in $(seq 1 20); do
  for k in canary candidate; do
    printf '%s\t' "$k" >> "results/<commit>/guard.tsv"
    "./bin/matmul_$k" >> "results/<commit>/guard.tsv"
  done
done

Derive the threshold from measured spread, not from a round number. Run the baseline against itself to get the distribution of ratios a no-change build produces, and set the threshold above that band. On QEMU the band is wide, so the guard catches large regressions and misses small ones; the minimum detectable regression belongs in the README.

Normalise by a canary

The machine changes between CI runs. Keep a frozen canary kernel that nobody edits, measure it in the same session, and compare the candidate-to-canary ratio rather than the absolute time. If the canary is also slower, the rig moved; if only the candidate moved, the code did. This is the same pairing argument as the interleaved benchmark, applied to a build instead of a kernel.

Store the raw samples, the commit and the flags with the run. A guard whose threshold lives only in a CI variable cannot be audited when it fires, and the first response to a noisy failure is to disable it.

Keep the harness, the command and the threshold in the repository, under the same version control as the code they judge.

Things that break

A threshold set from one baseline run fails on the next quiet machine or passes a real regression on a noisy one. A baseline rebuilt with different flags compares two changes at once. And a guard that runs only the kernel, without the reference check, will bless a fast wrong answer.