Reproducible artifacts

The bundle, not the post

A reader who wants to check the post needs more than the repository at HEAD. The bundle is: the commit SHA and a tag; the benchmark script; the raw results/<commit>.tsv for every table row; the plotting script that turns the raw file into every figure; the toolchain versions (LLVM and MLIR, QEMU, host compiler); the QEMU CPU string; the host CPU model; the sample count; and the estimator used for each ratio.

artifact/
  README.md            # rig, versions, how to run
  scripts/repro.sh     # one entry point
  results/<commit>.tsv # raw samples
  scripts/plots.py     # regenerates every figure
  CITATION.cff

Archive it with a DOI so the link survives a repository rename or a branch rewrite. A tag is not an archive if the remote can force-push; a release tarball plus a DOI is. CITATION.cff tells a reader how to cite the artifact, and a licence tells them what they may do with it.

What reproducible can mean

On the same rig, a rerun should land inside the published bands. On a different host, the absolute numbers will differ, and the honest claim is that the method reproduces and the qualitative ordering holds. Write which claim the artifact supports; “independently reproducible” without that qualifier is a promise the artifact cannot keep.

The check is the stranger test from the benchmark script: fresh clone, ./scripts/repro.sh, compare with the table. Every step you needed and did not write down is a missing line.

Things that break

A DOI that points at a mutable repository is not an archive. Raw data without the plotting script leaves the figures unverifiable. And a bundle that omits the QEMU CPU string reproduces a different rig, which is the failure the whole protocol exists to prevent.