Timing harness with medians

A reusable timing harness warms up, records repeated samples, and prints median GFLOP/s for any kernel.

What the harness must do

bench/harness.c exposes one entry point:

double bench_run(const char *name, void (*fn)(void *), void *ctx,
                 double flops_per_call, int warmup, int reps, int inner);

Each sample reads clock_gettime(CLOCK_MONOTONIC, &ts), runs inner calls, and converts the elapsed time to GFLOP/s as flops_per_call * inner / (seconds * 1e9). The harness then sorts the samples and prints median, p10, p90 and the sample count on one line. Use 1e9 rather than 2^30 if the figure is to match anyone else’s GFLOP/s.

Warm up, then sample

The first call pays for page faults, cold caches and QEMU’s translation blocks. Run the kernel warmup times and throw those timings away, then take reps samples. Size inner so that one sample lasts many milliseconds: when a sample is a single small kernel call, the clock read is a large part of what you measured. Timing the process from the shell has the same defect, because guest startup dwarfs the kernel.

Report the spread

Print the median and the p10-p90 band. A single best run measures host scheduling luck. When the band is wider than the gap between two kernels, you cannot yet tell them apart, and the output should say so rather than show whichever median came out lower.

Emulated runs are structurally noisy: host frequency scaling, other processes, and QEMU’s own bookkeeping. LLVM’s benchmarking notes list the usual mitigations, and they apply here in proportion.

Things that break

At -O2 a kernel whose result nobody reads can vanish, leaving a near-zero time that looks like a triumph. Consume the output through a volatile sink or a printed checksum. If the same binary reports wildly different medians across runs, look for a missing rebuild before you look for a hardware explanation.

Sources