One-command benchmark

One target, no arguments

BINS     := bin/matmul_scalar bin/ew_add
QEMU_CPU ?= rv64,v=true,vlen=256,elen=64,vext_spec=v1.0
CFLAGS   ?= -O2 -march=rv64gcv -mabi=lp64d -static

.PHONY: all bench clean FORCE
FORCE:
all: $(BINS)

bench: all
	@./scripts/run_bench.sh

bin/%: kernels/%.c bench/harness.c FORCE
	@mkdir -p bin
	riscv64-linux-gnu-gcc $(CFLAGS) -Ibench -o $@ kernels/$*.c bench/harness.c

FORCE is an empty phony target, so every binary is always out of date and every make bench relinks before it measures. Without it, make compares timestamps, finds the binary current, and times whatever was built in an earlier session. That is the most common way a results table becomes fiction.

scripts/run_bench.sh does three things in order: print the environment, run each binary, write one line per kernel. The header is part of the output:

# gcc <version>  qemu <version>  cpu rv64,v=true,vlen=256  commit <sha>
matmul_scalar	512x512x512	<median>	<spread>	20
ew_add	1048576	<median>	<spread>	20

Name, shape, median GFLOP/s or GB/s, spread, sample count. Fixed order and tab separated, so a later script turns the file into the README table instead of a person retyping it.

Make it fail loudly

set -eu at the top of the script, and a check that each binary exists before it is run. A bench target that prints four of five kernels and exits zero is worse than one that fails, because the missing row looks like an omission rather than an error.

Run it serially. make -j8 bench overlaps kernels and the numbers stop being comparable with anything you recorded before.

Things that break

If make bench finishes suspiciously fast and prints numbers you do not recognise, check whether the rebuild is real: make -n bench prints the commands without executing them. If the header line and the binary disagree about the QEMU CPU string, one of the two is hard-coded where it should be a single variable.