Phase 3, week 23retrospective

Upstream regression test

A regression test must fail before the patch and pass after it, pinning the exact behavior fixed.


Fail first, and record it

Check out the parent commit, run the test, and save the output. A test that passes before the fix pins nothing, and that is the commonest way a regression test becomes decoration. Put the recorded failure in the pull request description, where a reviewer can see the test is connected to the bug rather than to the commit.

Check the semantics

CHECK-LABEL for the function, CHECK for the instruction that must appear, CHECK-NOT for the scalar loop that must not. Prefer the opcode and its operand shape over register numbers, unless the register allocation is the bug. vfredusum appearing at all is the claim; v8 carrying it is an allocation detail that changes for unrelated reasons.

Put it where the suite looks

A lit test in the directory for the component, with a RUN line carrying the minimum -mattr that reproduces the case, and a comment linking the issue. For MLIR, the same shape with the pass pipeline and a check prefix. A test in the wrong directory runs in the wrong configuration, or in none.

Prove the negative case

The tail, the mask, the dynamic shape: each needs its own test, because the fix that makes the common case work is the one that breaks the edge. If the patch changes an existing check, read the change before regenerating the file.

Things that break

CHECK: ret, which passes on the broken output and on almost anything else. A test that fails before the fix for the wrong reason, such as a missing declaration rather than the wrong instruction. Register numbers that break on an unrelated allocation change. And no REQUIRES line, so the test fails on bots that cannot run it.

Sources

A version records the documentation the claim was checked against, not a house pin. Confirm flags with --help on the release you installed.

Check yourself

What makes a codegen test a regression test rather than a description?

Why prefer the opcode over the register number in a `CHECK` line?

What does `CHECK-NOT` add?

0 / 3 correct

← back to the map