A compiler's unit tests tell you very little, because the thing that matters is whether the emitted program produces the right output when it actually runs. Every test here is therefore a full pipeline: source in, compiled to LLVM IR by our compiler, assembled by llc, linked against the runtime library by clang, executed, and its stdout compared to an expected file. Memory correctness is checked in the same run rather than as a separate exercise.
What it's built with.
How it works.
End-to-end or it proves nothing
It is easy to write compiler tests that assert on intermediate structures, that the parse tree has a certain shape or that a type resolves as expected. Those tests pass while the compiler emits code that segfaults, because nothing in them ever runs the program.
Making every test a full compile-assemble-link-run-diff cycle means a passing test is evidence about the artifact users actually get. It is slower and it needs real toolchain plumbing per test, and it is the only version that answers the question that matters.
The icebox
A suite of tests for features that are known not to work yet occupies an unusual position: they should be written, because they encode the intended behaviour while it is still fresh, and they must not fail the build, because a permanently red pipeline is a pipeline nobody reads.
So the icebox runs on every build, counts its failures, and reports them as an expected condition, flipping to a success message on the day they all pass. The information stays in front of the team without costing the signal value of the main suites.
The things I'm proudest of.
- ▹Every test exercises the whole toolchain, four external steps per case: our compiler produces LLVM IR, llc produces an object file with position-independent relocation, clang links it against the shared runtime library, and the resulting binary runs and has its output diffed.
- ▹Runs under Valgrind memcheck in CI on all three suites, so a memory leak or invalid access in the compiler is a test failure rather than something discovered later. For a C++ codebase generating code, that is the check that actually protects you.
- ▹Five configuration profiles for different environments and suites, including a separate Mac profile, so the suite runs on the machines the team actually develops on rather than only in CI.
- ▹An icebox suite for features that are known to be unimplemented. It runs on every build and reports its failures, but deliberately does not fail the build, which keeps the known gap visible without training anyone to ignore a red pipeline.
- ▹Runs on a self-hosted runner inside the course's toolchain container, with the CMake build cached against a hash of the sources, so a test-only change does not pay for a full LLVM-linked rebuild.
- ▹Path-filtered triggers so the compiler suite only runs when compiler inputs change, and the build publishes its binaries on main.