A platform that a flying club depends on to know which aircraft is airworthy cannot ship on optimism. The test strategy is three-layered: unit tests for pure logic, integration tests against a real throwaway Postgres, and Playwright end-to-end tests against a production build. What makes it interesting is not the count but the plumbing, because getting a suite this size to run in a sane amount of time on self-hosted hardware turned into a genuine systems-debugging exercise involving fsync, tmpfs, and Postgres inode behaviour.
What it's built with.
How it works.
Why the database runs in RAM
The integration suite truncates every table between every test so that no test can inherit another's state. That is the right call for isolation and it is brutal for durability, because each TRUNCATE forces an fsync and the suite issues over a thousand of them. On a hosted runner writing straight to an SSD the cost is close to invisible. On the self-hosted box it was not, because every flush had to cross four storage layers before it could return, and the symptom showed up in pg_stat_activity as waits on DataFileImmediateSync during TRUNCATE.
Putting the database on tmpfs with durability disabled removes the problem entirely, and it is safe here for a specific reason worth stating plainly: the database is created when the job starts and destroyed when it ends, and fsync only protects data across a crash. There is nothing in it worth recovering. That reasoning is documented in the workflow next to a warning never to carry the pattern to a real database, where disabling fsync risks corrupt data files rather than merely lost recent commits.
An ENOSPC that was not about space
The follow-on failure was more interesting than the original problem. Partway through the suite Postgres would fail to create a file with 'No space left on device' on a machine with hundreds of gigabytes free, and the mount itself still reported free bytes. What had actually run out was inodes.
TRUNCATE does not reuse a relation's file; it assigns a new relfilenode and leaves the old file to be unlinked at the next checkpoint, truncated to zero length in the meantime. At 441 relations per truncate and one truncate per test, several hundred thousand zero-byte files pile up between checkpoints, and tmpfs surfaces inode exhaustion using the same errno as a full disk. Forcing checkpoints often enough keeps the file count bounded, which one manual checkpoint mid-run demonstrated by dropping base/ from 406,609 files to 6,763 while the blocks in use did not move.
Promotion, not per-PR CI
Running the full suite on every pull request would have meant running it constantly against hardware it takes ten minutes to satisfy. The pipeline instead treats staging as the candidate: work merges to dev after local review, an hourly job advances staging to dev only when dev has moved, and the suite runs against that specific commit. Only a green run permits the promotion into main.
The consequence is that main only ever receives a tree that has already been tested as a unit, so pushes to main deliberately do not re-trigger the suite. Concurrency is the exception to the whole scheme and gets its own repeated gate, because a race condition that passes once has not been shown to be absent.
The things I'm proudest of.
- ▹675 test files split across three layers: 192 unit, 256 integration against a real ephemeral Postgres, and 227 Playwright end-to-end specs driven against a production build on that same database.
- ▹Cut the integration suite from over 33 minutes (never finished, killed) to 9m51s by running the throwaway Postgres on tmpfs with durability off, roughly a 3x win. The cause was that the harness truncates every table between every test, each TRUNCATE forces an fsync, and on a WSL2-backed self-hosted runner every flush had to traverse container overlayfs, WSL2 ext4, the virtual disk, and NTFS before returning.
- ▹Diagnosed a 'No space left on device' failure on a runner with 945 GB free. TRUNCATE assigns every table and index a new relfilenode and defers unlinking the old file to the next checkpoint, so at 441 relations per truncate and one truncate per test, roughly 500,000 zero-byte files accumulated and tmpfs reported inode exhaustion as ENOSPC. Forcing frequent checkpoints took base/ from 406,609 files back to 6,763 with block usage unchanged.
- ▹Gave concurrency its own promotion gate: the stress suites for aircraft meter updates and schedule double-booking run five times over, on the reasoning that race conditions are interleaving-dependent and one lucky pass inside the normal integration run is not evidence.
- ▹Built a promotion pipeline rather than per-PR CI: work lands on dev after local review, an hourly job advances staging to dev only when dev has actually moved, and a green suite against that exact candidate commit gates the merge into main, so a promotion always carries an already-tested tree.
- ▹Guarded the blast radius in the harness itself. The Postgres service is a job-scoped throwaway container that is never the shared dev database or the live club, and the test setup validates its target host so a misconfigured connection string fails loudly instead of truncating something real.
- ▹Kept 245 forward-only Drizzle migrations replayable as a standing discipline, since a chain that cannot be replayed from a fork point blocks staging from deploying at all.