The Test Suite Shipped
A Python test suite that has grown from 5,159 to 7,360 tests in about a month, green at every recorded checkpoint.
Growth, from the record
Failing was zero at every one of these checkpoints. The dip stays on the chart because it is real: the run-watch subcommand was rewritten in-process and removed, which is what a healthy suite shrinking looks like.
And a lesson about what tests cannot catch
A critical routing bug once slipped through the end-to-end suite entirely: a handoff document reached no specialist worker because a synthetic file change carrying no technology signals was filtered out of every discretionary lane. The mock-LLM test could not possibly have caught it — a mock that ignores its prompt cannot reveal that the prompt was missing its most important input. Routing is now asserted deterministically.
A second one, from the field
A five-thousand-test green suite once hid a fully broken production path, because every test called an inner function and none of them went through the entry point. Written down at the time, and worth repeating: coverage of the units is not coverage of the program.