A Check That Never Ran Looks Exactly Like One That Passed

The green that meant nothing

We had a step in a code-generation pipeline whose only job was to type-check the generated output before it got swapped into the app. For a little over a week, it passed every single time.

It had never looked at the generated output once.

The generator wrote into a scratch folder whose name started with a dot — deliberately, so git, the test runner, the formatter and the spell-checker would all leave a twenty-second-lived folder alone. The type-checker is a glob-driven tool too, and src/** doesn’t descend into dot-folders. So the checker walked the 549 files of the app, found the 178 type errors that already lived there, and the step filtered for errors under the scratch folder — of which there were none. Zero rows. Clean bill of health. Swap it in.

When we fixed the glob, the very next run refused with 157 errors in the generated code.

An earlier version of the same step had tried to guard against exactly this by checking the tool’s own COMPLETED <n> FILES line. That count can’t answer the question either: TypeScript follows imports out of the target folder into the rest of the app, so the total comes back in the hundreds whether or not the folder exists. I measured it by hand: COMPLETED 268 FILES 0 ERRORS, against a folder that did not exist.

It’s not one bug, it’s a class

Once you start looking for this, it’s everywhere. From one codebase, in one quarter:

A missing Python virtualenv turned three tools green. The linter, the type-checker and the test runner each skipped quietly when .venv was absent. “Skipped” rendered as a tick.
A 404 that should have been a 403. A security scan fetched its stored baseline with a CI job token. GitLab answers that endpoint with 404, not 403, when the caller can’t see the project — so a credentials failure wore the costume of “no baseline yet, establishing one.” It established a baseline on every run for the life of the feature.
curl -sS without -f. Exit code 0 on an HTTP 500. The script carried on with an error page as its input.
A security scan that stopped for 19 hours. Three sensible rules: a held promotion exits 0 because a hold isn’t a failure; the alerts channel only speaks when something is wrong; and “nothing changed” scan results had just been muted because ninety identical posts a day is spam. The scans only ran after a promotion. Promotion was held. Nothing ran, and every one of those rules correctly said nothing.

None of these is exotic. Each is the default behavior of an ordinary tool meeting an ordinary edge case. That’s what makes the class dangerous: the failure mode of a check is silence, and silence is the same signal as success.

What actually fixes it

Count your inputs, on both sides. The type-check step now counts the generated folder’s own .js and .svelte files itself — before the run and after — and refuses on zero or on any change between them. Don’t ask the tool how much it looked at; it will answer a different question confidently.

Make “did not run” its own result. Our review runner grew a tool-did-not-run violation. A tool that skips because its environment is missing is a failure of the gate, not a pass of the code. A cached result prints as a loud SKIPPED (cache) line naming the cache — never as a green tick.

Alarm on absence, not confirmation on presence. The obvious fix for the silent security scan was “post on every clean run so we know it’s running.” That’s ~90 messages a day, and a confirmation stream nobody reads is one whose absence nobody notices either — you trade one silent failure for a noisier silent failure. Instead, a scheduled job (not a deploy-triggered one; the deploy is what stopped) checks when each scan last reported, and alarms when one is overdue.

Its threshold is 24 hours. That would not have caught the 19-hour gap that prompted it — and a test pins that on purpose. An alarm that fires on every slow weekend gets muted; one that fires rarely and is always right gets read. Late and believed beats immediate and ignored.

Treat a reassuring number with the same suspicion as an alarming one. The security scan’s ratchet once compared against a report from a scan that hadn’t reached its target. The report was valid JSON with zero findings, so it announced every known issue as fixed. A false improvement is worse than a false alarm, because nobody investigates good news.

The test you’re not writing

Every gate has a test that feeds it bad code and asserts it goes red. Write the other one:

Delete the input and assert the gate fails.
Remove the tool (or its environment) and assert the gate fails.
Break the credential and assert the gate fails with a credentials error, not a “first run” message.
Stop the trigger and assert something, somewhere, notices within a bounded time.

If any of those go green, you don’t have a gate. You have a decoration that’s correct whenever nothing is wrong — which is exactly when you didn’t need it.

The principle

Absence is never assent. A check earns trust by proving it looked, not by failing to complain. Design every gate so that “I didn’t see anything” and “I saw nothing wrong” are different outputs — and then test that it can tell them apart.