Gates That Are Red on Day One Get Switched Off

The gate that dies in a week

You add a security scanner to the pipeline. You set the rule everyone agrees with: no medium-severity findings. The first run comes back with eleven — all pre-existing, none caused by the change that happened to trigger the scan. The pipeline is red. Someone has a release to ship.

By Friday the job has allow_failure: true. A month later nobody remembers it exists.

That’s the fate of most quality gates introduced into an existing codebase, and it isn’t because people don’t care about quality. A gate that blocks work it didn’t cause teaches the team that the gate is wrong. We had a local example sitting in the repo: a layout checker script, red and stale, wired into nothing.

Ratchets: block on the delta

The alternative is a ratchet. Store a baseline of what’s already there. On each run, compare against it. Block only on regression — new findings — never on debt the change didn’t introduce. The existing findings are recorded and can’t grow, and nobody’s release is held hostage to last year’s decisions.

The ratchet also made the cadence make sense. Our dev branch promoted to staging on every green pipeline — about fifteen merges a day, about ninety scans. What a passive security scan measures (headers, cookie flags, transport settings) changes maybe weekly. “Is our posture perfect?” is a question with a stale answer 14 runs out of 15. “Did this release change the posture?” is the only question worth asking ninety times a day.

That’s the easy part. The hard part is that a ratchet compares two things, and either of them can be missing, empty, malformed or wrong. Every one of those cases has to fail in a direction you chose on purpose.

Designing which way it fails

A missing baseline must establish, never accuse. On day one every finding is new. A naive ratchet blocks the very release that turned it on. So every way the baseline can be absent — first run, new app, expired artifact, refused token, network blip, malformed document, a baseline belonging to a different app — collapses to the same null, and null means “record this run as the baseline and block nothing.” That’s the single most important test in the module.

An empty run must never become the baseline. If a scan that reached nothing writes its zero findings as the new truth, the ratchet forgets everything, and the next real run reports every pre-existing finding as new — the false regression most likely to get the gate switched off. We guarded it twice on purpose: the job checks the scanner’s exit status before calling the writer, and the writer independently refuses a report with no alerts. The duplication is the point. The cheap guard stops the common case; the second holds when someone rewires the job.

A false improvement is worse than a false alarm. This one we found in review, after the first two were in place. A scanner can fail to reach its target and still leave syntactically valid JSON behind with zero findings. Compared against a real baseline, that reports every known finding as fixed. The channel would have printed “1 finding fixed, none added” right beneath “the scan did not run, so any result here means nothing.” Nobody investigates good news. The comparison is now gated on whether the run was actually comparable, not on whether the report parsed.

Key findings on identity, not instance count. A scanner that reports the same missing header on 40 URLs this run and 41 next run hasn’t regressed. Key on risk level plus finding name, so the ratchet tracks kinds of problems, not crawl noise.

Report before you block

Even with all that, we didn’t turn on blocking. The comparison had unit tests and had never met a real scanner report. Blocking a release because of a bug in the blocker is precisely the incident that teaches a team to disable a gate. So the first slice only reported the diff, with blocking as a one-line follow-up once the reports had been checked against reality.

That paid off immediately. Review found the comparison receiving the whole baseline document — {findings, recordedAt, commit} — where it expected the findings array. The type check said “not an array,” the code substituted an empty baseline, and every pre-existing finding was reported as new, on every run, forever. The pure comparison function was correct and well tested. The glue that called it was wrong, and nothing executed the glue. Had blocking been on, it would have been a permanent false block on day one.

A cousin: start at zero by cleaning up first

A ratchet isn’t the only way to avoid a red day one. For a clone-detection gate, we did the cleanup in the same merge request that introduced the gate, and seeded it at zero. That works when the debt is small enough to pay off in one sitting — and it gives you a stricter gate than any baseline would.

What doesn’t work is the third option: introducing the absolute rule against existing debt and hoping people will fix forward. They won’t. They’ll switch it off.

The principle

A gate’s first job is to survive its first week. Block only on what the change caused. Then spend your real design effort on the edges: every way the comparison can be missing, empty or misread must fail toward no false accusation and no false praise. Report before you block, and let reality check your fixtures before your fixtures hold up a release.