Documentation
The thesis in one line: a rule in markdown is followed almost never, a rule in CI is followed every time. What follows is the evidence behind that sentence.
The problem, measured
In a forensic pass over 161 commits across six repositories, the three without CI were exactly the three with a broken lint at that moment. The repository with the most governance paperwork — a strict AGENTS.md, SECURITY.md, GOVERNANCE.md, CONTRIBUTING.md — had 35 lint errors and a verification script that nothing ever ran.
- Repositories without CI: 3 of 6
- The same 3 with a broken lint right now
- Commits with AI co-authorship: 41 of 161
- Repository with the most governance: 35 lint errors
If a rule can move down a level, it should
The levels run from N0 to N7. N0 is the compiler, N1 static analysis, N4 CI, N5 the hook. Above N5 it is a machine; below it is a request. Every rule that lives in a request and would fit in a machine is debt, and the bill is measurable: a hundred lines of always-present rules, across thirty sessions a week for a year, run past two million tokens.
- N0 compiler — fails as "it does not compile"
- N1 static analysis — "it does not pass lint"
- N4 CI — "it does not land on main"
- N5 hook — "the action does not happen"
- N6 instruction to an AI — depends on reading and obeying
Every rule is born with two cases
Each rule has a miniature repository it fails and another it passes, assembled in a temporary directory with a git of its own. Nothing is ever written into the live repository. The passing side deliberately carries the expected false positives: the example inside a comment, the sample file that is the fix rather than the flaw, the legitimate pre-hash sitting next to bcrypt.
- Reading the exit code alone was not enough: passed and not-applicable both come out as zero
- Loosening a rule has to paint the passing side red on the first run
- A pair impossible to assemble is a rule that does not ship
What does not become a rule, and why
Eight classes of flaw were left out on honesty grounds: in each of them, two identical trees on disk get opposite verdicts. Rate limiting lives in Cloudflare, authorization lives in a policy, RLS may be switched on from a dashboard. Each one became a written question, answered with a file path, instead of an invented score.
- IDOR and per-object authorization
- Rate limiting and brute force
- IP treated as identity
- RLS enabled outside the repository
Exit codes
127 dominates 1, and that is not a detail: you do not accuse a repository with a ruler that broke. An exception inside a rule is a defect of the tool and never enters the target's score.
- 0 — everything that applies passed
- 1 — failed, a real violation
- 2 — invalid target or wrong invocation
- 127 — broke, a defect of the tool
What is not proven yet
rebar is enforced in two repositories. One rule still only warns, because close to twelve per cent of what it flags is noise from interface vocabulary. And the most expensive lesson of the tree: the gate stayed green for six commits while the generator was dead, because no step ever generated a project. All the coverage sat on the tooling and none on what is shipped.
- The generator got a step of its own after that, proven by mutation
- The secret scanner had no proof at all until an outside audit found a leak in UTF-16