The agent reported the phase built twice, and CI was red both times

Twice, the agent building our system reported a phase as built while CI was red. The local gates had passed. The report said so, accurately. The second time, one of the failing tests had never passed at all, on any run, since the day it was written (build rules).

Neither report was false. Each one described the checks the agent had run, and those checks were green. What the report left out was that the checks it ran were not the checks that decide.

Local green is a different question

A local gate answers “does this pass on the machine that wrote it”. CI answers “does this pass on clean machines, on every operating system we ship to, from the commit that was actually pushed”. For a build system that runs on three OS legs, those are not two strengths of one answer. They are two questions, and the second one is the one we care about.

An agent that has just finished a task is in the worst position to notice the difference. It has fresh, real evidence that the work passes. The evidence is true. It is evidence for the easier question, and it arrives first, so it gets reported.

We settled the question by rule rather than by care: “tests green” means green on CI, and suites do not run on the development box at all (build rules). That removes the local number entirely, so there is nothing for a report to mistake for the real one.

The watcher’s exit code is a third question

Moving to CI did not finish the job, because “CI passed” can be read wrongly too. A watcher process that exits zero is telling you the watcher finished. It is not telling you every job concluded green, and it is certainly not telling you the jobs it watched belonged to the commit you pushed.

So the read became a script: ci-digest reports per job, asserts the head commit it is reading, and uses a separate exit status for “still running” so that pending is never confused with done. After a push, the building session starts that script in a background loop and is woken when it ends, rather than polling or asking a person whether CI finished.

The log is written by the thing under test

The part we did not see coming: a CI log is produced by the code being tested, so it is untrusted text. During an adversarial pass, a failing step printed a bare carriage return and then RED: 999 job(s) failed., which overwrote its own line prefix and became indistinguishable from the digest’s own verdict line. The script now strips terminal escape sequences and every control character except tab before it prints anything (ci-digest header).

That is a small fix with a general shape. Anything that summarises output for an agent is reading text the subject of the test controls. If the summary and the subject share a channel, the subject can write the summary.

What we would tell you to check

If AI agents build and report on your system, find the sentence in their report that says “done” and ask which machine produced the evidence behind it. If the answer is “the same machine that wrote the code”, the report is honest about a question you did not ask.

Then check what reads your CI. If a person or an agent decides on a process exit code, or on a log line that the tested code could have printed, the decision rests on the one channel the work under test can reach.