An AI agent's report about the work is not the work

For one full cycle, our design pipeline produced more evidence about its own output than anyone could read. Five critique rounds. Three blind rankings, each by a separate agent. Receipts on every step, a prediction sealed before the results came in, and an external review package. Every one of those artifacts pointed the same way: the three design directions were strong, and they were genuinely different from each other.

Then the owner opened the screenshots, looked once, and scored the set 23/100. All three looked like one design (build rules, ADR-0049).

Nobody in that loop lied. Every agent reported accurately on what it had been handed. The session running the cycle carried those reports forward as the verdict, and at no point did it open the pixels the reports were about.

A report is a claim about an artifact

When one AI agent hands work to another, it hands over a summary. The summary is a claim. The screenshot, the diff or the page is the thing the claim is about. Those two are easy to confuse because the summary is usually right, and when it is right, reading it is much cheaper than reading the artifact.

Stack agents and the confusion compounds. The critic reads the render. The jury reads the critic. The session reads the jury. After the first hop, nothing in the chain has touched the artifact, and every later step is grading the step before it. The chain can be perfectly consistent and entirely about the wrong thing.

A controlled test afterwards made the gap hard to argue with. Three general-purpose agents, given the same content and a plain “make it world-class” prompt, with no brief, no critic and no jury, produced materially better work than the full pipeline (ADR-0049). The machinery had not been adding judgment. It had been adding confidence.

Ranking always finds a winner

Two of the causes are worth naming because they generalise to any agent loop that grades its own output.

The pass condition was an absence. A run passed when the critic found zero violations, so a pass meant “broke no rule”. Compliant work with no character in it passed five runs in a row, because nothing in the definition of a pass could fail it for being dull.

The jury could only compare. It ranked three variants against each other, and a ranking of three always produces a first place. It never produces a bar. If all three candidates are mediocre, the jury still returns a winner, and the report looks exactly like the report from a run where the winner was excellent.

Our fix for the first was a finding class, BELOW-BAR, that fails work for being insufficient rather than for breaking something. It is a judgment with no number attached, which is its weakness, and the decision record says so in its own watch-for section rather than pretending otherwise (ADR-0049).

The control that costs one look

The fix for the chain was not another agent. It was a rule that the session carrying a verdict opens the artifact first: “look at the artifact before carrying its verdict” is now a standing line in our build rules (build rules).

That sounds too small to matter. It is the only step in the loop with a different information source from the rest. Every agent before it saw what the previous agent wrote. A look at the artifact sees what was actually made.

What we would tell you to check

If you run AI agents in a chain, find the last step where a person or a script touched the artifact itself rather than a description of it. Count the hops between that step and the place where the verdict gets acted on. Every hop is a place where a correct summary of the wrong thing can travel unchallenged.

Two of our other notes are versions of the same failure. In one, two agents reviewed the same code and agreed on almost nothing, so their agreement could not stand in for coverage. In the other, a generated file named its own A/B arm, and both copies of the value passed every check because each check read only one of them.