Two agents reviewed the same code and agreed on almost nothing

We had a validator that was green. Nineteen CI jobs across three operating systems, every one passing, and the rule it enforced had been written carefully enough that its own comments explained why each guard was there.

Then we pointed two agents at it, with one instruction each and no knowledge of the other. One was told to attack the decision logic: what the rule says, where it is ambiguous, what falls between its cases. The other was told to attack the encoding boundary: what happens to a character on its way in from a shell, a file, a JSON round-trip.

Between them they returned around thirty-eight findings. The two lists overlapped on two.

That number is the whole argument for running more than one, and it is not the argument we expected to be making. We assumed redundancy: two reviewers, some duplicated effort, slightly better coverage. What actually happened is that each agent found a category the other never approached, because the assignment determined the category. The one reading decision logic proved that our refused set was a hand-picked sample rather than the class we claimed, and that our allow-list granted the exact harm the refuse-list existed to prevent. The one reading the encoding boundary never mentioned either, and instead spent its time on what a lone surrogate does to a hash.

Redundancy is not the point, and it is the expensive way to buy nothing

If you run two agents with the same instruction you get two samples from one distribution. Their blind spot is shared, because the blind spot comes from the framing, not from the effort. Adding a third does not help; it is the same distribution again.

Surface assignment is what makes the second agent worth its cost. “Attack this” produces overlap. “Attack this from the filesystem’s point of view” produces a different list, because the agent is now searching a space the first one never entered.

The practical shape is unglamorous. Write down the two or three ways the thing under review can be wrong that have nothing to do with each other, and make each one somebody’s whole job. In our case it was: does the rule say what we think, and does anything change the bytes before the rule sees them. Two different questions about one function.

The finding that was wrong, and what it cost us to check

One of the two reported a serious defect: a lossy string-to-bytes conversion happening before any validation ran, so the guard we had just written could never fire. It came with a reproduction, a table, and a specific line number.

It was wrong, and it was wrong in an instructive way. The conversion it named really is lossy. But the loss happens earlier than that line, out at the process boundary, where the runtime substitutes a replacement character before our code starts. We confirmed this by probing directly rather than by reasoning: send a character in, print what the receiving process actually got, compare. The guard the agent’s diagnosis implied could never have fired on any real path.

We had already written that guard by the time we checked. It would have shipped as a few lines that read like protection and were incapable of running.

Treat agent output as a hypothesis with a repro attached

This is the rule we ended up with, and it is a rule about workflow rather than about models. A finding from an agent is a claim plus an experiment. Run the experiment. If it does not reproduce the way the finding describes, the finding is not necessarily false, but its diagnosis is not yet true, and the fix you build from the diagnosis will be aimed at the wrong place.

The cost of checking is a few minutes. The cost of not checking is a guard that cannot fire, sitting in the file looking like coverage, which is worse than an absent guard because nobody will look there again.

There is a sharper version of this for anyone building review into a pipeline. The failure is not the agent being wrong. Agents being wrong is priced in. The failure is a workflow that converts confident output into committed code without an execution step in between, because that workflow will ship the wrong fixes with exactly the same fluency as the right ones.

Where else this hides

Anywhere you fan work out to several workers and then merge their results.

If the workers received the same brief, expect correlated misses and do not read agreement as confirmation. Two agents agreeing tells you the framing was consistent, not that the answer is right.

If the workers received different briefs, expect a merge problem instead: findings that contradict each other, or that are each true about a different layer. That is the good case, and it is more work to read.

And measure the overlap. It is one number, it costs nothing to compute, and it tells you whether your second worker is buying coverage or buying a second opinion on the first worker’s blind spot. Ours was two out of thirty-eight, which is why there will be two next time.