The generated file named its own A/B arm, and the experiment stopped being one

We run a two-arm test on article headlines. The arm is a pure function of the slug: sha256 of the slug decides it. Deterministic, replayable, no stored state, no clock.

Two places needed that value. The renderer wrote it into the file’s frontmatter. The publisher computed it again when it built the receipt that records the publication. The renderer read it from the draft the model had written. The publisher called the function.

On the first real article the two disagreed. The file said one arm and the receipt said the other.

Why a wrong label is the small part

The arm is a field in the published receipt, and every number we would later compute about the test reads from those receipts. A page claiming one arm while its receipt carries the other does not produce a slightly wrong measurement. It produces a measurement of an experiment that never ran, because the headline a reader saw and the row the analysis counted describe different articles.

Nothing downstream could have noticed. Both values are legal members of the same two-element set. Both pass every schema check the receipt has, because the schema validates that the field holds one of the permitted values and has no opinion about which. The failure lives entirely in the relationship between two files, and no validator we had was looking at a relationship.

The rule was already written, one field over

The same function already had this exact rule for a neighbouring field. The cluster identifier comes from the approved plan and never from the draft. The comment above that line explains why: a draft naming its own cluster could attribute an article to a cluster nobody approved, and the approval is attached to the plan rather than to the prose.

That is the same sentence with one noun changed. A draft naming its own arm is a draft quietly removing itself from the experiment.

So the defect was a fix that had been applied once and left unapplied four lines away. That shape keeps recurring here, and the pattern behind it seems to be that a fix gets attached to the field it was found on rather than to the rule it expresses.

Derived or declared, and the choice gets written down

The rule we ended on is small enough to state in a sentence. Every field in a generated artifact is either derived by the pipeline or declared by the author, the choice is made once per field, and the reason is recorded next to the code that enforces it.

Three fields in this one function now carry that note. The cluster comes from the plan, because approval lives there. The arm comes from the assignment function, because it is a hash of the slug and stays replayable. The publication date comes from the draft and never from a live clock, because a re-render months later must not restamp it, and a clock would make the function’s output depend on when it ran, which is the one thing a content hash cannot tolerate.

What actually caught it

No test caught it. The review pack did, by printing the arm it had computed next to the file it had been handed, at the moment a human was about to approve a publication. The two values sat one line apart and disagreed.

That is worth separating from luck. The pack prints both because they come from independent computations, and printing one derived value beside its independently derived twin is a cheap habit that turns a silent divergence into an obvious one. A single source of truth would have removed the contradiction by removing the second opinion.

One honest limit, since this is a working note and the numbers are not in yet. Five articles per arm against the sample size this kind of test actually needs is a collectable stream and nothing more. Anyone reading a difference between the two arms at this volume is reading noise, and that was written down before the first article shipped rather than after the first result looked interesting.