The eval that could not tell a fix from variance

The plan for the last day of a build was tidy. Run the adversarial eval suite for a baseline. It comes back four of six. Read the two failures, fix them in the prompt, re-run, report six of six. A clean before-and-after, the kind that goes on a slide.

I got the four of six. I fixed the two rows. And then, because my own plan had a note saying the suite wasn't deterministic and to run each label twice, I ran the baseline again. Same prompt, untouched, ninety minutes after the first run. It came back six of six.

Nothing had changed. Both defects had vanished with no code at all.

What the numbers actually were

Here is the grid. It is the whole essay.

Each defect showed up in exactly one run out of four. My "after" was two runs with zero failures. But two clean runs is exactly what a fix that did nothing would produce, because the failure only happens about one time in four to begin with. The after runs look like more baseline. I had no delta. I had noise with a story wrapped around it.

Why this is worth writing down

The comfortable move is to delete the second baseline and ship the six of six. You could even defend it. The defects were real, I saw them, I have the transcripts and the exact sentence in the prompt that produced the double-broker behaviour. The fix is a reasonable fix. Nobody would ever know the effect size was unmeasured.

But the reason you run an eval is to know something, and what this eval could tell me was smaller than what I wanted to claim. One before run and one after run can't separate a real improvement from model variance when the failure is intermittent. To measure the fix I'd need many runs on each side and a test that the difference is bigger than the run-to-run spread. I didn't have the runs. So the honest report is: the defects are real, the fix is plausible, the effect size is unknown.

That sentence went into the artifact verbatim, as a numbered decision. It is the least impressive line in the whole build document and the one I'd most want a hiring manager to read, because it is the difference between someone who produces green checkmarks and someone who knows what a green checkmark is worth.

The general shape

Agents are non-deterministic. So an eval on an agent is a measurement with error bars, and most agent evals I see get reported as if they had none. A number with no sense of its own variance isn't evidence yet. It's a feeling with a decimal point.

The discipline that falls out of this is short.

• Two baselines before you touch anything. If they disagree, your noise floor is above the effect you're hoping to find, and any single after run is theatre.
• Report the spread, not just the point. "Passed two of two" and "passed twelve of twelve" are different claims. Say which one you have.
• Separate the observation from the effect. "The model did this bad thing, here is the transcript" is a solid claim from one run. "My change fixed it" is a claim about a distribution, and it needs distribution-sized evidence.
• When you can't afford the runs, say the size is unmeasured. That is a real result. It is worth more than a fake one.

The suite still did its job. It found the model arguing itself into double-brokering a load, and it found that the tool layer stopped it anyway, which was the point of the whole project. What it couldn't tell me was how much my prompt edit helped. The valuable part was noticing that and writing it down instead of rounding it up.