Do LLM judges give the same verdict twice?

We ran the same benchmark, twice. 8.2% of the verdicts flipped. This is the reproducibility test almost no one runs on their LLM judge, and the reason a single-run dashboard number lies to your team.

An LLM judge is a probabilistic system grading a probabilistic system. Both sides of that sentence carry noise. Product teams size the noise on the AI feature they're grading (reruns, temperature settings, side-by-side comparisons) and then they take the judge's verdict as if it were ground truth: one run, one number, case closed.

We ran the identical judge benchmark twice at the lowest sampling setting available. 205 of 2,490 verdicts changed between the two runs. Same 8 judge prompts, identical. Same 34 responses. Same 10 models. Same settings. 8.2% of the verdicts said PASS one week and FAIL the next, with nothing changed but the sampling dice.

Every LLM eval dashboard we've seen treats a verdict as a fact. On this data, one in twelve verdicts is a coin flip pretending to be a fact.

The setup

The full experiment is written up in the battle of the LLM judges: 8 judges, 10 models, 34 human-labeled AI agent responses, over 5,000 verdicts checked against a human expert. The reproducibility test was built into the design from the start.

We ran the full benchmark twice. Identical prompts. Identical models. Identical dataset (17 correct answers, 17 real errors, all with human labels and written justifications). Identical settings, with temperature and sampling pinned as low as each model would allow.

That's 2,490 verdicts per run. What changed between them was nothing that a team running this in production could control.

Then we counted, for every judge-model-case combination, whether the second run agreed with the first.

The result

The self-agreement was worse than the judge's agreement with the human on many pairings. Which is the setup for the argument this piece exists to make: the judges disagree with themselves before they ever get to disagree with the human. A judge that can't reproduce its own verdict has no business grading yours.

Which models flip, which don't

The 8.2% number is an average. Verdict stability turned out to be a model property, not a judge-prompt property. The same judge prompt was rock-steady on one model and shaky on another:

gpt-4.1 barely moves: seven verdicts in a thousand flip. Anything from that model, one run is basically two.

The small GPTs are the shakiest by an order of magnitude. gpt-5.4-nano flipped one verdict in six between two identical runs at the lowest possible temperature. Any single-run number for those models is not what it looks like on the dashboard.

The pattern to sit with: the models most teams reach for when they want a cheap judge (nano, mini, Haiku) are the least self-consistent. Cost efficiency and verdict stability point in opposite directions. And no one is checking, because nobody re-runs.

Why this breaks the single-score dashboard

The habit almost every team we've talked to has settled into: run the eval, look at the score, move on. Green means ship, red means investigate, and yellow gets a shrug and a stand-up mention. The whole workflow assumes the score is telling you something durable.

Two things happen on this data that break that habit.

One flipped verdict on a 34-case set is 2.9 points. An 8% flip rate moves 2–3 verdicts per pairing per re-run. Which means a pairing's headline agreement number carries roughly ±10 points of run-to-run noise on 34 cases. Bump your test set to 300 and the noise-per-verdict shrinks, but only if the model itself is stable. If it isn't, more cases give you a more confident measurement of an unstable judge.

Individual pairings swing far more than the averages. Our own custom judge on gpt-5.4 rose from 61.8% to 76.5% between the two runs. On gpt-5.4-mini it fell from 61.8% to 50.0%. Identical prompt, identical responses, identical settings, and the only variable that moved was the calendar date.

The averages across all 10 models moved less than 2 points per judge and the ranking didn't change. Averages are stable. Records are not. The single-pairing headline number that read like a fact was noise pretending to be one.

Every claim of "our judge is 76% accurate" on a single run is a claim that could read 66% or 86% next Tuesday, unless the model under it is one of the stable few.

What the flips mean when you're comparing judges

A team pilots two judge prompts, run once each, and picks the winner. That's the standard comparison. On our data, a 3-point difference between two candidate prompts on a single 34-case run is inside the noise. The "winner" and the "loser" might swap places on the next run.

Which is why the parent benchmark quotes averages across models and ranges, not records. It's why we ran the full experiment twice. And it's why the numbers you'll see cited from this work (76.5% top pairing, 68.8% average, 8.2% flip rate, 26-point spread across models) are the ones that survived a re-run at their reported values.

If your judge picks are based on single-run numbers, half your comparisons are wrong direction.

What to do

If you're already running an LLM judge, this is a one-hour experiment you can do this week:

1. •Take your existing golden set. Small is fine, 30 cases is enough to see the effect.
2. •Run your judge on it. Save the verdicts.
3. •Re-run it, same settings, same data, later the same day. Save those verdicts.
4. •Count the flips. Divide by total.
5. •If the flip rate is over 5%, treat any single-run number your judge produces as a rough estimate, not a measurement. Report averages across at least two runs. Or move to a more stable model: the Claude family and gpt-4.1 clocked under 5% on our data.

For teams building a judge from scratch, the reproducibility check goes into the validation loop from the beginning. Not because it's exotic. Because it's cheap, and because the numbers you build a product decision on should agree with themselves before they compete with the human's.

The broader point

The judge you installed to remove noise from your AI evaluation is adding a layer of noise you never measured. The dashboard is not lying, exactly. It's telling you the truth of this run, at these sampling dice, and letting you assume that generalizes.

Some judges do. gpt-4.1 as the judge model, on our data, generalizes across runs almost perfectly. gpt-5.4-nano as the judge model does not, and there is no version of "look at the dashboard" that will tell you which one you have.

Run the benchmark twice. Count the flips. Then decide what the score means.

---

The full 5,000-verdict experiment is in The battle of the LLM judges. The reproducibility measurement is Finding 5.

If you're building a judge for a production AI feature and want it stable enough to actually act on, that's what we do. Talk to us.