Which model should you run your LLM judge on?

The identical judge prompt scored 76.5% on one model and 50.0% on another in the same run. Same instructions, same data, same day. 26-point spread. The model is not an implementation detail on your LLM judge, it's half the judge. Here's how to pick, and how to price the choice.

Most teams choose their LLM judge model up front, before they've looked at any accuracy data. The choice comes from the observability tool's default, the model the API keys are already provisioned for, or the vendor blog post that ran an eval last month. Cost and latency get weighed. Agreement with the human rarely does.

On the benchmark this piece pulls from (8 judges, 10 models, 34 real AI agent responses, over 5,000 verdicts against expert human labels) the model choice moved the same prompt from 76.5% agreement with the human to a coin flip. Not across different tasks. On the same one.

Picking the model first, without measuring, is where roughly a quarter of your judge's accuracy goes to die.

The full experiment is The battle of the LLM judges. This piece is Finding 6, expanded, with the cost math laid alongside it.

The 26-point spread

Our custom judge prompt, held constant, run across 10 models in the final run:

Half your judge's accuracy sits inside a decision most teams make in five minutes on cost per token. That's what "the model is half the judge" means, in numbers.

The model has a temperament

The spread is not random. Certain models have durable temperaments that persist across prompts:

gpt-4.1 rubber-stamps. It agreed with the human at 58.8% on our custom judge, and it landed in a similar low-agreement zone on almost every generic template we ran. gpt-4.1 as a judge model tends to pass responses. That trait is unrelated to the rubric it's given.

gpt-5.4-mini and nano are the least stable. Both had the widest verdict swings between identical runs (14–17% flip rate, see do LLM judges give the same verdict twice?) and the lowest agreement in this table. Cheap and shaky are the same trait for these two.

Claude models are moderate and steady. Sonnet 5 and Opus 4.8 landed in the 70s and moved less than 5 points between re-runs. If you want a judge whose single-run number roughly matches its next-run number, the Claude family is where that lives.

gemini-3-flash and gpt-5.4 were the top pairings. Both hit 76.5% on the final run. gemini-3-flash held that ceiling across both runs; gpt-5.4 spiked to it in the second and sat at 61.8% in the first. Which is a reminder that even inside the top pairings, single-run scores need at least one confirmation.

Model temperament is the thing you're actually picking when you pick a model. And no one is running the measurement that would tell them which temperament matches the rubric they wrote.

Pairings do not transfer between prompt versions

Here's the mistake we made once and now watch for. On version 1 of our custom judge prompt, gemini-3.5-flash was the best model. We validated the pairing, shipped it, iterated the prompt to version 4, kept the same model. In the final experiment, gemini-3.5-flash on version 4 rubber-stamps.

Change the prompt, and the best model changes with it. The pairing is a single unit. You don't inherit last version's winner just because the prompt improved.

Every non-trivial prompt edit needs a re-measurement across models. Not the full 10, if you don't need to, but the 2 or 3 that most recently produced good pairings, plus one wildcard. Otherwise you'll ship a prompt improvement that's worse in production than the version it replaced, because you're now paired with a model that didn't like the new rubric.

The cost table, and the argument it settles

Averaged across the 34-response dataset (rough numbers, orders of magnitude, not procurement-grade):

Two things fall out of this table that most vendor decks don't tell you:

The best judge in the entire battle runs on a mid-tier Gemini for well under a cent a verdict. Accuracy came from the rubric, the pairing, and the validation. Not from spend.

Paying more does not buy a better judge. Claude Sonnet 5 costs three times as much as gemini-3-flash for equal-band agreement, and it can be as slow as 20 seconds per verdict. Claude Opus at $0.04/verdict is 16× more expensive than gemini-3-flash and lands three points lower on agreement.

DeepSeek grades as well as the middle of the Claude pack, and costs less, but is the slowest by far on our runs. Wrong buy on speed even though the accuracy holds up.

The "at least generics are cheaper to run" argument dies here too. A generic pairing costs the same tokens per verdict as the custom one. Its error catches cite the real failure 0–31% of the time (see Langfuse vs PostHog vs openevals). A judge whose verdicts you can't act on has an infinite cost per useful verdict. Not cheap. Wasted.

The one number that matters more than any of these

Every number above is for one product, one rubric, one judge prompt. The rankings do not port to your product.

The best judge model for our lead-tiering agent might be the worst for your document extraction feature. There is one way to find out, and it isn't a benchmark from a blog. It's a measurement on your own labeled data:

1. •Take your judge prompt (custom, ideally, since the generic templates cap out around 58% on our data).
2. •Run it across at least 5 models. Include one from each of the low-cost / mid-cost / premium bands.
3. •Score each pairing against a human-labeled set of at least 30 real outputs, half of them containing real errors.
4. •Track three numbers per pairing: agreement, missed real errors, false alarms on good answers.
5. •Run the top two pairings a second time. Count the flips. If a candidate flips more than 5% on identical inputs, drop it. A stable second-place pairing beats an unstable first-place one every time.

That's a couple of hours of work with a tool that runs experiments across models on the same dataset. It costs less than one week of an eng sprint. It saves you from donating a quarter of your judge's accuracy to whichever model happened to be the default.

The heuristic, if you want one

You don't want a heuristic. You want a measurement. But if you're going to make the choice before you have the data:

Prefer moderate-cost, moderate-latency models with known verdict stability. On our data, gemini-3-flash and the Claude Sonnet family fit that description. gpt-4.1 for stability if you're willing to accept a rubber-stamp lean and validate around it.

Skip the nano tier for judge work. The cost saving is real; the accuracy penalty is bigger, and the verdict instability multiplies it. Fine for cheap classification tasks. Not for a judge whose output your team is going to trust with quality decisions.

Skip the premium tier for judge work, unless a specific measurement earns it. Opus and Sonnet-3× tokens buy you nothing on this data over a Gemini or a mid-tier GPT.

And then measure the pairing, because half of what those brackets buy you is variance the ranking can't predict.

---

The 5,000-verdict experiment this piece draws from: The battle of the LLM judges. For the reason a single-run measurement isn't enough, do LLM judges give the same verdict twice?. For the templates the pricing is arguing against, Langfuse vs PostHog vs openevals.

Lovelaice runs multi-model judge evaluations on your data in an afternoon. Talk to us.