How to validate an LLM judge before you trust it

The annotate → encode → validate → iterate loop is the only thing that moved our judge's agreement with a human expert from 65.1% to 76.5%. This is the how-to, one step at a time, with the traps we hit written in.

You've decided to run an LLM as a judge for your AI feature. You've either written a custom prompt or you've picked a template from your observability vendor. Either way, before it's trusted to grade what ships to your customers, the judge itself has to be graded. The question is how.

We ran a large-scale experiment on this (8 judges, 10 models, 34 real AI outputs with expert human labels, over 5,000 verdicts, run twice, written up in full here). Across four iterations of our custom judge and every generic template we tested, exactly one process moved a judge's agreement with the human: the four-step loop below.

Not the model choice, not the prompt in isolation, not which vendor's tooling we ran it through. The loop was the variable that mattered.

This piece walks the four steps as a repeatable workflow you can run on your own judge in an afternoon.

Before the loop: what a "judge you trust" actually means

A judge you trust does three things, not one:

1. •

   Agrees with the human at a rate you can defend. For most product decisions, the useful floor is well above the 50% coin-flip. Our best pairings landed at 76.5%. The seven generic templates we tested averaged 52–58%. Anything under 60% is not a measurement; it's a random number generator with a rubric attached.

2. •

   Agrees for the right reason. When our custom judge caught an error, it named the same failure the human named ~100% of the time. When the generic Correctness and Hallucination templates caught the same error, they named the real failure 0–31% of the time. A judge that scores well by pointing at the wrong thing tells you nothing you can act on tomorrow.

3. •

   Agrees with itself on a re-run. We re-ran the identical benchmark and 8.2% of verdicts flipped. Some models flipped 14–17% (more in do LLM judges give the same verdict twice?). A judge whose verdicts change with the sampling dice is not a judge, it's a mood.

The loop below is what gets you all three at the same time.

Step 1. Annotate: label 30 real outputs yourself

Take 30 real outputs from your AI feature: production outputs sampled to include the failure modes you already know about and the ones you suspect, not the synthetic examples or happy-path cases you tested during the build.

Split the set: roughly half correct, half containing a real error. That balance matters. A set of 27 correct answers and 3 known failures rewards a judge for rubber-stamping (pass everything, and it scores 90%). The balanced set punishes both directions of failure and forces the judge to actually discriminate.

For each output, write two things:

• •A verdict. Pass or fail. Nothing in between.
• •A one-sentence justification. The real reason. Not "looks off" or "wrong tone." The specific failure. "Classified this Tier 3 when it should be Tier 1, target ICP company, decision-maker persona." Or, for a pass: "Correctly identified the internal @company.com email and disqualified without further processing."

The justifications are the real deliverable of this step. They are how you'll check, later, whether the judge agrees with you for the right reason or by accident.

The trap. Skipping the labels because they're slow. 30 careful annotations is one to three hours of focused work, and it is the load-bearing hour of the entire validation process. Every generic-template failure we measured traces back to a team that didn't do this step.

Done looks like. A spreadsheet with three columns: the output, your verdict, your one-sentence reason. Numbered 1 to 30. Balanced ~50/50 on the verdicts.

Step 2. Encode: write the judgment into a prompt

Read your 30 justifications back. The failure patterns should now be obvious to you in a way they weren't when you started. Rules will start writing themselves. "Internal-domain email addresses are always disqualified." "Under-prioritizing a high-value lead is worse than over-prioritizing a low-value one."

Encode those rules directly into the judge prompt.

Not as an abstract rubric. As specific instructions the model can execute. The generic templates we tested asked questions like "Is the response helpful?" and averaged 55%. Our custom prompt names the decision the product exists to make and the rules that decision has to satisfy. Same task. Twenty-plus points of difference.

Between our first version of the judge prompt and our final version, exactly two rules were added, both pulled straight from annotation sessions: the internal-domain disqualification, and the asymmetric-severity rule for under-prioritization. The best pairing went from 72% to 76.5%. Multiple points of accuracy, for two lines of encoded judgment.

The trap. Writing the prompt from your head, without reading the annotation justifications. The whole point of Step 1 is to force you to notice what you actually judge on. If you skip that reading and write the prompt on gut feel, you end up with a version of the generic templates: right question, shallow rubric. Ours failed the same way on an early iteration until we caught it.

Done looks like. A prompt whose rubric names the specific failure modes from your annotations, in the language your team uses in review meetings. Not vendor-generic language. Yours.

Step 3. Validate: measure the pairing, not just the prompt

Now measure the encoded judge against the 30 annotations. Three numbers per pairing:

• •Agreement. How often the judge's verdict matches yours.
• •Missed real errors. How often the judge passed a response you failed. (Rubber-stamping.)
• •False alarms. How often the judge failed a response you passed. (Trigger-happy.)

The critical part: do this across models, not one. The same prompt scored 76.5% on one model and 50.0% on another in the same run on our data. The unit you're validating is the prompt × model pair, not the prompt.

At minimum, run the prompt across three models from different bands: one low-cost, one mid, one premium. The winner is rarely the premium one. On our data, the best pairing was a mid-tier Gemini at well under a cent per verdict. See which model should you run your LLM judge on? for the full model breakdown.

Then, the step almost no one runs, run the top pairing a second time on identical inputs. Count how many verdicts changed. If more than 5% flip, the model isn't stable enough for the judgments you're about to build a dashboard on. Drop it and use the runner-up.

The critical part everyone skips: read the justifications. For every error the judge caught, compare its written justification to yours. If the numbers match but the reasons don't, the judge is right by accident. Our custom judge caught errors for the human's actual reason ~100% of the time on the best pairings; the generic templates cited the real failure 0–31% of the time. A judge that scores 70% by pointing at the wrong things will drift the moment your traffic changes. A judge that scores 70% for the right reasons will hold.

The trap. Reading the score, not the justifications. Every finding in our experiment worth knowing (the wrong-reason agreements, the invented hallucinations, the rubber-stamps that looked competent) lives in the judge's written justifications and is invisible at score level. If you're only looking at the number, you're grading the wrapper.

Done looks like. A shortlist of prompt × model pairings, each with agreement / miss / false-alarm numbers, run twice, plus a reason-level check on the caught errors of the top two.

Step 4. Iterate: change one thing, re-validate all three numbers

Now you edit the prompt. Or swap the model. Or both. And you re-run every measurement above.

Non-trivial prompt edits invalidate previous pairings. In our experiment, the best model on version 1 of our prompt rubber-stamps on version 4 (same model, same rubric shape, different specifics). The pairing is not a piece of infrastructure you inherit. Every prompt change gets its own measurement across models.

The iteration is where the number moves. Between our first iteration and the final one, we edited the prompt to encode two rules from annotation. Best pairing improved 4.5 points. Average across models improved a few points. The seven generic templates we ran alongside stayed byte-identical throughout and moved less than a point.

A judge you iterate without re-validating is drift with version numbers.

The trap. Editing the prompt to fix a case the judge got wrong, without running the whole 30-case set again. You'll fix the one case and quietly break three others. Every prompt change re-runs the full validation, or the "improvement" is not measured.

Done looks like. A judge whose agreement, miss rate, false-alarm rate, and reproducibility all pass the thresholds you set at the start. And a git-tracked history of prompt versions, each with the pairing measurements attached, so you can roll back if the next iteration doesn't hold.

What the loop actually costs

One afternoon of annotation. One afternoon of prompt writing and measurement. A quarter-hour to re-validate on each subsequent prompt change. With the right tooling, the whole cycle runs in under an hour end-to-end after the first pass.

That's the cost. What you get back is a judge whose verdicts your team will actually act on, and a documented reason for every score it produces. Which is the thing every team said they wanted when they installed the observability vendor's templates and got neither.

The loop as the actual moat

The temptation, once you have a custom judge that works, is to treat the prompt as the asset. It isn't.

The prompt is the product of the loop. The loop is the moat.

Every AI feature you ship deserves this. The judge grading it is the second AI product you own. Give it the same rigor.

---

The 5,000-verdict experiment behind every number in this piece: The battle of the LLM judges. For why the model choice is half the validation, which model should you run your LLM judge on?. For why one measurement is never enough, do LLM judges give the same verdict twice?.

If you'd like to run this loop on your own AI feature and get a validated judge out the other side in a session, that's what Lovelaice does. Talk to us.