Once you have more test cases than a person can read by hand, you reach for automation. The appealing option is to have a second model grade the first one — LLM-as-judge. Give a model the input, the output, and a rubric, and ask it to score whether the answer is good.
It works well enough that it's become the default way teams scale evaluation, and it also fails quietly enough that plenty of teams are trusting scores that don't mean what they think. The useful question is whether your judge is reliable for your task, which is something you can actually measure.
When a judge earns trust, and when it doesn't
Some checks don't need a judge at all. If you can verify an output with code — a date is in the right format, a required field is present, a number matches a source value — use a deterministic check. It's cheaper, faster, and it can't have an opinion. Save the LLM judge for the things code can't assess: is this summary faithful to the document, is this answer actually responsive to the question, is the tone appropriate for a regulated context.
The mistake is reaching for a judge on a task that has a precise right answer, then debugging the judge's mood instead of writing four lines of code.
The blind-spot problem
A judge built from the same model family as the system it grades shares that family's blind spots. If the model is bad at a certain kind of reasoning, it will also be bad at noticing when it got that reasoning wrong. The judge nods along to the exact errors you most need it to catch. This is why a high score from an LLM judge is, on its own, evidence that the judge approves — which is a different thing from evidence that the feature works.
How to validate the judge
You validate a judge the same way you'd validate a new human grader: check its agreement against a trusted standard.
- •Take a sample of outputs and have a domain expert score them by hand. This is your ground truth.
- •Run the LLM judge on the same sample.
- •Compare. Where the judge agrees with the expert, you can trust it to scale. Where it diverges, you've found a category the judge can't be trusted on yet.
- •Tighten the rubric on the cases it got wrong, and re-check. A vague rubric produces a vague judge; specificity is most of the fix.
The point is that the judge is itself a thing you evaluate before you rely on it. You don't have to trust it on faith, and you shouldn't.
The role the expert never gives up
LLM-as-judge doesn't remove the domain expert from the loop. It changes what they spend time on. Instead of reading every output, the expert defines what good looks like, validates the judge against a sample, and audits the disagreements. The judge handles volume; the expert owns the standard. That division is what lets a small team evaluate hundreds of cases without pretending a model's self-approval is the same as quality.
Used this way, a judge is one of the most useful tools you have. Used on faith, it's a confident number with nothing behind it. The difference is whether you checked.
Want to learn how to design and validate your own LLM judge from scratch? Our Maven course, Ship AI features with confidence for PMs, covers the full workflow.
Related reading: LLM as a judge: How to evaluate AI features without checking every answer by hand · The regression trap
Enjoyed reading? Add us as a preferred source in Google to support us.
Add Lovelaice as a preferred source

