Twelve checks before you trust an AI to evaluate another AI.
A few months ago I sat in on an AI evals workshop. A big e-commerce team walked through the LLM judges they were using on their shopping assistant: relevancy, helpfulness, hallucination — all shipped by their observability vendor. So I opened the chatbot and tried one conversation. It dropped a constraint every turn, told me the platform did not carry a product it did carry, and burned 11 turns narrating a search it never ran. By any reasonable standard, the conversation failed. The dashboard was green.
So we staged a battle: 8 judges, 10 models, 34 real agent responses, 5,000+ verdicts, run twice. The best judge — a custom rubric built from human annotations — hit 76.5% agreement with the human expert and took the top 8 of all 80 pairings. The seven generic templates, from Langfuse, PostHog and openevals, averaged 52–58%, on a set where flipping a coin scores 50%. The 12 checks below are what separates the judge that swept the leaderboard from the ones that landed on the coin flip.
1. Start with real outputs
Do not build the rubric from imagined failures. Review outputs from your real prompt and use case.
The e-commerce team's generic judges (helpfulness, relevancy, hallucination) all passed the failing conversation, because they had never seen what failure looked like in that product. The custom judge in this battle was built from manual annotation of real agent responses — patterns first, judge second. That ordering is the entire methodology argument, compressed: annotate before you automate.
2. Create a failure taxonomy
Annotate why outputs fail and group repeated issues into independent failure patterns.
For the lead-qualification agent tested here, the failure mode was wrong tier, and the annotations surfaced two non-obvious rules: any contact at a target-ICP company is Tier 1 even if they are not the buyer persona, and a contact from your own company is disqualified on sight. Those rules never came out of a template. They came out of reading real responses and asking "what did the human actually decide, and why?"
3. Use deterministic checks first
Handle format, structure, required fields, length, language and other objective rules in code.
Every judge in the battle outputs the same JSON shape: a 0-or-1 score and a written justification. Code enforces the shape. The LLM only decides the verdict and explains it. If a "quality metric" can be a regex or a JSON-schema check, it does not need a model behind it, and running it in code is faster, cheaper, and non-flippy.
4. Pick one subjective failure
Choose a failure that requires domain judgment and matters enough to evaluate at scale.
The subjective failure here was tier assignment. It is exactly the kind of question no template can answer for you: any composite "is this a good response?" judge collapses several failure modes into one score and loses the signal on the one that matters. One agent, one failure pattern, one verdict is not a stylistic choice — it is what makes the next four checks measurable at all.
5. Give the judge one job
One agent, one failure pattern, one verdict. Avoid composite quality rubrics.
The generic templates in the battle broke this rule and paid for it. Adapted Generic (posthog) — PostHog's scaffold with the correct one-sentence requirement ("assess whether the agent correctly classifies the lead's priority") written in — scored 56.2% on average, and its "passes" quietly reduced its own job to a presence check: it confirmed the agent named a tier and called it correct, without ever testing whether the tier was right. Right question, shallow rubric, coin-flip result.
6. Use pass/fail plus justification
Binary labels make human agreement measurable. The justification exposes reasoning gaps.
The single most important discovery in this experiment — that generic judges are "right for the wrong reason" — is only visible in the justifications. At score level, a generic and a custom judge can post the same agreement number. At reason level, one names the tier error and the other cites a stray future date. Every finding in this piece that changed our minds lives in the written justifications, and is invisible if you look at scores alone.
7. Build an expert-labeled dataset
Include both passing and failing examples, edge cases and adversarial cases. Keep the classes balanced enough to expose an always-pass or always-fail judge.
The dataset here was 34 real agent responses, human-labeled with written justifications, deliberately balanced: 17 correct, 17 containing a real tier error. On a balanced set, 50% agreement is a coin flip — a judge that passes everything scores exactly 50%, and so does one that fails everything. That balance is what unmasked the rubber-stamp templates: their impressive-looking 55% collapses the moment you notice it is only five points above "always pass."
8. Benchmark prompt × model pairs
Do not assume the most expensive or newest model is the best judge for this task.
The custom judge's best pairing landed at 76.5% agreement. The same prompt on a different model scored 50.0% in the same run — a 26-point spread on identical instructions. In the final run, the custom judge took the top 8 of all 80 pairings outright; the best generic pairing landed at rank 9.

The unit being ranked is never a prompt and never a model — it is the pairing. Neither half predicts the result on its own. Pick your judge model upfront without comparing, and you can hand away half your judge's accuracy to a default. And paying more does not buy you a better judge: the best pairing in the battle runs on a mid-tier Gemini for well under a cent a verdict; claude-sonnet-5 costs 3× as much for equal agreement.
9. Track both error directions
Measure good responses wrongly failed and real errors wrongly passed separately. They create different product risks.
Every generic judge in the battle fell cleanly into one of two failure personalities — and the personality came from what the template asks about, not from which vendor wrote it.

Rubber stamps pass nearly everything. PostHog's Helpfulness misses 88.8% of the real errors in this dataset on average across the 10 models. Your actual failures ship while the dashboard stays green. Trigger-happy judges fail nearly everything: the Correctness and Hallucination judges flag 53–69% of the good answers on average. A team living with that drowns in false alarms, stops reading the eval, and the whole system dies of distrust. Which direction hurts more depends on your product, but a single "agreement %" that averages the two hides which one you have.
10. Audit the reasoning
Check whether the judge identifies the same failure and uses the same reasoning as your experts.
When the custom judge caught an error it named the real failure ~100% of the time. When the generic Correctness and Hallucination judges caught the same error, they named it 0–31% of the time — and the PostHog Hallucination judge cited the actual failure mode in zero of its catches.

One case: the agent received an inbound from an @lovelaice.com address — an employee of the seller itself — and treated it as an external prospect instead of disqualifying on sight. The human failed it for the obvious reason. The hallucination judge failed it too, but because of "substantial fabricated specifics presented as verified facts." It never noticed the agent was evaluating its own company as a lead. A judge that is right for the wrong reason today tells you nothing about how it will score tomorrow's case.
11. Test stability
Run the same cases more than once. Measure how often the verdict changes.
Same judges, same models, same 34 responses, same settings: 195 of 2,480 verdicts (7.9%) flipped between the two runs. And verdict stability turns out to be a model property, not a prompt property.

gpt-4.1, Claude Opus and Claude Haiku flip under 5% of the time; Claude Sonnet 5 about 10%. The small GPTs flip 14–17%. Any single-run number on gpt-5.4-mini is a lot shakier than its average lets on. If you have not measured your judge against itself, you do not know how much of what your dashboard says today survives to next week.
12. Revalidate every material change
Treat a changed prompt, model, rubric or product behavior as a new judge version that must earn trust again.
Between the first iteration of the custom judge and the final runs, one thing changed: two rules pulled straight out of new annotation sessions (internal contacts auto-disqualified; under-prioritizing a high-value lead is worse than over-prioritizing a low-value one). Average agreement moved from 65.1% to 68.8%; the best pairing moved from 72% to 76.5%. The seven generic templates, untouched, stayed exactly where they started at 52–58%.
The loop also earned its keep by catching our own bugs — a pairing that looked healthy while its reasoning was nonsense, an internal-contact rule that backfired on weaker models before we trusted it in production. A judge you iterate without re-validating is just drift with version numbers.
The score is not the deliverable. A generic judge can post the same agreement number as a purpose-built one and still be useless, because it landed on the right verdict for the wrong reason. No template from any vendor grades your product the way your experts would, straight out of the box. A judge is an AI product like any other, and you make it trustworthy the same way you make any AI feature trustworthy: run it, read why it scored what it scored, find where its reasoning drifts from yours, tighten the instructions, run it again — across multiple models, not a single one. That loop is the work.
Where to start, depending on where you are
Building your eval suite and your vendor provides LLM judge templates. Do not start there. Review 30 real outputs yourself, leave written feedback on the failures, and let the failure taxonomy drive the rubric. Adapted Generic is the proof that even the right question with a shallow rubric stays at a coin flip.
Already running a judge in production. Validate it this week. Take 30 real outputs, half of them containing known errors — a judge tested only on happy cases looks great right up until it rubber-stamps a failure. Measure agreement, missed errors, and false alarms separately. Then read the justifications, not just the scores.
Already built a custom judge. You are halfway. Measure the prompt × model pair (this one swung up to 26 points across models with identical instructions), then run your golden set twice and count the flips. Re-validate after every prompt change. With the right tooling in place this takes under an hour — there is no budget that cannot afford it.
A judge you have not validated against your own judgment is not a judge. It is a random number generator you pay real dollars for.
Want to build the entire workflow hands-on? Join the AI Feature Validation workshop, read the full battle of the LLM judges benchmark, or start validating your judge in Lovelaice.
Enjoyed reading? Add us as a preferred source in Google to support us.
Add Lovelaice as a preferred source


