The AI product whose outputs were judged. An AI agent that qualifies and prioritizes inbound leads. It takes the company's full context, ICP, buyer personas, qualifying and disqualifying signals, existing customers, plus an inbound contact (a name, an email, sometimes a company or a message), researches the person and the company, and assigns a priority tier: Tier 1, immediate outreach, down to Tier 4, disqualify.
The failure mode that human annotations confirmed. The agent's real errors are wrong tiers. We found this through error analysis in Lovelaice: manual annotation of agent responses first, patterns second, judge third. The annotations showed exactly where judgment lives in this product, any contact at a target-ICP company is Tier 1 even when they're not the buyer persona; a contact from your own company is disqualified on sight. The custom judge encodes those rules. That ordering, annotate before you automate, is the methodology argument of this entire piece, compressed into one paragraph.
The contenders. The Lovelaice custom judge against 7 generic templates drawn from popular eval tooling: a Correctness rubric in the openevals style; Langfuse's Hallucination, Helpfulness, and Relevance templates; PostHog's Hallucination and Helpfulness templates; and the important middle ground, custom_posthog, PostHog's generic scaffold with the right one-sentence requirement written in ("assess whether the agent correctly classifies the lead's priority"). Right question, shallow rubric. It exists in this battle to test the most common objection: "I'll just adapt the template."
The 2 runs. We ran the full benchmark twice, on identical settings. That design makes it 2 experiments in one: a cross-vendor test of the generic templates, and a controlled measurement of something almost nobody checks in their eval setup, how many of a judge's verdicts survive running the exact same benchmark again. (Finding 5 has that answer.)
The dataset. 34 real agent responses, human-labeled as passed or failed with written justifications, deliberately balanced: 17 correct, 17 containing a real tier error. On this set, 50% agreement is a coin flip, a judge that passes everything scores exactly 50%, and so does one that fails everything. We tracked 3 numbers per judge: agreement with the human, missed real errors (false negatives), and false alarms on good answers (false positives). Every judge outputs the same JSON: a 0-or-1 score and a written justification. Those justifications matter, they're what makes Finding 3 possible.
One scope note, stated plainly: this is one product, one failure mode, one expert annotator. A case study, not a universal benchmark.
Finding 1: The best judge requires the right prompt × model pairing, and no generic pairing came close
The custom judge's best pairing landed at 76.5% agreement with the human, and it hit that same ceiling in both runs (on gemini-3-flash both times, and on gpt-5.4 in the second run), and in the final run it took the top 8 of all 80 pairings outright. We quote a steady ceiling rather than one lucky record because, as Finding 5 shows, any single pairing's score carries about ±10 points of run-to-run noise. The gap to the generic judges is clear, within a single run, the best custom pairing beat the worst generic pairing by 32 points (76.5% vs 44.1%).
The final run's leaderboard, all 8 judges competing:
That table is a testament to the idea that the best results come from pairing the prompt with the right model for the problem you're solving with AI. This you can only find through experimentation. The unit being ranked is never a prompt and never a model, it's the pairing, because as you'll see in Finding 6, neither half predicts the result on its own. (The best generic pairing, at rank 9, looks almost respectable at 70.6%, hold that thought, because it gets there by rubber-stamping, missing 41% of the real errors, and Finding 3 shows the trigger-happy generics' agreement is a coincidence.)
Most teams pick their LLM judge model upfront, usually based on cost and/or latency. Picking the model first without comparing the performance of different models might cost you half your judge's accuracy. Averaged across all 10 models, the 7 generic templates landed between 52.1% and 58.2% agreement. The baseline on this balanced dataset: 50% is a coin flip.
(Final-run averages on the 10 models; every judge moved less than 2 points between the 2 runs and the ranking didn't change, the averages, unlike single pairings, are stable.)
The staircase across these tables matters more than any single row. A generic template gets you 52–58%, and that now replicates across different vendors, so this is a property of generic, not of any vendor's prompt-writing. A generic template plus the right question, the "I'll just adapt it" strategy, gets you 56.2%, still missing 76% of the real errors. Encoded judgment gets you ~68% on average. Encoded judgment on a validated pairing gets you 76.5%. The judgment lives in the rubric, and the rubric only performs on a pairing you've measured.
And for anyone still planning to adapt the template: custom_posthog's passes get there by quietly reducing its own job to a presence check, it confirms the agent named a tier and calls it correct, without ever testing whether that tier was the right one. The agent picked a tier. Any tier. Pass.
Averages, though, hide the more useful diagnosis. Look at the 2 outer columns of that table and you'll see the generic judges don't fail randomly, they fail in exactly 2 directions.
Finding 2: Every generic judge has one of 2 personalities, and both cost you
All 7 generic judges are cleanly and clearly split into 2 camps, and the split is prompt-determined and reproducible, the same pattern held in our earlier iterations months before, and all the generic judges each snapped straight into one of the 2 camps on arrival.
The personality comes from what the template asks about, not from which vendor wrote the prompt. Helpfulness and relevance questions pass all responses and miss up to 90% of real errors, in every library; hallucination questions over-flag in every library.
Rubber stamps pass nearly everything. PostHog's Helpfulness misses 88.8% of the real errors in this dataset; PostHog Custom judge misses 76.5%. Your actual failures ship while the dashboard stays green, and since "always pass" scores 50% on a balanced set for free, their 52–56% agreement means they add almost no information. You're paying tokens for a yes-machine. Their passes read the same way every time, variations of "accurate, well-grounded, no hallucinations", an endorsement of the facts that never assesses the decision the product exists to make.
Trigger-happy judges fail nearly everything. The Correctness and Hallucination judges flag 53–69% of the good answers. PostHog's Hallucination judge false-flags 68.8%, better than 2 out of 3 correct responses come back marked as failures. A team living with that drowns in false alarms, stops reading the eval, and the whole system dies of distrust.
Which direction hurts more depends on your product. For lead qualification, a missed high-value lead costs far more than a wasted review, which is why the custom judge encodes asymmetric severity instead of treating every mistake equally. That judgment call is itself something no template can make for you.
An unvalidated judge gives you false confidence or noise, and either one kills the scaling promise that justified the judge in the first place.
Both personalities, though, share a stranger trait, and it's the most important finding in this experiment: even when these judges land on the right score, you can't trust why. If you can't trust why, you can't trust them with the production AI quality assessment.
Finding 3: When generic judges agreed with the human score, they were right by accident
For the caught-error cases in each run, agent answers which both the human and the judge marked as failed, we compared the judge's written justification against the human's. The human's reason, every time, is the tiering decision: a perfect Tier-1 bank marked Tier 2, a high-value target over-disqualified to Tier 4. Here is what each judge was actually looking at when it "agreed":
This is a comparison we've never seen an eval benchmark run, and it's only possible because the human labels carry written justifications. When the custom judge fails an answer, it fails it for the reason the human did. When Correctness or the Hallucination judges fail the same answer, they cite a stray future date, a "hallucination," or a formatting glitch, and the PostHog Hallucination judge cited the actual failure mode, the wrong priority tier, in zero of its catches. Every catch fired on something else.
One case shows what that means in practice. The agent received an inbound contact from an @lovelaice.com address, an employee of the seller itself, and instead of disqualifying on sight, treated it as an external prospect. The human graded it a fail, for the obvious reason. The hallucination judge also graded it a fail, but because of:
"substantial fabricated specifics presented as verified facts."
It never noticed the agent was evaluating its own company as a lead.
The consequence deserves spelling out, because it's the entire argument: a judge that's right for the wrong reason today tells you nothing about how it will score tomorrow's case. You can't debug it, you can't improve it, you can't keep it current with your customers' use cases, and the trend line it produces deserves none of the trust your team puts in it. A broken compass that sometimes points north.
Score-level agreement overstates every generic judge. Reason-level agreement is the number that means what you think agreement means. For contrast: the top custom-judge pairings caught errors for the human's actual reason 100% of the time.
You might assume the factuality judges at least earn their keep on actual fabrications. That assumption is what the next finding takes apart.
Finding 4: The hallucination judge hallucinated
The one "real fabrication" the hallucination judges seemed to legitimately catch, a seed-stage startup claiming roughly 1,550 employees, doesn't exist. The agent had written "~15–50 employees." The judges misread the range as a single number and manufactured the hallucination themselves: 44 judge outputs in run one invented the phantom figure, 23 of them failing the answer over a number the agent never wrote, and the re-run repeated it almost exactly, with 40 more outputs doing the same. Every one of the 10 models fell for it in at least one run. Even our own custom judge, paired with a weak model, fell for it.
Here's the part that's super interesting: the mistake wasn't spread evenly across the judges, it concentrated on the exact prompts you'd trust most to catch it.
The 2 prompts whose entire job is to catch hallucinations, the hallucination judges, invented one in 9 or 10 of the 10 models. hallucination (langfuse) hit all 10 on the re-run. Meanwhile the holistic prompts that don't order the model to hunt line-by-line for fabrications mostly read the number correctly: Adapted Generic (PostHog) invented it in only 2 models, our own Lovelaice judge in 3 and then just one on the re-run. The judge you install specifically to catch fabrication is the one most likely to fabricate, because instructing a model to go looking for hallucinations is, in part, instructing it to find them, whether or not they're there.
The judges received "~15–50 employees" exactly as written, and misread it anyway, some even arguing with themselves about it in their justifications, quoting the correct range and the invented number in the same sentence, then failing the answer over the number they'd invented.
And the prompt was only half the story. Change the prompt and a fragile model still fails you: Claude Opus 4.8 misread the same string in 7 of 8 prompts, in both runs, you could not prompt your way out of it. Change the model and the wrong prompt still provokes it, while at the other end, GPT-5.4-nano read the number correctly almost everywhere. So the same figure is a "hallucination" or a non-event depending on which model-and-prompt pair you happened to install, and no single knob fixes it.
A judge that fails answers for errors it invented is worse than a rubber stamp. A rubber stamp merely misses your problems; this sends your team debugging problems that don't exist. The judge you installed to catch hallucinations can be the thing introducing them into your metrics.
Every phantom number above came from a single run of a judge. Which is exactly how teams read their eval dashboards: one run, one score, taken as the truth. The second run of this benchmark measured what that habit is worth.
Finding 5: We ran the identical benchmark twice, 8% of the verdicts flipped
On identical judge-model-case combination, same prompts, byte for byte; same 34 responses; same models and settings, 205 of 2,490 verdicts (8.2%) changed between the 2 runs. The judge said PASS one week and FAIL the next, with nothing changed but the sampling dice, and this is with sampling pinned as low as it goes.
The judges disagree with themselves before they ever get to disagree with the human. And verdict stability turns out to be a model property:
The Claude family and gpt-4.1 are the most self-consistent, gpt-4.1 barely moves at all (0.7%). The small GPTs are the least: one verdict in 6 changes between 2 identical runs. Any single-run number for gpt-5.4-mini is a lot shakier than its average lets on.
The flips are also why this article quotes averages and ranges instead of records. Per-judge averages barely moved between runs, every judge stayed within 2 points, and the ranking didn't change, but individual pairings moved up to ±10 points: our custom judge on gpt-5.4 rose from 61.8% to 76.5% between runs, while on gpt-5.4-mini it fell from 61.8% to 50.0%. On a 34-case set, one flipped verdict is 2.9 points, and an 8% flip rate moves 2–3 verdicts per pairing.
The flip table also foreshadows the next finding: the model under the prompt isn't an implementation detail. It's half the judge.
Finding 6: The model is half the judge
Lovelaice custom judge agreement spans 76.5% down to 50.0% across the 10 models in the final run. Identical instructions, up to a 26-point spread. The same prompt is a different judge on a different model.
Model temperament persists across prompts, too. gpt-4.1 rubber-stamps nearly everything it grades regardless of the prompt; gpt-5.4-mini and nano are the least stable everywhere. And pairings don't transfer: the best judge model from our very first version of the Lovelaice judge prompt (gemini-3.5-flash) rubber-stamps in the latest version in both of these runs. You cannot predict a pairing. You measure it, which means running your judge prompt across models is not a nice-to-have experiment, it's how you avoid donating a quarter of your judge's accuracy to a default model choice.
An important caveat here, because a custom prompt grants no immunity. In an earlier iteration, Lovelaice custom judge on one model scored a healthy-looking agreement while catching errors for the right reason far less often, it had picked up a surface habit from the model, not the rubric. If we had picked our judge model by its agreement score alone, we would have shipped a broken judge with a good number. The moat is the validation loop, not the custom prompt.
The model choice is also where the money conversation gets interesting, because the price tags point the wrong way. Paying more does not buy a better judge.
The best judge in the entire battle runs on a mid-tier gpt at well under a cent a verdict, accuracy came from the rubric, the pairing, and the validation, not from spend. claude-sonnet-5 costs 3 times as much for equal agreement and can be as slow as 20s per verdict. DeepSeek grades just as well here but is the slowest by far, around 19s a verdict on V4, so it's the wrong buy on speed and price even though its accuracy holds up.
And the "at least generics are cheaper to run" argument dies here too. A generic pairing costs the same tokens per verdict as the custom one, but Finding 3 showed its error-catches cite the real failure 0–31% of the time. A judge whose verdicts you can't act on has an infinite cost per useful verdict. It isn't cheap; it's wasted.
6 findings about how judges fail. The last one is about the only thing that made ours better.
Finding 7: The only thing that moved the number was the loop
Between our first iteration and these 2 final runs, exactly one thing changed on our side: the judge prompt, rewritten from what the human annotations taught us. The generic prompts stayed byte-identical throughout. The result:
| Iteration 1 (V1) | Final runs (V4) |
|---|
| Custom judge, average across models | 65.1% | 67.0% / 68.8% |
| Custom judge, best pairing | 72% | 76.5% |
| Generic judges | 51–57% | 52–58% (unchanged) |
V4 added 2 rules straight out of annotation sessions: internal contacts are automatically disqualified, and under-prioritizing a high-value lead is worse than over-prioritizing a low-value one. Several points on the best pairing, from encoding judgment we already had on paper, while 7 generic templates, untouched, stayed exactly where they started.
The loop earned its keep more than once by catching our own bugs. It caught a pairing looking healthy while reasoning nonsense. And it caught our internal-contact rule backfiring on weaker models before we trusted it. A judge you iterate without re-validating is just drift with version numbers.
Transparency requires saying this plainly: our judge was tuned on this same golden set, so a held-out test set is our next step. We can say that comfortably because the thesis of this piece survives the disclosure. "Validate, then iterate" holds either way; accuracy-bragging wouldn't.
A judge is an AI product you build, annotate, encode, validate, iterate, and the loop is the only part of it that generalizes to your product.
What to do with this
The arc of the data, in 5 sentences. The best judge in the battle was a pairing, a custom rubric on a measured model, steady across 2 runs, while 7 generic templates from 3 vendors' libraries averaged within a few points of a coin flip, and adapting the template didn't fix it. Their failures come in 2 personalities, false confidence or noise, and even their correct verdicts were right for the wrong reason. The model contributes as much as the prompt, no pairing can be predicted, only measured, and 8% of all verdicts flip on a re-run, so a single score is not guaranteed. The only thing that moved agreement was the annotate–encode–validate–iterate loop.
Your LLM judge is the second AI product you ship. It has a prompt, a model, failure modes, and opinions you didn't put there, and it deserves the same rigor as the feature it grades. What that means depends on where you are:
If you're currently building your eval suite for your AI product and your vendor provides LLM judge templates: don't start there. Start with manual annotation, review 30 real outputs yourself, and leave written feedback on the failures. The instructions are where your judgment lives, and Adapted Generic Judge is the proof that even the right question with a shallow rubric stays at a coin flip. The failure patterns uncovered from human annotations are the raw material the template can't contain.
If you're already running a judge in production: validate it this week. Take 30 real outputs, half of them containing known errors, a judge tested only on happy cases looks great right up until it rubber-stamps a failure. Label them yourself, then measure agreement, missed errors, and false alarms. Then read the justifications, not just the scores: every discovery in this piece, the wrong-reason agreements, the judges' own hallucinations, lives in the justifications and is invisible at score level.
If you've already built a custom judge: you're halfway. Measure the prompt × model pair, ours swung up to 26 points across models with identical instructions, and check that the pairing agrees with itself: run your golden set twice and count the flips, because some models change one verdict in 6 between runs. Then re-validate after every prompt change. With the right tool in place, this can be done in under an hour; there is no budget that can't afford it.
The boundary of this case study: it says nothing about guardrail-style checks, toxicity, PII, safety, which are a separate question this experiment didn't set out to settle. What it says is narrower and sharper: a generic judge cannot measure whether your AI did its job in your product, for your users.
A judge you haven't validated against your own judgment isn't a judge. It's a random number generator for which you pay real dollars.
Madalina
Madalina Turlea
Catalina Turlea