We ran the generic LLM judge templates from all three tools against 34 human-labeled AI responses, across 10 models, twice. Same dataset, same protocol, three vendors. This is what we found.
Every AI observability vendor now ships a set of "quality judge" templates. Helpfulness. Relevance. Correctness. Hallucination. Drop them into your pipeline and get scores back. Product teams adopt them because they arrive already written, they look like measurement, and nobody wants to be the person who couldn't produce an accuracy number for the AI feature.
We wanted to know which vendor's templates were the strongest. So we put them side by side.
Over 5,000 verdicts later, the answer is that the vendor barely matters. All three vendors' generic templates cluster inside a 6-point band, well within run-to-run noise. And every one of them fails the same way, in one of two directions, determined by what the template asks, not by which vendor wrote it.
The vendor is not the variable. The rubric is.
The full experiment (8 judges, 10 models, 2 runs, 34 real agent responses) is written up in the battle of the LLM judges. This piece pulls the vendor-vs-vendor slice out of it, because the comparison keeps coming up in evaluation-tool decisions and we've never seen anyone publish the numbers.
The setup, in one paragraph
The product under test was an AI agent that qualifies and prioritizes inbound leads into four tiers. We took 34 real agent responses, 17 correct and 17 containing a real tiering error, all labeled by a human expert with a written justification for each verdict. That balance is deliberate: on this dataset, a judge that passes everything scores 50%, and so does a judge that fails everything. Any real signal has to beat 50%.
We ran seven generic templates against that set, on 10 different models each, twice:
- •openevals (Arize): the Correctness rubric
- •Langfuse: Hallucination, Helpfulness, Relevance
- •PostHog: Hallucination, Helpfulness, plus one custom variant using PostHog's scaffold with the correct one-sentence requirement written in
All seven were left byte-identical between runs. We tracked three numbers per judge: agreement with the human, missed real errors (false negatives), and false alarms on correct answers (false positives).
The numbers side by side
Averaged across all 10 models in the final run:
| Judge template | Vendor | Agreement | Missed real errors | False alarms |
|---|---|---|---|---|
| Helpfulness | PostHog | 52.4% | 88.8% | 6.5% |
| Helpfulness | Langfuse | 55.3% | 82.4% | 7.1% |
| Relevance | Langfuse | 55.9% | 80.6% | 7.6% |
| Custom PostHog scaffold | PostHog | 56.2% | 76.5% | 11.2% |
| Hallucination | Langfuse | 56.8% | 41.2% | 45.3% |
| Correctness | openevals | 57.6% | 32.9% | 51.8% |
| Hallucination | PostHog | 58.2% | 27.6% | 55.9% |
The full band, from the weakest generic (PostHog Helpfulness at 52.4%) to the strongest (PostHog Hallucination at 58.2%), is 5.8 points. For context, the same benchmark run twice on identical settings moved individual pairings by ±10 points on run-to-run noise alone. The gap between vendors is smaller than the noise floor.
The custom judge we built for this product from human error analysis landed at 68.8% averaged, 76.5% on its best model pairing. That gap, generic to custom, ran to 32 points at the extremes inside a single run. Between vendors of generic templates, the gap ran to 6.
The vendor is not the variable.
Two personalities, three vendors, same failure
The false-negative and false-alarm columns above show what the average score hides. Every one of these judges falls into one of two personalities, and the split is prompt-determined:
Rubber stamps: Helpfulness in both libraries, plus the custom PostHog scaffold. They pass almost everything.
- •PostHog Helpfulness missed 88.8% of the real errors in this dataset
- •Langfuse Helpfulness missed 82.4%
- •Custom PostHog scaffold missed 76.5%
The "I'll just adapt the template" variant, with the correct question written in, quietly reduced its job to a presence check: the agent picked a tier, so pass. Any tier.
Their agreement scores land in the low 50s because "always pass" scores 50% on a balanced set for free. The template adds barely two to six points of signal over a coin flip, at the cost of the tokens you spent to run it.
Their justifications read the same way every time, variations of "accurate, well-grounded, no hallucinations", an endorsement of the facts that never assesses the decision the product exists to make.
Trigger-happy judges: Correctness (openevals), Hallucination in both libraries. They flag more than half of the correct answers.
- •PostHog Hallucination false-alarmed on 55.9% of the good answers
- •openevals Correctness false-alarmed on 51.8%
- •Langfuse Hallucination flagged 45.3%
A team running one of these in production sees more alerts than real failures, learns to ignore the eval, and the whole system dies of distrust.
The personality tracks the template's question, not the vendor that wrote it. Helpfulness templates rubber-stamp in every library we tested, and Hallucination templates over-flag in every library. The same pattern held in earlier iterations of our experiment months before, and every generic judge snapped straight into one of the two camps on arrival.
The catches were right for the wrong reason
For the errors each judge did catch, we compared the judge's written justification against the human's. The human's reason was always the tiering decision: a target-ICP contact marked Tier 3, an internal @lovelaice.com email treated as an external prospect.
When our custom judge caught an error, it named the tier mistake ~100% of the time.
When openevals Correctness or the Hallucination judges caught the same error, they named the tier mistake 0–31% of the time. Every other catch fired on something else: a stray future date, a formatting quirk, a "fabrication" the agent never wrote.
PostHog's Hallucination judge cited the real failure mode, the wrong tier, in zero of its catches.
A judge that agrees with the human today for reasons unrelated to the human's actual concern tells you nothing about tomorrow's cases. The trend line it produces deserves none of the trust your team places in it.
Score-level agreement overstates every generic judge on the market. Reason-level agreement (reading the judge's justifications against the human's) is the number that means what you thought agreement meant.
What actually varies between vendors
Not accuracy, and not the failure mode. What varies is the packaging: what the judge prompt looks like, how much of it you can see, how easy it is to fork.
Langfuse's templates are the most opinionated and the most verbose. PostHog's are shorter and easier to adapt, which is why we tested a "custom PostHog" variant (write in your one-sentence requirement, keep the scaffold). That variant scored 56.2%. Adapting the template is not the fix.
openevals ships the tightest Correctness rubric of the three, and it's the closest to what a team would write on their first try if they built one themselves. It also over-flags 52% of the good answers.
If you're picking an eval tool for reasons unrelated to the judge templates (the traces, the pricing, the integrations, the UI), that's a real decision, and any of the three will serve. If you're picking one because you want a judge you can trust straight out of the box, none of them earns it.
What to do instead
The generic judges are not broken. They're generic. They ask questions that fit any AI product, and the price of fitting any product is measuring none of them well. Every failure mode above traces back to the same root: nobody encoded what "good" looks like for the product being graded.
If you want a judge you can act on:
- •Start with 30 real outputs from your product, half of them containing real errors you've labeled yourself.
- •Write down why each failure is a failure, in the language your team uses in review meetings.
- •Turn that written justification into the judge prompt: the specific rubric your team would apply in a review meeting, not the generic Helpfulness or Correctness rubric your observability tool ships with.
- •Measure the prompt × model pair. Don't pick a model on cost or latency alone. In our experiment, the same custom judge prompt scored 76.5% on one model and 50.0% on another.
- •Run it twice. If a verdict flips on identical inputs, you're seeing model instability, not signal. Some models flipped 14–17% of verdicts on a re-run at the lowest possible temperature. (More on that in do LLM judges give the same verdict twice?.)
Every step above is what turned a 65% average judge into a 76.5% one in our data. The generic templates from every vendor stayed exactly where they started, byte-identical, across every version we tried.
The comparison, plainly
Langfuse, PostHog and openevals ship generic judge templates that land inside a 6-point band on a balanced human-labeled dataset. All three fail in one of two directions determined by the template's question. All three catch errors for reasons unrelated to the human's judgment 69–100% of the time on our data.
They are useful for what they are: interchangeable priors you can run on day one to prove your pipeline works. They are not a substitute for a judge built from your product's actual failure modes.
The vendor is not the variable. The rubric is.
The 5,000-verdict experiment this piece pulls from is written up in The battle of the LLM judges: 8 judges, 10 models, 2 runs, human labels, all data shown.
At Lovelaice we help product teams build judges you can trust for the AI features you've shipped, on the data those features actually see. Talk to us.
