Braintrust. The Factuality scorer requires you to supply an expert answer as expected, then picks one of five lettered options. Option A — "the submitted answer is a subset of the expert answer and is fully consistent with it" — scores 0.4 out of 1.0. A response that omits the actual answer to the question lands nearly halfway up the scale. Braintrust ships no coherence and no helpfulness scorer.
Arize Phoenix. Faithfulness is a binary single-token classifier, faithful or unfaithful. Its relevance evaluator is DocumentRelevance, which scores whether a retrieved document can answer the question. That is retrieval relevance, not answer relevance. Its Correctness rubric folds five different things into one binary label — factual accuracy, completeness, logical consistency, terminology and ambiguity — so when it returns "incorrect" you cannot tell which of the five it meant, and you cannot act on it.
Confident AI / DeepEval. The Hallucination prompt instructs the judge to "FORGIVE cases where the actual output is lacking in detail" and to "ONLY provide a 'no' answer if IT IS A CONTRADICTION." Faithfulness adds that hedged language such as "may have" or "possibility" does not count as a contradiction, and routes unsupported claims to an idk verdict rather than a failure. Answer Relevancy scores each statement for whether it addresses the question, which means a confidently wrong answer that stays on topic scores 1.0.
LangSmith / openevals. The Answer Relevance prompt tells the judge to focus on whether a statement addresses the question, "not whether the statement is true or well-written." The RAG Helpfulness prompt goes further: a helpful response "may contradict your built-in knowledge but still be correct for the given context." The RAG Groundedness prompt repeats it — stay true to the context "even if context conflicts with common knowledge." These are coherent design decisions for a grounding metric. They also mean a green helpfulness score carries no claim about the answer being right.
Galileo. The largest catalogue in the survey, 37 metrics in total across eight categories, with two computation tiers. The cheap tier, Luna, runs small proprietary evaluation models and returns no explanation. The expensive tier, Plus, runs Chainpoll on a frontier model and does return written reasoning. If you are on the cheap tier you get a score with nothing behind it, which means the one check that separates a working judge from a lucky one — reading why it failed the response — is not available to you.
PostHog. The Hallucination rubric says that if the response "makes specific claims that seem plausible and well-grounded, return true." The Helpfulness rubric says that if the response "attempts to assist but provides limited or unclear guidance, return true." The data model is boolean-only, there is no first-class field for retrieved context, and the docs recommend sampling 5–10% of traffic. That is a monitoring posture — catch drift, spot-check quality — rather than a release gate that regression-tests every change against a fixed set.
Seven structural problems with default judge templates
The individual quirks above matter less than the pattern underneath them.
1. Most templates never see your context. All therefore grade against the judge model's world knowledge. If your product answers from your documents, the metric named after your biggest risk is not looking at your documents.
2. Every template bakes in assumptions you never chose. To write a general-purpose rubric, a vendor has to decide what counts as wrong — and that decision now sits inside your evaluation whether or not it matches your standards. DeepEval's hallucination prompt, for instance, treats only an outright contradiction as a failure and explicitly forgives missing detail. That is a defensible position, and it may not be yours. In your product the expensive failure might be the incomplete answer, or the hedged one, or the one that is technically supported by the context and still useless to the user. What "correct" means is a product decision, and a template quietly makes it for you before your team has had that conversation.
3. The same metric name means different things in different tools. DeepEval's hallucination judge forgives thin detail and hedged language. The openevals rubric requires "perfect accuracy in dates, numbers, and specific details." Identical metric name, opposite bar. Move from one vendor to the other and your pass rate moves with nothing changed in your product.
4. Each vendor makes its own design choices. Beyond the definitions, tools differ on the mechanics: whether a score is binary, continuous or a set of discrete options; whether the judge grades the whole response, each sentence, or each individual claim; and which end of the scale means good. Langfuse's Hallucination scores 1 for a complete hallucination, so there 1 is the worst outcome, while Phoenix's Faithfulness and Galileo's Context Adherence score 1 for the best. None of those choices is wrong on its own — each follows from what the tool was built to do. They do mean a score carries no meaning until you know which convention produced it, and that comparing numbers across two tools requires holding both conventions in your head.
5. Relevance and helpfulness are decoupled from truth by instruction, not by accident. Both openevals and DeepEval tell the judge to ignore whether the statement is true. That is defensible for a metric that isolates one dimension. It is dangerous as a release gate, because the dashboard says relevance 94% and the number contains no information about correctness.
6. Both kinds of template lead back to annotation. Templates come in two shapes, and neither escapes the work.
Most are reference-free: they need an input and an output, and nothing else. No expected answer, no human label. That is why they install in minutes, and it is also why they stay generic — with nothing of yours to compare against, they can only tell you whether an answer looked right.
The rest ask you to supply the expected answer yourself. Braintrust's Factuality needs an expert response for every test case. That is presented as a quick setup field, which makes it look like the rigorous option is also an easy one. Writing a correct expert answer for every case is the same work as annotating your own outputs — and annotation gives you more back, because it also tells you which errors your product actually makes. An expected-answer field never will.
The fast option cannot see your product. The thorough option asks for the annotation work anyway. Either route ends up where a custom judge starts.
7. A single binary hides which thing failed. When a template collapses five criteria into one pass/fail, a red mark tells you something is wrong and nothing about what. You cannot write a prompt fix from that, so the eval produces a number instead of a next action.
What a response that passes every generic metric looks like
I had one conversation with a large retailer's shopping assistant and hit three failures in a single session. Every one of them clears a dashboard built out of default templates.
It sent me elsewhere for a product that was on the shelf. I asked for an item. The assistant told me they didn't carry it and suggested I look somewhere else. They did carry it — it was in the catalogue, in stock, two clicks away. Read that answer as a generic judge and it looks flawless: fluent, directly on topic, and inventing nothing, because "we don't stock that" fabricates no facts. Relevance passes because the answer addresses the question. Helpfulness rubric passes anything that "attempts to assist," and redirecting a customer is an attempt to assist. That is a lost sale, scored green on every axis.
It ran a web search for no reason. Mid-conversation, for a question the product catalogue already answered. No response-quality judge can see this, because they all grade the final text and never look at the path taken to produce it — the wasted latency and the token bill are invisible to hallucination, relevance and helpfulness alike. There is a second-order problem too: once the assistant pulls facts in from the open web, a judge scoring faithfulness against your retrieved context has no way to know the claims came from outside it.
My constraints lasted about one turn each. I stated what I wanted, the assistant held it for a turn, then quietly lost it and recommended things I had already ruled out — and this repeated for the rest of the conversation. Read any single turn in isolation and it is relevant, helpful and grounded. The failure only exists across turns, and generic judges score one input-output pair at a time, so per-turn scoring averages straight to green. A few conversation-level judges exist in the catalogues, but none of them measures whether a constraint the user stated survived the conversation.
Those three failures are blind in one important way: they are too generic and broad, for the specific attributes of quality the specific product needs.
What is a custom LLM judge?
A custom LLM judge is one whose rules come out of your own annotated failures rather than out of a vendor's dictionary. It has four properties that a template structurally cannot have.
It targets one failure mode. One judge, one error, one binary score, one justification. "Is this specific error present in this response?" is a question a model can answer consistently. "Is this a good answer?" is not.
Its rules are traceable. Every rule in the prompt exists because a human marked a real output as wrong and wrote down why. If you cannot point from a rule back to an annotated failure, the rule is a guess.
It encodes your severity trade-offs. Which error is worse in your business is a decision only your team can make, and it belongs in the judge prompt.
It has a known agreement rate. You measured it against human labels, so you know how often it is right and in which direction it is wrong.
In one example, Lovelaice custom built judge for this benchmark ran 7,950 characters against 449–1,388 for the generic templates, and the difference is not verbosity. It is eight explicit rules, each traceable to a real annotated failure — including one stating that an internal employee of the seller is an automatic Tier 4, and one stating that under-prioritising a high-value lead is a worse error than over-prioritising a low-value one. No template can make that severity call for you, because it is a decision about your funnel.
Can you just adapt a generic template?
This is the obvious objection, so we tested it directly rather than arguing about it.
Adapting a template means keeping the vendor's scaffold and swapping in a sentence about your use case. It feels like it should work, because the scaffold is the part that looks technical. What the scaffold actually contains is the pass criterion, and the pass criterion is the part that decides everything.
We took PostHog's generic scaffold and filled in the correct one-sentence requirement: assess whether the agent correctly classifies the lead's priority based on the information it has. Pointed at exactly the right target, it scored 56.2% and missed more than three-quarters of the real errors (76.5%) — because it checked that a tier had been assigned, not whether the tier was right. The agent picked a tier, any tier, pass.
The judgment lives in the rules, and the rules come from annotation. There is no shortcut that skips that step.
How to tell whether an LLM judge is working: four checks
Before any results are worth reading, you need the four numbers that describe a judge. Every one of them requires a set of outputs a human has already graded.
1. The baseline. On a test set balanced between correct and incorrect responses, a judge that passes everything scores 50% and a judge that fails everything scores 50%. Fifty per cent is a literal coin flip. Any agreement number quoted without its baseline is decoration.
2. Agreement. How often the judge's verdict matches the human's, on the same responses.
3. False negatives — errors it misses. A judge that rubber-stamps produces a beautiful dashboard and catches nothing.
4. False positives — good answers it flags. A judge that over-flags is technically catching your errors, but a team that gets a red mark on two out of three correct responses stops opening the eval within a fortnight. The system then dies of distrust, which looks the same as never having built it.
There is a fifth check, and it is the one almost nobody runs: does the judge fail the response for the reason the human failed it? You can only answer that if your human labels carry written justifications. Score-level agreement will not show you a judge that reaches the right verdict for an irrelevant reason.
What the checks showed across 8 judges and 10 models
The AI product under test was an inbound lead-qualification agent. It takes a company's context — product, ICP, buyer personas, qualifying and disqualifying signals, existing customers — plus an inbound contact, researches them, and assigns a priority tier from 1 (immediate outreach) to 4 (disqualify).
We found its dominant failure mode through error analysis: wrong tier assignment. We found it by sitting with real outputs and annotating them, in that order — annotations first, patterns second, judge third.
The test set was 34 real responses, human-labelled with written justifications, deliberately balanced at 17 correct and 17 containing a real tier error.
The custom judge's best pairing reached 76.5% agreement with the human expert, and the custom judge took the top eight of all 80 prompt × model pairings, every one of them at 70.6% or above, across eight different models. The best generic pairing in the whole benchmark reached 70.6% and got there by missing 41% of the real errors.
Averaged across all ten models, the picture is the same and the gap is narrower:
| Judge | Agreement with human | Misses real errors | False-flags good answers |
|---|
| Lovelaice custom judge | 68.8% | 38.2% | 24.1% |
| Correctness (openevals-style) | 58.2% | 25.9% | 57.6% |
| Adapted generic (PostHog scaffold) | 56.2% | 76.5% | 11.2% |
| Hallucination (Langfuse) | 55.0% | 37.1% | 52.9% |
| Helpfulness (Langfuse) | 54.9% | 79.2% | 11.8% |
| Hallucination (PostHog) | 54.4% | 22.4% | 68.8% |
| Helpfulness (PostHog) | 52.1% | 88.8% | 7.1% |
Three findings from that data matter more than the ranking.
Every generic judge has one of two personalities, and both cost you. The helpfulness and relevance templates rubber-stamp, missing 76–89% of real errors, with PostHog's Helpfulness missing 88.8%. The hallucination and correctness templates over-flag, marking 53–69% of perfectly good answers as failures. PostHog's Hallucination false-flags 68.8%, so more than two in three correct responses come back red. Which personality you get is decided by the template's wording, not by your product.
When generic judges agreed with the human, they were mostly right by accident. Because our human labels carry written justifications, we could check whether the judge caught the error for the reason the human did. The custom judge, on its two best pairings, caught 14 of 14 and 11 of 11 errors citing the human's actual reason — 100% both times. The Correctness judge caught 16 errors and named the real reason in 5 of them. The PostHog Hallucination judge caught 10 and named the real reason in zero, flagging a stray future date and a hyphen-stripped number instead. A judge that scores correctly for the wrong reason is a broken compass that occasionally points north, and score-level agreement hides it completely.
The hallucination judge hallucinated. The agent had written "~15–50 employees." Judges across the board misread the range as "1,550 employees" and failed the answer over a number that was never written: 46 judge outputs in the first run, 24 of them scoring FAIL, and 41 in the re-run. Every one of the ten models fell for it at least once. Claude Opus 4.8 misread it under seven of eight prompts, in both runs. We checked whether our own pipeline was mangling the en-dash. It wasn't. The misreading happens inside the judge models. The judge you install specifically to catch fabrication turned out to be the one most likely to fabricate.
How to build an LLM judge, step by step
Seven steps. The order matters more than the tooling, and steps 2, 5 and 7 are the ones teams skip.
- - Run your feature as an experiment across prompt versions and models, on a fixed set of test cases. In Lovelaice you import those as CSV or JSON, with whatever column names you already have.
- - Review the responses yourself. Mark pass or fail, and leave a written note on every failure explaining what went wrong. Lovelaice hides model identity during review so you grade the output rather than the brand. The written notes are what your judge gets built from, so this is not the step to rush.
- - Read the error categories. Cluster your annotations into failure patterns and look at which ones are severe, frequent, and not yet covered by any metric. Lovelaice does the clustering automatically, with severity, cross-iteration shift analysis, a model-by-category heatmap, and a coverage column showing which categories have nothing watching them. Those uncovered, high-severity, judgment-dependent categories are your judge candidates.
- - Write one judge per error. One failure mode, one binary score, one justification. In Lovelaice the judge is a first-class experiment in its own right, with its own prompt and its own model config, which is what makes step 6 possible.
- - Validate it against your human labels. Run the judge across your labelled set and score it on agreement, false positives and false negatives. Then sort to the disagreements and read the justifications rather than the scores. Sometimes the judge is wrong and you tighten a rule. Sometimes the judge is right and your own label was wrong.
- - Benchmark the prompt against every model, not just the one you assumed. You cannot predict a pairing, you measure it. Ours cost about $0.0073 per verdict and roughly $0.25 to validate the full 34-case set. There is no evaluation budget that cannot afford that.
- - Iterate, then re-validate. Our judge went from 65.1% to 76.5% across four versions, adding rules that came out of annotation sessions. The generic prompts, untouched, stayed exactly where they started. A judge you iterate without re-validating is drift with version numbers.
A PM or domain expert can run all seven without writing code. The prompts are plain prose, the models are dropdowns, and the validation metrics — human-judge agreement, false positive rate, false negative rate — come out of the box.
How to build an LLM judge with no production data
The most common reason teams never build a judge is that they haven't shipped, so they believe they have nothing to measure. The confusion is about which data a judge actually needs. It needs your own AI product's answers, and you can generate those today without a single user.
Write synthetic test inputs from domain knowledge. A test case is an example of input data your AI product will face in production. When the feature is not live this data can come from expertise rather than traffic. If you're building an inbound lead classification product, the synthetic data could be based on real product and real contacts from prospects. Ten to twenty inputs is enough to start: the obvious cases, the edge cases you already worry about, and the cases where a wrong answer is expensive.
Run your main AI product on them. This is the step that gets skipped. The synthetic inputs are not your judge's dataset — they are what produces it. What the judge needs to be validated against is a set of real answers from the feature you are about to ship.
Review those answers and find the error. Go through the outputs and mark what went wrong, with a written note each time. The failure patterns that surface here are what tell you whether you need a judge at all and which error it should watch. You do not choose the judge first.
Label the same answers for that one error. Go back over the outputs and mark each one pass or fail on the specific error the judge will look for, rather than on general quality. That labelled set is your judge's validation data. Aim for roughly half passes and half fails, so that a judge which rubber-stamps everything and a judge which fails everything both score 50% and neither can hide behind the average.
All of it happens before a single user sees the feature. The judge's ground truth is your own product's output on your own test cases, graded by the person who knows what right looks like — and by the time you ship, the feature and the judge watching it have both been measured.
Best platforms to build an LLM judge in 2026, compared
The seven platforms below are the ones a team choosing where to build a judge realistically shortlists. Galileo and Agenta appear in the catalogue table earlier but not here — Galileo sells to enterprise ML teams rather than product teams, and Agenta ships the judge machinery with only three rubrics, so there is little to compare on the template row.
| Lovelaice | Langfuse | Braintrust | Arize Phoenix | Confident AI | LangSmith | PostHog |
|---|
| LLM judge templates shipped | None, by design | 23 | 16 | 16 | ~30 | 7, plus 14 legacy | 4 |
| Guides you from error analysis to a judge | Yes | No | No | No | No | No | No |
| Judge validated against human labels | Core workflow | Manual, DIY | Manual, DIY | Manual, DIY | Manual, DIY | Manual, DIY | Not supported |
| Prompt × model benchmarking for the judge | Native | No | Partial | No | No | No | No |
| Reason-level agreement, not just score | Yes | No | No | No | No | No | No |
| Built for non-engineers | Yes | No | No | No | No | No | Partly |
| Benchmarked in this experiment | Yes | Yes | Read only | Read only | Read only | Yes | Yes |
FAQ
What is the easiest way to build an LLM as a judge?
Start from annotations, not from a template. Grade 20–30 of your own outputs by hand with written notes on every failure, pick the one failure mode that shows up most, and write a judge that asks a single binary question about it. That takes an afternoon and produces a judge you can measure. Selecting a template from a dropdown is faster and produces a number you cannot interpret.
Can I build an LLM judge without writing code?
Yes. The judge is a prompt and a model, and both are things a PM or domain expert can specify in plain language. What usually requires engineering is the validation harness around it — running the judge across a labelled set and computing agreement, false positives and false negatives. In Lovelaice that part is built in, which is what makes the workflow available to non-engineers.
How many labelled examples do I need?
Ten to twenty to start finding failure patterns. Thirty to fifty to validate a judge, which is the range this benchmark used at 34. Fifty to a hundred for a golden set stable enough to track across iterations. In every case, roughly half should contain known errors. A judge tested only on happy-path cases looks excellent right up until it rubber-stamps a failure.
How long does validating a judge take?
With the labelled set already in place and tooling that computes agreement for you, well under an hour.
Which model should I use as the judge?
There is no universal answer, and that is the finding. The same prompt swung 26 points across ten models in our benchmark, and the best model for one version of a prompt rubber-stamped the next version. Run the matrix.
Is a generic judge always useless?
No, and this experiment doesn't claim that. It says nothing about guardrail-style checks such as toxicity, PII, safety and prompt injection, where a general definition genuinely does transfer. The claim is narrower and harder to escape: a generic template cannot measure whether your AI did its job, in your product, for your users.
The bottom line
Every eval platform will sell you a judge in about four minutes. None of them can sell you a judge that knows what "wrong" means in your product, because that knowledge is in your team's heads and the only way out is annotation.
A judge you haven't validated against your own judgment isn't a judge. It's a random number generator you pay real money to run.
Want to see the full 80-pairing results matrix, or validate the judge you're already running in production? Book a call and we'll walk through it on your data.
Related reading: LLM as a judge: how to evaluate AI features · Error analysis for AI features · How to build a test dataset for an AI feature · Deterministic metrics: automating your checks
Madalina Turlea
