LLM-as-judge
Definition
Using a model to score another model's output against written criteria, for things you can't check deterministically — tone, completeness, whether the answer addresses the real question. Only as good as the criteria you write, and only trustworthy after you've validated it against human-scored examples.
An LLM-as-judge is a model prompted to score another model's output against specific written criteria. It's how evaluation scales past what a human can read. Two catches: the judge is only as good as the criteria you write (a domain expert's job), and you have to validate the judge itself against human-scored examples before you trust it. An unchecked judge is a vibe check with extra steps.
Why it matters
Judges unlock the criteria a computer can't verify, but they inherit whatever ambiguity is in the prompt. A judge asked 'is this helpful?' agrees with humans 60–70% of the time — barely better than random. A judge asked to check a specific, binary criterion, validated against real annotations, hits 85–95%.
Related terms