Twelve checks before you trust an AI to evaluate another AI.

A few months ago I sat in on an AI evals workshop. A big e-commerce team walked through the LLM judges they were using on their shopping assistant: relevancy, helpfulness, hallucination — all shipped by their observability vendor. So I opened the chatbot and tried one conversation. It dropped a constraint every turn, told me the platform did not carry a product it did carry, and burned 11 turns narrating a search it never ran. By any reasonable standard, the conversation failed. The dashboard was green.

So we staged a battle: 8 judges, 10 models, 34 real agent responses, 5,000+ verdicts, run twice. The best judge — a custom rubric built from human annotations — hit 76.5% agreement with the human expert and took the top 8 of all 80 pairings. The seven generic templates, from Langfuse, PostHog and openevals, averaged 52–58%, on a set where flipping a coin scores 50%. The 12 checks below are what separates the judge that swept the leaderboard from the ones that landed on the coin flip.