AI evaluation benchmarks

Original benchmarks we run so product teams don't have to guess. Every study here publishes its dataset, judge, prompts and scoring rubric — so you can reproduce the numbers, or challenge them, before betting a roadmap on the result.

Article18 SEPT 2026

Mental health AI: An expert rubric does not make a trustworthy judge

7 lessons on calibrating a judge for a high-stakes AI product: an AI mental-health clinician. 15 LLM judges, 60 conversations expert-annotated by 4 licensed psychologists, 900 verdicts.

Madalina Turlea

Article05 AUG 2026

The ultimate guide to building an LLM judge you can trust

A step-by-step guide to building and validating an LLM-as-a-judge — grounded in a benchmark of 8 judges across 11 models and 3,000+ verdicts. Generic judge templates agreed with human experts only 52–58% of the time. Here's the process that gets you a judge you can trust.

Madalina Turlea

Article27 JUL 2026

The best platform to build an LLM judge in 2026

How to build an LLM judge that actually works, and why the default templates score close to a coin flip. We benchmarked eight judges across 10 models against 34 human-graded responses — 5,440 verdicts. The custom judge hit 76.5% and swept the top 8 of 80 pairings; seven generic templates from Langfuse, PostHog and openevals never cleared the coin flip. Plus what Braintrust, Arize and Confident AI's default judge prompts actually say.

Madalina Turlea

Article21 JUL 2026

The battle of the LLM judges: Which LLM judge can you trust with your AI product's quality?

Lovelaice's custom judge built from human annotations hit 76% and swept the top 8 of 80 pairings. Seven generic templates averaged a coin flip.

Madalina Turlea

Article22 MAY 2026

What document extraction actually costs across 15 models

Two numbers that shouldn't coexist: an 11× cost gap between models on the same extraction task, and a 43-point accuracy gap on the same model driven by prompt structure alone. Cost is not a proxy for accuracy.

Madalina Turlea

Article15 MAY 2026

Which model should you run your LLM judge on?

The identical judge prompt scored 76.5% on one model and 50.0% on another in the same run. 26-point spread. Paying more does not buy accuracy — the best judge in the battle runs on a mid-tier Gemini for well under a cent a verdict.

Madalina Turlea

Article12 MAY 2026

Do LLM judges give the same verdict twice?

We ran the identical LLM judge benchmark twice at the lowest sampling setting. 8.2% of the verdicts flipped. Verdict stability is a model property, and small GPTs flip 14–17% of the time on identical inputs.

Madalina Turlea

Article05 MAY 2026

Langfuse vs PostHog vs openevals: the judge templates compared

We ran the generic LLM judge templates from Langfuse, PostHog and openevals against 34 human-labeled AI responses across 10 models. They cluster inside a 6-point band. The vendor is not the variable — the template's question is.

Madalina Turlea

Article15 JAN 2026

We tested the viral prompt tricks. Most of them do nothing.

Threatening the model, all caps, high-stakes framing — we ran the viral prompt techniques across nine models on real extraction tasks. Structure and clarity beat hacks every time.

Madalina Turlea

Masterclass04 DEC 2025