Mental health AI: An expert rubric does not make a trustworthy judge

So for each of the 60 conversations, the data we have is four psychologists, each giving 6 scores (5 criteria plus overall) and a written justification. That is 24 expert scores per conversation, plus the reasoning behind them. When we say "the experts' score" in the lessons, we mean the average of the four on that criterion, unless we say otherwise.

What we ran: 15 judges, 60 conversations, 900 verdicts

We took the study's rubric as the judge's system prompt and ran it as the judge across 15 models on all 60 conversations: 900 judge runs. The rubric itself is unchanged: the same scale, the same five criteria, the same descriptions of what a 1, a 3 and a 5 look like. We changed what the judge returns. The original judge in the study wrote five criterion scores and nothing else. Ours also writes an overall score, the way the psychologists did, and a short rationale for every score, so we can read why it landed where it did. We also left out the study's worked example conversations, so the judge sees the rubric alone. The models: gpt-5.5, gpt-5.4, gpt-5.4-mini, gpt-5.3-chat, gpt-5.2-chat and gpt-4.1 from OpenAI; claude-opus-4.8, claude-opus-4.7, claude-sonnet-5, claude-sonnet-4.5 and claude-haiku-4.5 from Anthropic; gemini-3.1-pro, gemini-3.5-flash and gemini-3-flash from Google; and Qwen3-235B.

Then we scored the judges 3 ways, because a production judge has to pass 3 different tests.

First, whether it scores like the experts. We counted a judge as agreeing on a conversation when its score was within 1 point of the four psychologists' average. Why 1 point: a single psychologist differs from the average of her 3 colleagues by about 1 point, so we asked the judge to agree with the panel about as well as the panel members agree with it. We computed this on the overall rating and on each of the five criteria, and we report it two ways: on the overall rating alone, and averaged across all 6 scores. The two do not give the same winner, so we say which one we are using each time.

Second, whether it ranks like the experts: does it put the conversations in the same order, worst to best, that the psychologists did. We used the study's own metric for this, Kendall tau.

Third, whether it decides like the experts. We turned every score into a verdict, for the judges and for the psychologists, at a production bar: a conversation passes if the experts' average is 4 or above, which the rubric calls "solid performance with minor issues." That is 10 passes and 50 errors. Then we counted missed errors and false alarms for each judge, the same way we did in the Judge Battle. This is the test that matters for production, and it is the one the original study did not need.

Everything in the lessons below comes from the same 60 conversations and the same 900 judge runs.

---

The leaderboard most teams would ship from

Here is the full leaderboard, before any of the lessons. Agreement means the share of conversations where the judge's score was within 1 point of the four psychologists' average. It is shown for each of the 5 criteria, for the overall rating, and averaged across all 6. The last column is Kendall tau on the overall rating, the ranking metric from the original study, from minus 1 to 1. The bottom row is the dummy judge: a constant score, the best one for each metric, written on every conversation without reading it.

Percentages are over the conversations each judge returned a readable score for. Every judge scored all 60 except claude-sonnet-5, which returned a readable overall score on 54 (54 to 56 on the criteria), and gemini-3-flash, which returned 58 on AI communication. For reference, on our 60 conversations one psychologist against the average of the other 3 lands at 0.23 on Kendall tau for the overall rating.

In the next section, we will drill down into this data to see the lessons on how to calibrate an LLM judge for a high-stakes product. They come in 3 groups: how to read the numbers, what the judge is actually doing, and how to build the judge. In short:

Reading the numbers

1. •Before you trust 95% judge accuracy on a scale, run a judge that scores 3.5 on everything. A dummy judge that writes 3.5 on every conversation agrees with the experts on the overall rating 95% of the time. That ties gpt-5.4, the winner of the averaged leaderboard, and sits 3 points behind the best judge on that one rating, because the top judges use two points of a 6-point scale. Turn the scores into a pass/fail verdict and a dummy can only be perfect on one side.
2. •Build the calibration set around the failures, or the judge cannot fail your test. Only 9 of the 60 conversations are clearly bad by the experts' standard. A calibration set that is mostly good conversations can only prove the judge agrees on good conversations.
3. •Decide the production bar first, then rank the judges on missed errors. The agreement score winner missed 68% of them. On agreement averaged across the 6 scores, gpt-5.4 is first and claude-opus-4.7 sixth. On the 9 clearly bad conversations, claude-opus-4.7 lands within 1 point of the experts on 9 of 9 and gpt-5.4 on 6. As a pass/fail verdict at the production bar, gpt-5.4 misses 68% of the errors.

What the judge is actually doing

1. •No matter how detailed the rubric or how fine the scale, expect an indulgent judge. On the overall rating, all 15 judges sit above the experts on average, from 0.07 to 1.7 points. Most of them write a 3 or a 4 on almost everything.
2. •Read the judge's reasoning against your experts' reasoning. The top judges saw the failures and scored a 4 out of 6 anyway. On the clearly bad conversations, the top judges name most of the problems the psychologists named, then score a 3 or a 4 anyway. The worst judges do not see the problems at all. Those are two different failure modes with two different fixes, and you only tell them apart by reading the reasoning.

Building the judge

1. •Keep each judge narrow: one criterion, one verdict. 15 judges scored five criteria at once, and their scores moved together 47% of the time, against 13% for the psychologists. Five criteria have five different best judges. Inside one judge, leniency points up on one criterion and down on another, and a judge can write the same score on every conversation for one criterion while looking fine on the average.
2. •Decide whose standard the judge follows before you calibrate, and do not switch it. The same judge was first against the panel average and one of the experts, and ninth against the fourth expert. Against the strict or moderate psychologist the leaderboard looks like the panel's. Against a lenient one it flips. Whichever standard you calibrate to, keep it between iterations, or you cannot tell an improvement from a change of standard.

The rest of the article walks through each of these with the data behind it.

---

Reading the numbers

The first 3 lessons are about the numbers a team sees after a calibration run, and what each of them can and cannot tell you.

Lesson 1: Before you trust 95% judge accuracy on a scale, run a judge that scores 3.5 on everything

In the original study, 4 clinical psychologists scored every conversation on a 1 to 6 scale, on 5 criteria plus an overall rating. It is a well-built scale, and the psychologists used it the way it was meant to be used. Every point has a definition (1 means major failures that undermine safety, 3 means acceptable with notable limitations, 4 means solid with minor issues), and on the overall rating the psychologists used 1 to 5, never a 6.

So we scored the 15 judges the way most product teams would: how often does the judge land within 1 point of the experts' overall average? gpt-5.4, the winner of the leaderboard above, was within 1 point on 95% of the conversations. Then we compared it with a dummy judge that writes 3.5 on every conversation without reading it. It was also within 1 point on 95%. Even the best judge on this one rating, claude-opus-4.7 at 98%, beats the dummy by only 3 points.

The top GPT judges were not really using the full scale. Across 60 conversations they wrote almost nothing but 3s and 4s, 2 of the 6 points, and a judge that sits in the middle of the scale is always within a point of an expert average that also sits in the middle. The metric could not tell a judge using two points from a judge using all 6. A finer tolerance makes it worse: at half a point, the dummy beats every model, 78% to 65%.

When building a judge that can help in decision making on production readiness, we always ask for pass or fail, not a score on a scale. So the next check was to see what the dummy judge does once the scores become verdicts. We turned the same scores into verdicts, for the judges and for the experts, and ran the dummy test again.

Pass means the experts rated the conversation 4 or above, which is 10 of the 60. Everything below 4 is an error, 50 of them. The dummy is still strong. An always-fail judge scores 83% accuracy. But it can only win on one side: it misses 0% of the errors and raises a false alarm on 100% of the passing conversations. Put those two numbers next to each other and the dummy has nowhere to hide, while a real judge like claude-opus-4.7 misses 38% of the errors and raises a false alarm on 10% of the good ones.

On a scale, a judge can be wrong in a way that looks like agreement. A verdict forces the disagreement into the open. Never read an agreement score on its own. Put a dummy judge next to it, and if the product decision is ship or not, measure the judge on that decision, with missed errors and false alarms reported as a pair.

Lesson 2: Build the calibration set around the failures, or the judge cannot fail your test

Lesson 1 showed that the top judges wrote almost only 3s and 4s. The dataset explains why this might happen. Of the 60 conversations, the experts rated only 9 as clearly bad (an overall average of 2.75 or below), and only 3 landed below 2.5. Twenty-nine of the 60 sit between 3.0 and 3.25. So a judge that writes a 3 or a 4 on everything is within a point of the experts on most of the set, and the calibration mostly tested the judges on conversations where they could not fail.

We saw this happen as we went. We split the 60 test conversations into two batches of 30. When we tested the second half, with the same prompt and the same judges, 8 of the 15 models scored higher on the overall rating compared to the first half, and 8 of the 15 on the averaged leaderboard. In a real product cycle that reads as progress. It was selection: the second batch had no conversation below 2.5, and the dummy judge passed 100% of it.

For a high-stakes product the judge has two jobs, in this order. First, spot the failures reliably, and spot them the way your experts do. A missed failure here is a member with severe depression whose risk was never assessed, and a judge that told you the conversation was solid. Second, pass the good conversations without noise, so the judge does not create more work than it saves. A dataset that is mostly average conversations can only prove the second job. It cannot tell you whether the judge can do the first one, and here it hid that most of the judges could not.

In the Judge Battle we built the dataset on purpose: 34 responses, 17 correct and 17 with a real error, so an always-pass judge scored exactly 50% and had nowhere to hide. Pass/fail makes that easy to design. With a scale you need enough conversations at every point the experts actually use, which is harder to collect and easy to skip. The rule is the same either way.

Design the calibration dataset around the failures you have observed and need caught, not around the conversations you happen to have. If you only look at the agreement score, you will not see that the dataset did the hiding. Read the score together with what the judge was tested on.

Lesson 3: Decide the production bar first, then rank the judges on missed errors. The agreement score winner missed 68% of them.

Here is the leaderboard a team would see after the run, ranked by agreement with the experts' score, averaged across all 6 scores (5 criteria plus overall): gpt-5.4 first at 93.9% within 1 point of the experts, the two GPT chat models second and third, claude-opus-4.7 sixth. We know what that meeting sounds like, because we have sat in it: "gpt-5.4 agrees with the experts 94% of the time. Let's go with that one and move on." It is the obvious choice, and on this data it is the wrong one.

The original study did not rank judges by agreement. It used a ranking metric, Kendall tau, which asks whether the judge puts the conversations in the same order as the experts, from worst to best, on a scale from minus 1 to 1. On that metric the leaderboard already looks different. claude-opus-4.7 is first at 0.48, gpt-5.4 second at 0.42, and gemini-3.5-flash third at 0.39 even though it agrees within 1 point only 53% of the time. The GPT chat models drop to tenth and eleventh at 0.21 to 0.25. gpt-4.1 ranks conversations worse than random, at minus 0.16, while passing 48% of the within-1 checks on the overall rating. For reference, on our data one psychologist against the average of the other 3 lands at 0.23.

Each metric has a blind spot. Agreement cannot see ranking: a judge can be within 1 point of the human expert on 90% of conversations and still not agree with the expert that one conversation is worse than another. Ranking cannot see leniency, meaning when the judge consistently scores conversations higher than the human expert. And leniency is exactly the problem here: the judge writes a higher score than the experts on the same conversation, again and again, so a conversation the experts rated 2.75 gets a 3 or a 4. Neither metric tells you the one thing you need to know in a high-stakes product: does the judge catch the real failures (here the bad conversations)?

So we ranked the same judges on that job alone. We called a conversation clearly bad when the four psychologists' average overall rating was 2.75 or below, which in the rubric's own words means somewhere between "major failures that undermine safety" and "acceptable with notable limitations." There are 9 of them. We called a conversation clearly good when the average was 3.5 or above, which is 22 of them. The 29 in between have no clear verdict, so we left them out. "Catches" here means the judge's overall score lands within 1 point of the experts' average, so a 3 on a 2.75 still counts. On this check, claude-opus-4.7 catches 9 of 9 clearly bad conversations and confirms all 22 clearly good ones. claude-sonnet-5 catches 8 of the 8 it returned a score on. gpt-5.4, the leaderboard winner, catches 6 of 9. The two GPT chat models catch 3 of 9. gpt-5.4-mini, gpt-4.1 and qwen catch none. Two of the 3 gemini judges, gemini-3.1-pro and gemini-3.5-flash, catch 7 of 9, but they also miss 9 and 8 of the 22 clearly good conversations, so they catch failures by scattering their scores, not by judging. A wide judge is not a calibrated judge.

There is one more reason why neither agreement, nor ranking make a good judge metric and why a pass/fail score works better for production AI. In the calibration, we counted a clearly bad conversation as caught when the judge's score was within 1 point of the experts' score. In production there is no experts' score. There is only the judge's score. And when the judge writes a 3 or a 4, nobody on your team will open that conversation and mark it as a failure, even if it is one. There is no ranking to fall back on either, because ranking needs a list of conversations ordered by experts, and in production you do not have one. So the calibration told you the judge was "close" on the clearly bad conversations, and in production close does not fail anything.

Go back to what a judge is for in production: spot the failures, and do not create noise. Two numbers measure exactly that. Missed errors: take the 50 conversations the experts put below 4, and count the share the judge passed anyway. Every one of those is a failed conversation that ships. False alarms: take the 10 conversations the experts passed, and count the share the judge failed anyway. Every one of those is a passing conversation someone has to review by hand. A dummy can only be perfect on one of the two. The always-fail judge misses 0% of the errors and raises a false alarm on 100% of the passing conversations. Read together, the pair tells you whether the judge is doing the job or hiding.

Ranked on these two numbers at the "production quality" bar, where pass means the experts rated it 4 or above, the leaderboard changes for the third time. Keep in mind that the errors here are the 50 conversations below 4, not only the 9 clearly bad ones, so a judge that catches all 9 can still miss plenty.

claude-opus-4.7 leads again: it misses 38% of the errors and raises a false alarm on 10% of the passing conversations. claude-sonnet-5 misses 52%, claude-opus-4.8 56%. gpt-5.4, the winner of the agreement leaderboard, drops to sixth on missed errors: it misses 68% of them and raises zero false alarms. The two GPT chat models miss 76% and 84%. gpt-5.4-mini, gpt-4.1 and qwen miss 98% to 100%.

Look at the false-alarm column and the whole pool has the same shape: nine of the 15 judges raise zero false alarms, none raises more than 20%, and every one of them misses more than a third of the 50 errors. That is what leniency looks like as a verdict. The judge lets too much through. For comparison, the strict psychologist scored the same way misses 0% of the errors and raises a false alarm on 80% of the passing conversations. The moderate psychologist misses 16% and raises false alarms on 30%.

No judge in the pool is that balanced yet.

Decide the production bar first, then rank the judges on missed errors, with false alarms next to it. 3 ways of reading the same 900 judge scores gave two different winners. Agreement with the experts' score, averaged across the 6 scores, put gpt-5.4 first and claude-opus-4.7 sixth. The ranking metric and the missed-errors leaderboard both put claude-opus-4.7 first and gpt-5.4 lower down. Agreement is the number a team sees first and it is not enough; ranking is useful but cannot be used for online evaluations.

What the judge is actually doing

The first 3 lessons were about reading the numbers. The next two are about what the judge does with the rubric, and why the numbers came out the way they did.

Lesson 4: No matter how detailed the rubric or how fine the scale, expect an indulgent judge

Take the 60 conversations, average each judge's overall rating, and compare that to the average of the four psychologists' overall ratings. Every one of the 15 judges lands above the experts. claude-opus-4.7 is barely above, by 0.07 of a point. qwen is 1.7 points above. The rest sit in between. Not a single judge scored lower than the experts on average. (Per criterion the picture is less uniform, and Lesson 6 gets to that.) This habit of scoring too high is called leniency. We call it an indulgent judge, and it is the default behavior of an LLM judge.

The simplest way to see it is to look at which numbers each rater actually writes. The strict psychologist writes a 2 on 68% of the conversations. gpt-5.2-chat writes a 4 on 87% of them. gpt-5.4-mini writes a 4 on 97%. qwen writes a 5 on 98%. These are the same 60 conversations. The judges are not disagreeing with the experts conversation by conversation. They have picked a comfortable number and they write it almost every time. Only one top judge moves around: claude-opus-4.7 writes a 2 on 10% of the conversations, a 3 on 43% and a 4 on 47%, which is close to the shape of the experts' own averages, cut at the same bands (2.5 or below on 8%, 2.75 to 3.25 on 55%, 3.5 or above on 37%).

A scale makes this easy to miss. Scoring one point too high on a scale still looks "close" under the agreement metric, so a judge can score everything too high and pass the calibration. A verdict has no "a bit too high." A pass is a pass, and if the experts failed the conversation, it shows up as a missed error. That is why the missed-error numbers in Lesson 3 were so bad for judges that looked fine on agreement.

So when your judge never writes a 1 or a 2, do not read it as "our product is good." Read it as "check the judge." The question to ask is simple. When an expert gives a conversation a low score, does the judge give it a low score too, and for the same reason? When an expert gives a high score, does the judge give a high score for the same reason? A judge that sits in the middle of the scale on everything has not answered either question. It has picked the number that is hardest to be wrong with.

Treat "the judge never scores low" as a finding, not a comfort, and check that the judge can go low where your experts go low. To check the reasons, and not only the numbers, you need the judge to write them down. That is Lesson 5.

Lesson 5: Read the judge's reasoning against your experts' reasoning. The top judges saw the failures and scored a 4 out of 6 anyway

The score tells you whether the judge landed in the right place. The reasoning tells you whether it got there for the right reasons, and that is the only thing you can fix. So the place to calibrate a judge to your experts is its reasoning. Ask the judge to write down why it gave the score, then put that next to what the expert wrote about the same conversation.

In one conversation, the four psychologists gave a 2.5 out of 6 on average. Their notes flag a conversation stuck in a loop of repeated closings, an AI clinician that never engages with the member's severe anxiety, a coffee ritual offered as an intervention where the psychologist wanted something else, and hearts in the messages that cross a boundary. Now read 3 judges on the same conversation. gpt-5.4 wrote "repetitive, unnatural AI communication and mild boundary blurring" and scored it 3. claude-sonnet-4.5 wrote "repetitive, mechanical communication patterns typical of AI systems, preventing the conversation from achieving natural therapeutic flow" and scored it 4. claude-opus-4.7 wrote that the wrap-ups "ignore the member's clear closure signals and never engage with her severe anxiety" and scored it 2. All 3 saw the problem. Only one wrote the lower score.

We did this by hand for all 9 clearly bad conversations, so the percentages are rough. claude-opus-4.7's reasoning names about 75% of the issues the experts raised. gpt-5.4 names about 60%. gpt-5.2-chat names about 50%, and on the worst conversation in the set it lists the missed depression assessment, the reminders the AI could not send and the repeated poetic closings, then concludes that it was "a solid supportive conversation" and scores it 4. qwen names about 0%, and often says the opposite of the experts: the AI was "avoiding robotic patterns or repetition" on a conversation 3 of the 4 psychologists flagged for repetition. To be fair to the judges, none of them invented failures on the clearly good conversations. Where they criticized, the experts had criticized too.

So there are two kinds of judges here, and they need two different fixes. The first kind sees what the experts see and still writes a 3 or a 4. Every model in the pool did this to some degree, which tells us it is a prompt problem: the judge has to be told, with examples, which failures pull a conversation from a 3 down to a 2. The second kind does not see what the experts see, and praises what they condemned. That is much harder to fix from the prompt, as it is an issue only with certain models. Better level descriptions and a few worked examples can help, but in our runs this looked like a model problem. Some models are simply better at seeing what your experts see. You will only know which kind you have by reading the reasoning.

Even here, a verdict makes the fix easier than a scale. With pass/fail you can write "repetitive, mechanical communication is a fail on AI communication," and the judge has a rule it can apply. With a scale you have to explain how much repetition separates a 2 from a 3, and then trust the judge to weigh it. The pass/fail version is an instruction the judge can follow. The scale version hands the judgment back to the model.

Ask the judge for its reasoning, not only its score, and compare that reasoning to your experts' reasoning on the same conversations. That is where you find out whether the judge cannot see the failure or can see it and will not say so, and each of those needs a different fix.

Building the judge

The last two lessons are about design decisions: how many judges to build, and whose standard they carry.

Lesson 6: Keep each judge narrow: one criterion, one verdict. 15 judges scored 5 criteria at once, and their scores moved together 47% of the time. The psychologists', 13%.

So far we have looked at the overall rating of the conversations. But the rubric asks the judge for 6 numbers on every conversation: clinical accuracy, ethical conduct, assessment and response, therapeutic alliance, AI communication quality, and then the overall. Our judges scored all 6 in one prompt, in one pass, the way the original study did. The question we ask every PM who shows us a rubric like this: one judge for the whole thing, or one judge per criterion? We have not run the split on this data yet. But the 900 runs we have already point one way.

Start with the simplest check. We looked for one model that is the best judge on every criterion. Here are the top 3 per criterion, once by agreement with the experts' score (within 1 point) and once by ranking (Kendall tau).

No model tops every column, and the two columns rarely agree. The gemini judges lead the ranking column on 3 criteria. On two of them they sit at 32% and 65% on agreement, because they order conversations well and score them badly (on the third, therapeutic alliance, gemini-3.5-flash is at 90%). The GPT chat models lead the agreement column on therapeutic alliance, where a constant 4 passes 98%, which is the dummy judge again. Pick per criterion the way we picked overall, agreement and ranking together, and it is still five different models: gpt-5.4 on clinical accuracy, gpt-5.5 on ethics, claude-sonnet-5 on assessment, claude-opus-4.8 on therapeutic alliance, claude-sonnet-4.5 on AI communication. claude-opus-4.7, the best single judge on every overall measure, tops no criterion on agreement and ranking at once (it ties gpt-5.4 for first on clinical accuracy agreement). However you cut the data, no model wins on everything. Each one is good at some criteria and weak at others, and that is a real sign that splitting the judge by criterion is the better design.

Then look inside one judge. The same judge scores too high on one criterion and too low on another. For ten of the 15 judges, the most over-scored criterion is assessment and response, the criterion whose expert critiques need the most clinical training, by 0.11 to 1.16 points. Several of the same judges are stricter than the experts on ethics: gpt-5.4 by 0.17, claude-sonnet-5 by 0.21, gpt-5.4-mini by 0.28. One "score lower" instruction for the whole rubric would push ethics further down and leave assessment where it is. Separate judges can be corrected separately.

Worse, a single judge can be a real judge on one criterion and a dummy judge on another. gpt-4.1 gave the same AI communication score, a 4, to all 60 conversations. gpt-5.4-mini gave the same assessment score, a 4, to all 60, and passes 90% of the within-1 checks on that criterion. gpt-5.2-chat gave the same assessment score to 58 of 60. On the averaged leaderboard these hide inside pass rates of 57% to 92%. Look at the spread of scores on each criterion and they are flat lines.

The last check is whether the 6 scores are really 6 judgments. For the psychologists, mostly yes: an individual psychologist gives at least four of the five criteria the same score on 13% of conversations. For the judges it is 47%. The bottom of the leaderboard scores the rubric as one thing (gpt-4.1 on 87% of conversations, qwen on 98%). The top judges are close to human (claude-sonnet-4.5 0%, claude-opus-4.7 18%), but even their mistakes travel together across criteria more than a psychologist's do. This is what you would expect if the judgment the model finds easy, spotting robotic communication, colors the judgments it finds hard. The data cannot prove that. Every judge here saw all five criteria at once, so we cannot separate "one criterion leaks into the others" from "the judge is flat everywhere." Splitting the judge is the experiment that would tell.

There is a practical reason to split as well, and it is the same one as in Lesson 5. A judge that answers one question can be given much more context about that one question. A pass/fail judge for AI communication can be told exactly how the failure shows up: the same sign-off 3 times, a poetic closing repeated turn after turn, "I'm doing well, thanks for asking" from an AI.

A pass/fail judge for assessment can be told what a missed suicide-risk screening looks like in this product. Try to fit that level of detail for 5 criteria into one prompt and you get a rubric the judge skims.

Keep each judge narrow: one criterion, one verdict, and the specific ways that criterion fails in your product written into the prompt. We have not tested this split on the mental-health data yet. It is one of the next experiments we will run: one judge per criterion, each on the model that did best on it, and we will come back with the numbers.

Lesson 7: Decide whose standard the judge follows before you calibrate, and do not switch it. The same judge was first when compared with the panel average and one of the experts, and ninth against another expert

Every number so far compares the judge to the average of four psychologists. We also have each psychologist's own scores, and they are four different raters. For this lesson we use agreement on the overall rating alone (within 1 point), not the 6-score average from the leaderboard table. On that rating claude-opus-4.7 already leads against the panel at 98%, with claude-sonnet-5 second and gpt-5.4 third. One is strict: her average is 2.32 and she writes a 2 on 68% of the conversations. One is moderate, at 3.12. Two are lenient, at about 3.86, and they write a 4 or a 5 on about two thirds of the conversations. They do not just differ on level. They order the conversations differently too: on our 60 conversations, the ranking agreement (Kendall tau on the overall rating) between any two of them runs from 0.01 to 0.49. None of them is wrong. They are four trained clinicians with four views of what good care looks like, and in a domain like mental health that is the normal case, not the exception.

So we asked: what if we had calibrated the judge against just one of them? Against the strict psychologist, or the moderate one, the leaderboard looks like the leaderboard against the panel. The same judges sit at the top, claude-opus-4.7 first. Against the lenient psychologists the leaderboard flips. The GPT judges take the top, claude-opus-4.7 drops to ninth against one of them and sixth against the other, and it now scores about half a point too low for both. The reason is Lesson 4. Every judge in the pool scores too high, and a lenient reference forgives exactly what a strict reference punishes. The panel average happens to sit close to the strict and moderate psychologists, which is why the panel leaderboard looks like theirs.

Which judge is best depends on who you compare it to. Change the standard and you change the winner. That has a direct consequence for how you run calibration over time. Say you calibrate the first version of your judge against the strict psychologist, and the next version against a lenient one. The agreement number goes up, the missed-error number goes up, and none of it means the judge got better. You changed the exam, not the student. The same thing happens, and you notice it even less, when one expert annotates the first 30 conversations and a different expert annotates the next 30. The numbers between iterations stop being comparable, and you can no longer tell an improvement from a change of standard.

There is a product side to this too. Whoever annotates the calibration set is putting their point of view into the judge, and from then on into the product. Calibrate against the strict psychologist and you ship a product that fails a conversation the moment it misses a risk screening. Calibrate against a lenient one and you ship a product that calls the same conversation solid. Average the four and you ship the middle. Any of these can be the right call. What is important is that you make that call intentionally and keep consistency as you iterate on the product and the judge.

In practice there are two ways to keep the standard steady.

Either one expert owns it, shapes it, and has the ultimate call on everything. Or you have a panel, and the panel rates everything together, every iteration, so the average means the same thing each time. What does not work is one expert on this batch and another on the next.

Decide whose standard the judge carries before you calibrate, write it down, and do not switch it between iterations. Then every change in the numbers is a change in the judge.

Bonus: Test self-preference on your own data. On 60 conversations, no judge favored its own model's work

One worry with LLM judges is that a model will score its own outputs higher. This dataset let us check. The AI clinician side of the 60 conversations was written by 3 different models, 20 conversations each: gpt-5-chat, Qwen3-235B and DeepSeek-R1. Our judge pool includes several GPT models and the same Qwen model. So we checked whether GPT judges score GPT-written conversations higher than other judges do, and whether the Qwen judge favors Qwen-written conversations.

On this data, no. We measured how far above the psychologists each judge family scored, split by which model wrote the conversation. The GPT judges were the least generous on GPT-written conversations, at 0.43 points above the experts on the overall rating, against 0.80 on Qwen-written and 0.75 on DeepSeek-written ones. The Qwen judge sat 1.73 above the experts on its own conversations, in between its 1.39 on GPT-written and 2.06 on DeepSeek-written ones. Neither family bumped its own work.

What does show up is something else. The experts rated the GPT-written conversations higher on average (3.61) than the Qwen-written (3.27) and DeepSeek-written ones (2.99), and all 9 clearly bad conversations were written by Qwen or DeepSeek. So a GPT judge looks good on the leaderboard for a simpler reason: it scores too high, like every judge here, and the GPT-written conversations happened to be the easy ones. One dataset, one rubric, one run per model, so treat this as a check that passed here, not as a rule.

Treat the judge as a product: what to do with this

This rubric is about as complex as evaluation rubrics get. Four clinical psychologists, five criteria, 6 points on the scale, each point defined, each criterion spelled out with examples of what a 1, a 3 and a 5 look like. Hand it to an AI model and it will score every conversation, write a confident rationale for every score, and the numbers will come back looking fine. 15 models did exactly that. Fourteen of them missed more than half of the conversations the experts failed.

A detailed, expert-built rubric is the right starting point, and you should still not trust the judge that runs it until you have checked it. The rubric makes the judge sound trustworthy. The checks make it trustworthy.

The score is the last thing to look at. Look at how much of its agreement a dummy judge would get for free, how many real failures the calibration set actually holds, how many of them the judge misses at your production bar, whether it ever goes low where your experts go low, and whether its reasons match their reasons. Every one of those checks is cheap. What is expensive is skipping them and finding out in production.

The reason to make the effort is what the judge is for. Before shipping, it tells you whether the feature is good enough. After shipping, it tells you where the AI failed on real traffic, so your team looks at the right conversations and not at all of them. A judge that scores everything a 3 or a 4 does neither. It lets bad answers through, and it turns the good ones into noise. So treat the judge the way you would treat any other AI feature you were about to ship: as a product of its own, with its own dataset, its own experiments, and its own validation before it grades anything that matters.

In practice, that looks like this:

1. •One judge per failure, with a verdict. Pick the failures that matter most in your product, and give each one its own judge that answers one question with pass or fail. Write the specific ways that failure shows up in your product into the prompt. A judge that answers one question can be given the context it needs to answer it well.
2. •Build the calibration set around the failures. Make sure it holds enough examples of each failure your experts actually flag, not only the conversations you happen to have. A set that is mostly good conversations can only prove the judge passes good conversations.
3. •Run it across several models, and rank on the decision. The model that wins on agreement is not the one that catches the failures. Read missed errors and false alarms as a pair: the share of expert-failed answers the judge passed, and the share of expert-passed answers it flagged. Put a dummy judge next to every number.
4. •Read the reasoning against your experts' reasoning. That is where you learn whether the judge cannot see the failure, or sees it and will not say so. The first is a model problem. The second is a prompt problem, and you can fix it.
5. •Iterate on the prompt, and keep the standard fixed. Keep the same experts, the same bar and the same dataset from one version to the next. Then every change in the numbers is a change in the judge, and you can build on it.

There is one more decision here, and it is the one most teams never make on purpose. Every decision in the rubric, and every choice about whose annotations the judge learns from, is a decision about what your product considers good. Four clinicians disagreed on the same 60 conversations, and any of their standards could be the right one. The judge will carry whichever one you pick into every conversation it grades. Make that decision on purpose, write it down, and let the domain experts who own the product quality own it.

This is the work we do with teams at Lovelaice. We build the judges together with your product people and your domain experts, on our platform, so that by the end the expertise sits inside your team: the dataset, the failure definitions, the validated judges, and the habit of checking them before trusting them.

If you are building an AI product where a wrong pass has a real cost, reach out. We will look at your judge together and show you what it is hiding.

---

Want to run these checks on your own judge, on your own data? Book a call and bring your worst-scoring conversation — we will walk through the calibration together.