Why your AI evaluation is lying to you

And Anthropic's own engineering team warns: "A bad judge is worse than no judge."

A bad judge doesn't just miss failures. It gives you confidence that things are working when they're not. That's worse than having no evaluation at all, because at least without a judge, you know you're flying blind.

Why this keeps happening

When I talk to teams about this, I hear the same three responses. They're reasonable on the surface, but each one hides a misunderstanding about how AI evaluation actually works.

"We're still in beta, we don't have many users yet, so we just rely on thumbs up/down."

This is exactly when evaluation matters most. Beta is your window to catch failures before they scale. And if you're relying on user feedback as your quality signal, you're building on sand.

The reality of AI products is that they fail silently. No error messages, no crashes, just confident-sounding responses that look fine unless you actually know what to check for. Users don't file bug reports for a recommendation that was merely "okay" instead of useful. They don't tell you the AI missed a nuance that a domain expert would have caught. They just close the tab.

39% of AI chatbots deployed in 2023-2024 were eventually pulled back due to performance issues. In 43% of failed AI deployments, the system was technically functional but produced low-quality outputs that weren't caught until users complained.

We want to benchmark quality first, then do error analysis

This is backwards, and it's the most common sequence mistake we see.

Teams want a score first: "What's our accuracy?" And then they plan to investigate errors later. But a benchmark score without understanding how and why your AI fails is a vanity metric. What does "72% accuracy" actually tell you? Nothing actionable.

Error analysis is how you benchmark. When you manually review outputs and document specific failures: "it confused milligrams with micrograms," "it ignored the user's constraint", you're simultaneously learning your quality and discovering what to fix.

The alternative we see in practice is even worse: a reactive, one-off process. A new error surfaces in production. A domain expert or PM says "handle it like this." Engineering patches the prompt. That's it. No systematic process. No tracking of whether the fix introduced new failures. No understanding of how that error relates to other errors.

"Evaluation is too complex, you need annotation infrastructure, massive datasets, specialized tools..."

This is the myth that keeps the most teams stuck. It sounds expensive and time-consuming, so they either skip evaluation entirely or jump straight to automation.

You don't need a data pipeline. You don't need a thousand labeled examples to start.

You need 10 to 20 test cases and the willingness to actually read the AI's responses and write down what went wrong.

Anthropic recommends starting with 20-50 examples. OpenAI's guide says 50-100 for establishing a human baseline. Not thousands. Not hundreds. A few dozen thoughtful cases with detailed notes.

50 cases with meaningful manual annotations will teach you more about your AI's failure patterns than any automated system running on generic criteria. Those notes, those patterns, that's your real evaluation foundation. Everything else, including knowing if and when you actually need LLM-as-a-Judge, builds on top of it.

What to do instead: The Evaluation Ladder

Here's the approach that actually works. We've refined this across 100+ product teams and 1,000+ experiments at Lovelaice. The core principle is simple: earn each step before moving to the next.

Step 1: Explore and compare

Before you commit to anything, test your task across multiple models with the same prompt. See how different models handle it. Don't pick a model upfront based on benchmarks or Twitter hype, let the data show you which model families work best for your specific problem.

Step 2: Manually annotate and actually write things down

This is where the real work happens, and it's the step most teams skip.

Read the AI's responses. Not skim, read. For each one, write down specifically how it failed. Not "wrong answer" or "incorrect." Instead: "It used milligrams instead of micrograms." "It recommended a product that's been discontinued." "It failed to handle the small sample size"

Compare multiple models' responses on the same test case, side by side. Note which responses handle what well and where each one breaks.

This is where domain expertise becomes your biggest differentiator. An engineer might mark a response as "looks fine." A domain expert sees that the AI referenced an outdated regulation, or that the tone is wrong for the audience, or that the recommendation technically works but misses the obvious better option.

Step 3: Iterate and expand

Run more test cases. Add edge cases: the weird inputs, the conflicting requirements, the ambiguous requests. Add adversarial inputs: the things users will actually throw at your AI. Keep annotating. Keep noting failures.

Step 4: Recognize patterns

This is the step that transforms scattered notes into actual insight.

Look across your annotations. What keeps coming up? What are the different categories of failures?

Prioritize these patterns by two dimensions: how often they occur, and how much impact they have on the user. A formatting error that happens frequently but doesn't change the meaning is lower priority than a factual error that's rare but dangerous.

In Lovelaice, we guide teams step by step on how to do this.

Step 5: Improve systematically

For each failure pattern, determine the right fix. Some patterns are prompt improvements: adding context, specifying constraints, providing examples of correct behavior. Some need better context engineering. Some require breaking a complex task into simpler steps. Some need tools, like a calculator for numerical operations instead of relying on the LLM.

The key: each improvement should target a specific, documented failure pattern.

Step 6: Write evals to automate checking for specific errors

Manual annotation and review of AI responses doesn't scale. You need to your checks and validations to measure improvements, error distributions and monitoring the AI feature's quality in production.

For this you need quantitative data.

Write deterministic checks first

This is the step that saves you the most money and gives you the most reliable signal.

Look at each failure pattern and ask: "Can I measure this with a rule?" You'll be surprised how many can be checked deterministically:

• - Does the response include required fields? (String matching)
• - Is the output in the correct format? (Schema validation)
• - Are numerical values within expected ranges? (Range check)
• - Does it reference only approved sources? (Allowlist check)
• - Is the response length within acceptable bounds? (Length check)
• - Does it contain prohibited content or phrases? (Keyword check)

Deterministic checks cost nothing, run in milliseconds, and achieve 95%+ accuracy for what they measure. Research suggests that up to 40% of evaluation criteria can be handled this way. Build these first. They're your foundation.

Write LLM-as-a-Judge but make it surgical, specific, validated

Now — and only now — you've earned the right to use LLM-as-a-Judge.

But not the way most teams do it. Here's what makes the difference between a judge that works and one that gives you false confidence:

Make it specific, not generic. Your judge's criteria should come directly from your documented failure patterns — the ones you couldn't check deterministically. Not "is this helpful?" but "does the response correctly identify all insights from the data?"

Make it binary, not scalar. Don't ask for a 1-10 rating. Ask: "Is this correct or incorrect?" for each specific criterion. Binary judgments are more reliable, more actionable, and easier to validate.

Validate your judge. Run it against your manually annotated examples — the ones where you already know the right answer. If your judge doesn't agree with your human annotations at least 80% of the time, it's not ready. Validated judges reach 80-85% agreement with humans. Unvalidated ones hover at 60-70%. That gap is the difference between a useful quality signal and noise.

Evaluate your judge continuously. Your judge can drift just like your AI feature can. Periodically check it against new human annotations. If agreement drops, update the criteria.

---

The bottom line

If you know you need to write evals for your AI feature, but you don't know where to start, this is your roadmap.

Start by reading real AI outputs. Write detailed notes on the failed responses. Find the failure patterns. Only then your shoudl start automating. Build deterministic checks first and use the LLM-as-a-Judge surgically — for specific, high-impact failure patterns where human judgment was genuinely required and couldn't be captured in a rule.

Organizations that discover AI failures post-deployment spend 10-15x more on fixes compared to those who invest in pre-deployment evaluation. The few days of manual work at the start isn't a cost — it's the highest-ROI investment you can make in your AI feature.

Stop automating evaluation before you understand what you're evaluating. LLM-as-a-Judge is not the first step. It's the last one. And most of what teams want to use it for should be a deterministic check instead.

Your AI judge is only as good as the failure patterns you teach it to look for. And you can only learn those patterns one way: by doing the work yourself first.