How to build an AI evaluation rubric from real outputs

An AI evaluation rubric usually gets written before anyone has looked closely at what the AI produces. Teams pick criteria such as helpfulness, relevance, coherence and tone because those sound like sensible definitions of quality.

Then the AI assigns the wrong priority to a valuable lead, drops a user constraint or claims a product does not exist when it does. The answer still sounds helpful. The rubric still passes it.

In a free Lovelaice lightning session, we built an evaluation rubric from scratch for a lead-classification AI. We started with the outputs, annotated the failures, grouped the annotations into patterns and turned those patterns into checks. This is the method, the worked example and the rubric you can use to do the same for your own AI product.

---

What is an AI evaluation rubric?

An AI evaluation rubric is a set of explicit criteria used to decide whether an AI output is good enough for a specific product and use case. Each criterion describes one observable requirement, the conditions for passing or failing it and, where needed, its severity or weight.

That last part matters. An AI evaluation rubric is not a list of qualities you hope the answer has. It is a quality standard your team can apply consistently to real outputs.

For a lead-classification AI, “the response is relevant” is not a useful criterion. A response can be relevant and still send the best lead in the pipeline to the bottom of the queue. “The assigned tier follows the ICP priority rules” is useful because it tests the decision the product exists to make.

For a support assistant, “the response is helpful” is not enough. “Every policy claim is supported by the supplied knowledge base” tells the reviewer what to check and catches the failure that matters.

The difference is specificity. Generic rubrics describe good writing. Product-specific rubrics describe a job done correctly.

This is also where building AI products departs from conventional software planning. A software team can often enumerate edge cases before implementation because the system follows rules the team wrote. An AI product can fail in ways the team did not specify and did not predict. Across the products and experiments behind this session, the failures we expected upfront were almost never the full set of failures we found in the outputs.

Planning still matters. Guessing every possible failure before you run the product does not. You can spend days writing checks for errors that never occur while leaving the real failure unmeasured.

Why generic AI evaluation criteria fail

Generic criteria look reasonable on a scorecard:

• •Helpfulness — did the answer appear to address the request?
• •Relevance — did the answer stay on topic?
• •Coherence — was the answer easy to follow?
• •Tone — did the answer sound appropriate?

All four can pass while the product fails.

Imagine a shopping assistant that recommends generic products instead of the specific products a customer requested. It drops one constraint on every turn. It says the catalog does not carry an item that is in stock. The conversation remains coherent, relevant and polite throughout.

A generic rubric grades the surface. The customer experiences the failure.

This is not a theoretical edge case. In our benchmark of eight LLM judges across ten models, seven generic evaluation templates averaged 52–58% agreement with a human expert on a balanced dataset, where 50% was a coin flip. The best purpose-built prompt and model pairing reached 76.5%.

The result does not mean every rubric needs to be long. It means the rubric needs to contain your judgment. No vendor template already knows what a dangerous mistake looks like in your HealthTech workflow, what makes a lead valuable to your sales team or which policy detail your support assistant must never invent.

If you remember one thing from this guide:

Do not start by asking, “What should a good answer look like?” Run the feature first and ask, “What did these answers get wrong?” Your failures will give you sharper criteria than a blank page ever will.

---

The four-step method, if you only skim

1. •Run one prompt across several models and representative test cases. Keep the inputs fixed so the outputs are comparable.
2. •Read the outputs and annotate them by hand. Record exactly what worked, what failed and why. “Bad answer” is not an annotation.
3. •Group repeated annotations into error patterns. Name each pattern, count it and decide how serious it is.
4. •Turn each pattern into an evaluation criterion. Use a deterministic check for rules a machine can verify exactly and an LLM judge for contextual judgment.

Then run the next prompt version against the same test cases. The rubric scores the iteration and shows whether you fixed the failure or merely moved it somewhere else.

That is the loop:

Experiment → annotate → find patterns → encode the rubric → run again.

The rubric serves two jobs in that loop. It becomes the specification for automatic evaluation, and it tells you what the next version of the prompt needs to fix.

---

Step 1: Run the same prompt across models and test cases

Start with one prompt and a set of test cases that represent the job your AI needs to perform. Run every case through each model you are considering. Keep the prompt, inputs and model settings consistent.

You are not picking a winning model yet — you are building a set of outputs wide enough to expose different failure modes on the same inputs.

One model may follow the requested format but make the wrong decision. Another may make the right decision but invent evidence. A third may handle the obvious cases and fail whenever two instructions conflict. You will not see that range from three happy-path examples on one model.

For the lightning session, we used a lead-classification AI. A prospect submits an email address, company and short message. The AI validates the information, researches the person and company, compares the lead with the ideal customer profile and assigns a priority tier from 1 to 4.

The output looks simple: a tier, the reasoning, the information found and the sources used. The judgment underneath it is not simple at all.

A useful first dataset should contain:

• •obvious successes;
• •obvious failures;
• •ambiguous cases where domain judgment matters;
• •missing or contradictory input;
• •cases that previously caused customer or reviewer complaints;
• •high-cost failures, even when they are rare.

You do not need hundreds of cases to start. Thirty well-chosen outputs are enough to reveal patterns and build the first useful version of a rubric. Expand the dataset as the product meets new situations.

Step 2: Read every answer and annotate what happened

This is the part teams try to skip. It is also where the quality standard comes from.

Open the responses one by one. Compare the answer with the input, source material and expected product behavior. Write down what was good, what broke and why it matters.

Weak annotation:

The answer is not good enough.

Useful annotation:

The company matches the target ICP, but the response assigned Tier 2 because the contact is not a buyer persona. Our prioritisation rule says any contact at a target company is Tier 1.

The second note contains a failure, the rule it violated and enough context to turn it into a check.

Annotate the decision, not just the prose. Fluent answers are dangerous because they make wrong decisions look finished. Verify claims against the source. Check whether the model followed every constraint. Read the reasoning behind a correct answer too; a right score reached for the wrong reason will not generalise to the next case.

When several people review outputs, ask them to leave written reasons rather than only pass/fail labels. A label tells you that the reviewer disagreed. The reason tells you what belongs in the rubric.

This manual review is error analysis for an AI feature. It is slow on purpose. You are converting domain expertise from something people recognise instinctively into something the team can repeat, measure and automate.

Step 3: Group annotations into failure patterns

Individual comments tell you what happened once. Error categories tell you what to fix.

After the review, zoom out. Group annotations that share the same underlying failure. Give each category a precise name, count how often it appears and mark its severity.

For example:

The categories should be specific enough that two reviewers understand the same thing by them. They should also point toward a fix.

“Low quality” does neither. “Assigned the wrong priority tier” points to the decision rules in the prompt. “Dropped a stated user constraint” points to instruction tracking. “Narrated a tool call it never made” points to tool-use handling and response policy.

Do not force every problem into one category. If the same answer contains a wrong decision and an unsupported claim, record both. The purpose is to understand the product, not to make a tidy chart.

Then prioritise. Frequency matters, but severity matters more. A formatting mistake that appears 20 times may be less important than one fabricated medical contraindication. Your rubric should represent the cost of failure, not just the count.

Step 4: Turn each error pattern into an evaluation criterion

Now write the rubric. Each repeated failure becomes a criterion that can be applied to the next set of outputs.

A strong criterion contains four things:

1. •The requirement — what must be true.
2. •The evidence — what the evaluator should inspect.
3. •The pass/fail boundary — what counts as success and failure.
4. •The severity — whether failure blocks release or reduces the score.

Here is what that looks like for the lead-classification AI:

Notice what is missing: helpfulness, relevance and coherence. Those qualities are not worthless. They are simply too broad to lead this rubric. The product exists to classify leads correctly, using supported evidence, in a format the next system can use. That is what the rubric measures first.

Write criteria that two reviewers can apply consistently

Avoid words such as “good,” “appropriate,” “sufficient” and “high quality” unless the rubric defines them. They ask each reviewer to import a private standard.

Compare these two versions:

Vague: The answer should provide sufficient reasoning.

Testable: The explanation must name the ICP signal or disqualification rule that determined the assigned tier. A list of company facts without a connection to the tier fails.

The testable version reduces reviewer drift and gives an automated evaluator a much better instruction.

Separate must-have criteria from quality improvements

Not every failure deserves the same weight.

A must-pass criterion protects the core job, safety, factual grounding or a downstream system requirement. One failure can be enough to fail the whole output.

A should-pass criterion improves usefulness but does not make the output unusable on its own. These criteria can contribute to a weighted score.

For the lead-classification AI, a wrong tier is a release-blocking error. A slightly repetitive explanation is not. A rubric that averages the two together can hide the failure that costs the business money.

Which rubric criteria should be automated?

Once the rubric exists, decide how each criterion should be checked. Use the simplest evaluator that can reliably detect the failure.

Use deterministic checks for exact rules

Deterministic evaluations return the same answer every time and are ideal for requirements that code can verify directly:

• •valid JSON;
• •required fields present;
• •value belongs to an allowed set;
• •response stays under a length limit;
• •all requested items appear;
• •a citation URL is present;
• •no forbidden phrase or PII pattern appears.

Do not pay an LLM to decide whether JSON parses.

Use an LLM judge for contextual judgment

An LLM judge is useful when the evaluator must interpret meaning:

• •whether the assigned tier follows a set of nuanced business rules;
• •whether the answer preserves the user’s constraints;
• •whether the reasoning is supported by the supplied evidence;
• •whether a summary omitted a decision-changing detail;
• •whether a response applies a policy correctly to an ambiguous case.

But turning a rubric criterion into an LLM judge does not make it trustworthy. The judge is another AI product. Validate it against human-labeled examples, compare the written reasons as well as the scores and test the prompt across models. Our guide to validating an LLM judge covers that process.

Keep human review for unresolved judgment

Some cases remain ambiguous because the business rule itself is ambiguous. Do not automate disagreement that the team has not resolved. Route those cases to a domain expert, record the decision and use it to improve the next rubric version.

Human judgment does not disappear from an evaluation system — it moves to the cases where resolving one output produces a standard the team can apply to every future one.

A reusable AI evaluation rubric template

Use one row per observable requirement:

The examples are part of the rubric. They resolve ambiguity faster than another paragraph of abstract instruction.

Start with the most frequent and expensive failures from your error analysis. A first rubric with five precise criteria is more useful than a 25-row template copied from another product.

How to know whether your AI evaluation rubric is working

A rubric is working when humans can use it consistently, its automated checks agree with trusted human labels and its results lead to product decisions.

Test it in three stages.

1. Check agreement between human reviewers

Give the same outputs and rubric to two people who understand the use case. Compare both the verdicts and their reasons. Disagreement usually exposes a vague criterion, a missing rule or an ambiguous case the product team has never decided.

Do not solve disagreement by averaging it away. Fix the standard.

2. Validate every automated evaluator

Build a small golden dataset of outputs that humans have labeled with written justifications. Include both good answers and known failures. Measure missed errors and false alarms, not only overall agreement.

A check that passes everything looks accurate on an easy dataset, and a check that fails everything looks accurate on a failure-heavy one. The direction of the error tells you whether the dashboard is producing false confidence or noise.

3. Re-run the same test cases after every change

Use the same dataset to compare prompt versions and models. When one criterion improves, inspect the others for regressions.

The next prompt can fix invented content and start omitting required details. A model change can improve the average score and fail the highest-risk case. One aggregate number will not show either problem.

Keep the rubric versioned alongside the prompt, model and dataset. When your product rules change, the rubric changes too.

The mistakes that make an AI evaluation rubric useless

Writing the rubric before reviewing outputs

You will encode imagined failures and miss the failures the product produces. Start with annotation.

Scoring writing quality instead of task success

Fluency is not correctness. Measure the decision, action or transformation the feature exists to perform.

Combining several requirements into one criterion

“The answer is accurate, complete, relevant and well-written” gives you no idea what failed. Split it into observable checks.

Using a 1–5 scale without anchors

One reviewer’s 3 is another reviewer’s 4. Define what each score means with passing and failing examples, or use pass/fail where the requirement is binary.

Giving every error the same weight

A polished answer with the wrong decision is still wrong. Use must-pass criteria for failures that make the output unsafe or useless.

Automating the rubric before humans agree

An LLM judge cannot settle a product decision your own experts have not made. Resolve the standard first. Automate second.

Tracking the score but not the reason

A score on its own tells you something failed; the reason attached to it tells you what to change. Keep both.

What building the rubric does not finish

The rubric is one part of an AI evaluation system. A complete loop also needs:

1. •a prompt and context strategy;
2. •representative test data;
3. •manual annotation and error analysis;
4. •a product-specific rubric;
5. •validated deterministic checks and LLM judges;
6. •repeated experiments that compare accuracy, cost and latency;
7. •regression testing after every meaningful change.

Building the rubric gives your team a definition of quality grounded in evidence. The next job is proving that the automatic evaluations apply that definition reliably and that a prompt or model change improved the product without creating a new failure elsewhere.

That is how AI product development moves past Ship and Pray.

Rubrics for multi-agent systems

Use the same evaluation loop for each step of the multi-agent system, then evaluate the assembled workflow end to end.

If one agent researches a company, a second classifies the lead and a third drafts the sales handoff, do not rely on one final “was the answer good?” score. Build criteria for the job each agent performs:

• •Did the research agent find supported information and preserve its sources?
• •Did the classification agent apply the correct ICP and tiering rules?
• •Did the handoff agent carry the decisive evidence into the sales report?
• •Did the complete workflow deliver a report sales can act on?

This tells you where the system failed. The final report may contain a wrong tier because the research was incomplete, because the classifier misapplied a rule or because context was lost between agents. One end-to-end score cannot distinguish those causes.

Run the prompt, annotation, pattern and rubric loop at component level first. Then put the components together and add end-to-end criteria for handoffs, state and final task success.

Frequently asked questions about AI evaluation rubrics

What is the difference between an AI evaluation rubric and an eval?

The rubric defines what good means for the product. An eval applies one part of that definition to an output. A rubric may contain several evals: deterministic checks for exact requirements, LLM judges for contextual criteria and human review for unresolved cases.

How many criteria should an AI evaluation rubric have?

Start with the smallest set that covers the core job and the main failure patterns. Five to ten precise criteria are usually more useful than a long generic checklist. Add criteria when new, repeated or high-severity failures appear.

Should an AI evaluation rubric use pass/fail or a score?

Use pass/fail for requirements with a clear boundary, such as correct classification, supported claims or valid schema. Use anchored scores when quality genuinely exists on a continuum. Never use a numeric scale without describing what each level means.

Can an LLM create the rubric automatically?

An LLM can cluster annotations and draft criterion wording. It cannot decide what quality means for your users without your domain judgment. Humans must review real outputs, confirm the failure patterns and own the final standard.

Can I use the same rubric for different AI models?

Yes, when the models perform the same product job. Keeping the rubric and test cases fixed makes model comparison useful. Inspect individual criteria and high-risk cases, not only the average score.

Do multi-agent AI systems need separate rubrics?

Each agent needs criteria for its own job, and the complete workflow needs end-to-end criteria. Component rubrics locate failures; the end-to-end rubric checks whether the whole system completed the user’s task and preserved the right information between steps.

How often should the rubric be updated?

Update it when product requirements change, new failure patterns appear or the AI gains new tools and responsibilities. Re-validate automated evaluators after every rubric change.

Build backward from the failure

The strongest AI evaluation rubrics grow out of the outputs the team has already reviewed, not out of a blank document.

Run the prompt wide. Read the answers closely. Annotate the exact failures. Group them into patterns. Turn those patterns into checks. Then run the same cases again.

Your rubric should sound like your product, your users and the decisions your team is responsible for. If it could be copied unchanged into any AI feature, it is not finished.

You cannot score what you have not seen.