This is the document we wish someone had handed us in 2024.
The AI evaluation category has produced a flood of blog posts, comparison tables, framework documentation, and SDK tutorials. None of them are written for the domain expert — the clinician on the healthtech team, the compliance officer at the neobank, the underwriter at the insurer, the lawyer reviewing the legaltech assistant. The category assumes the domain expert hands their judgment to an AI engineer who then encodes it into a scoring function.
That assumption is the bug.
This playbook is for the people who hold the judgment. It does not assume you know what a prompt is. It does not assume you have access to a Python REPL. It does not assume you have ever written a pytest fixture. It does assume you know your domain — clinical accuracy, regulatory compliance, financial correctness, legal precedent — better than anyone else in your organization. The job of this playbook is to give you a workflow for encoding that judgment into evaluations that the rest of the AI team can run, audit, and prove.
Twelve chapters. No marketing CTAs. Read it once, cite it forever.
Chapter 1: Why the domain expert owns the rubric
Evaluation is not a measurement problem. It is a judgment problem.
Every AI feature has to make decisions that range from trivially correct (the text contains the word "Tuesday") to deeply context-dependent (the clinical summary correctly identifies prior-authorization criteria for a specific payer). The trivially correct decisions can be evaluated by code. The deeply context-dependent decisions require domain judgment. There is no model, no LLM-as-judge, no benchmark, no automated scoring framework that can encode that judgment without input from someone who holds it.
When AI teams skip the domain expert, the evaluations end up measuring proxies. Faithfulness as a stand-in for clinical accuracy. Relevance as a stand-in for regulatory citation correctness. BLEU as a stand-in for legal-prose precision. Every proxy is wrong in the cases that matter most — the edge cases, the ambiguous ones, the ones a regulator would flag.
The domain expert does not need to write code to own the rubric. The domain expert needs to write the rubric in their own language, with examples, and the platform translates it into something the AI team can run.
Chapter 2: What makes a good rubric
A good rubric has four properties.
It is specific. "The answer should be correct" is not a rubric. "The summary cites at least one ICD-10 code if the source mentions a diagnosis, and the code matches the diagnosis" is a rubric.
It is decidable. A human reading the input and the output should be able to answer yes/no without hedging. If two reviewers disagree more than 10% of the time, the rubric is too vague.
It is enforceable. The platform should be able to apply the rubric automatically once you have shown it 20–50 calibration examples. If your rubric requires the reviewer to re-interpret each case from scratch, you have not written a rubric — you have written a job description.
It is audited. Every rubric criterion should have a one-line answer to "why does this matter." Often the answer is a regulatory citation, a customer-impact story, or a downstream consequence. "Why does this matter" is the question a regulator will ask if the rubric ever ends up in court.
Chapter 3: The three layers of every rubric
Every working rubric we have ever helped a team build has three layers. Skipping any layer produces an evaluation that misses real failures.
Layer 1: Factual correctness. Did the AI get the facts right. The diagnosis is accurate. The interest rate is correct. The policy clause is properly cited. The patient ID is preserved. The numbers add up. This layer is the easiest to write and the most likely to be the only thing the engineering-led eval suite tracks.
Layer 2: Domain appropriateness. Did the AI behave the way a competent practitioner in your domain would. The medical summary recommends a follow-up rather than asserting a treatment plan. The financial assistant declines to project portfolio returns. The legal assistant flags rather than answers ambiguous case law. This layer is where domain expertise shows up. It is also where 80% of the customer complaints come from.
Layer 3: Communication quality. Did the AI communicate appropriately for the audience. Tone is correct. Reading level is appropriate. Disclaimers are present where required. Forbidden phrases are absent. This layer is the most "soft" and the easiest to test, because tone failures are visible to non-experts.
A complete rubric scores against all three layers, separately, with separate thresholds. Aggregating them into a single number is what produces the "we are at 91% accuracy" claim that is meaningless to a regulator.
Chapter 4: How to build the golden dataset
The golden dataset is the set of inputs you score every release candidate against. Build it before you write a single line of code.
The minimum is 50 inputs. The minimum-useful is 200. The teams that produce reliable AI features in regulated industries have 500–2,000 in their golden set across the full lifecycle.
Composition matters more than size. The 50-input minimum should break down roughly as:
- •20 representative cases (what your typical user actually sends)
- •10 adversarial cases (what your worst user, your worst customer, or your worst regulator-scenario would send)
- •10 edge cases (rare but high-consequence — the cases where the cost of getting it wrong is asymmetric)
- •5 refusal cases (the AI should say no)
- •5 hand-crafted "honey trap" cases (designed to catch specific known failure modes)
The 20 representative cases come from production logs, support tickets, or customer interviews. The 10 adversarial cases come from your security and compliance team. The 10 edge cases come from you — the domain expert. The 5 refusals come from your legal and compliance review. The 5 honey traps come from your incident postmortems.
If you cannot point to where each of the 50 inputs came from, the dataset is not golden. It is a guess.
Chapter 5: Calibration sessions
The platform learns your rubric from calibration. A calibration session is a 90-minute working session in which you (the domain expert) score the same 30–50 outputs the platform will be scoring, and the platform compares your scores against its own.
You do this twice. Once with the AI team in the room — to align on the rubric's criteria and resolve ambiguous edges. Once alone — to ensure the platform's scoring matches yours without influence.
Calibration is not optional. Without it, you are accepting whatever an off-the-shelf LLM-as-judge thinks "faithful" means. With it, you are encoding your domain into the platform's scoring. The difference shows up in the cases that matter most.
A team is calibrated when the platform's scores and the domain expert's scores agree at least 95% of the time on the calibration set. If you cannot get to 95%, the rubric is too vague — go back to Chapter 2.
Chapter 6: What to measure (and what to stop measuring)
The category is full of vendors selling scorecards with 50+ metrics. Most of those metrics are noise.
For most regulated AI features, the metrics that matter are:
Pass rate by rubric layer. Three numbers: factual correctness, domain appropriateness, communication quality. Each independently. Aggregated only when reporting to executives.
Refusal precision. Of the cases the AI refused, what percentage SHOULD have been refused. Inversely: of the cases the AI answered, what percentage should have been refused.
Domain-segmented failure rate. Pass rate on the riskiest 10% of your dataset (the high-consequence edge cases). This is usually the metric that catches incidents.
Stability under model swap. When the engineer changes the model, what is the delta on pass rate by layer. If your stability is poor, you cannot swap models without re-validation. In regulated industries, you cannot swap models on a deadline at all unless this metric is well-understood.
What to stop measuring: aggregated quality scores, generic LLM-as-judge scores untethered to your domain, BLEU, ROUGE, METEOR. None of these will move when something actually breaks.
Chapter 7: Pre-ship vs post-ship evaluation
Both happen. They serve different purposes.
Pre-ship evaluation runs the golden dataset against a release candidate before the candidate is deployed. The result is a release decision: ship, hold, or revise. The data is generated by the candidate. The rubric is locked. The audience is the PM, the domain expert, and the compliance officer.
Post-ship evaluation runs the rubric against production traffic continuously. The result is a drift signal: are we still passing the same rubric we passed before launch. The data is generated by real users. The rubric is the same as pre-ship. The audience is the PM, with escalation to compliance if drift crosses threshold.
Most teams do one or the other. The teams that survive do both. Pre-ship is what prevents incidents. Post-ship is what catches the ones you did not anticipate.
The mistake is doing only post-ship — what the observability category sells you. By the time post-ship detects a problem, the problem has affected customers. In regulated industries, that means you have to disclose it.
Chapter 8: When your rubric disagrees with the engineer
This will happen. The engineer will say "the model's response is correct — it answered the user's question." You will say "the response is wrong — it should have refused that question entirely." You will both be right within your own frame.
The domain expert wins this disagreement. Always.
This is the most important principle in the playbook. AI features fail in regulated industries because the engineer's definition of "working" is too narrow. The engineer's frame is "does the model do what the prompt asked." The domain expert's frame is "would a competent practitioner do this." The second frame is the one the customer, the regulator, and the courtroom care about.
When the rubric and the prompt disagree, change the prompt. When the rubric and the engineer disagree, the engineer changes the prompt. When the rubric and the model's "natural" behavior disagree, lock the rubric and ask whether you are using the right model.
The rubric is the contract. Everything else is implementation.
Chapter 9: Documentation for audit
The rubric, the golden dataset, the calibration logs, and the per-release evaluation reports together form the audit packet. In a regulator inquiry, this is the document you hand over.
Each component must be:
Versioned. The rubric you used in March must be retrievable in November. So must the dataset. So must the calibration log.
Signed. The domain expert signs the rubric. The PM signs the release candidate. The compliance officer signs the audit packet for each release.
Reproducible. The platform should be able to re-run the exact evaluation that produced the release report on demand. If the model has changed, the platform should flag the discrepancy.
Tied to a change log. Every prompt change, every rubric change, every dataset addition gets a one-line entry. The entry is in plain language and is readable without engineering translation.
A team with these four properties can respond to a regulatory inquiry in days. A team without them takes months and often discovers they cannot answer the question at all.
Chapter 10: Handling production failures
When a production failure shows up — a customer complaint, an internal review, a regulatory inquiry — the workflow is:
- •Reproduce the input. What did the user actually send.
- •Re-run against the current rubric. Does the rubric catch the failure. If yes, the rubric is fine; the platform missed the case in pre-ship. If no, the rubric has a gap.
- •If the rubric has a gap, write a new criterion. Add 5–10 new inputs that exercise the same failure mode. These become a new section of the golden dataset.
- •Re-evaluate every previous release against the new criterion. Yes, every one. This is how you discover whether the failure has been present in production longer than you realized.
- •Update the audit packet. Document the gap, the fix, and the impact.
The fifth step is the one most teams skip. The cost of skipping it is that the next regulator inquiry asks "when did this start" and the team has to guess.
Chapter 11: When to bring in an external evaluator
For most rubrics, the domain expert on staff is sufficient. For three specific cases, you should bring in an independent reviewer.
Pre-launch in a new regulatory jurisdiction. If you are launching a feature into a new market with different rules, get an external reviewer to validate the rubric against the new rules. Internal experts can have blind spots from their home market.
Post-incident. After a customer-impacting failure, an independent reviewer should validate that the fixed rubric actually catches the failure and is not just measuring around it.
Pre-procurement audit. When a large customer or partner is going to audit your AI feature, an independent reviewer's report carries weight your internal report does not.
In all three cases, the independent reviewer scores the same calibration set and produces a written report. Their report becomes part of the audit packet.
Chapter 12: When evaluation is not enough
Some AI failures cannot be solved by evaluation. They have to be solved by scope.
If your evaluation reveals that the model fundamentally cannot handle a category of input — for example, the medical-summary model cannot reliably distinguish between two similar diagnoses with different treatment paths — the answer is not "improve the rubric." The answer is "the feature should not handle this category. Route it to a human."
The domain expert is the only person who can make this call. The engineer will try to fix it with prompt engineering. The PM will try to fix it with scope creep. The compliance officer will not know to push back. You have to be the one who says "this is out of scope for the AI. Send it to a human."
Saying no to scope is the senior version of the domain expert's job. It is also the one that prevents the worst incidents.
How to use this playbook
Read it once front-to-back. Pick a current AI feature you own. Walk through chapters 2-7 for that feature. Run a real calibration session. Build a 50-input golden dataset. Run the rubric against your current prompt. Watch what fails.
Then keep the document at hand. Cite specific chapters when the AI team pushes back on your judgment. The chapters are designed to be quotable in a Slack thread without the entire team having to read the playbook.
If you find a chapter that is wrong for your domain — fintech, healthtech, insurance, legaltech, education — write to us. We update this playbook quarterly based on field feedback. Domain-specific appendices live alongside the main playbook for fintech, healthtech, and insurance. More verticals coming as we work with more teams.
This is the playbook the category should have written three years ago. It did not. We are writing it now.
[NEEDS INPUT: Domain-specific appendices for fintech (KYC, AML, payments, lending), healthtech (clinical summary, triage, prior auth, claims), insurance (underwriting, claims processing, fraud detection), and legaltech (contract review, discovery, research). Each appendix should be 1,500-2,500 words and include a worked golden dataset example.]
[NEEDS INPUT: PDF download version, table-of-contents anchor links, and a printable A4 version for circulation inside enterprise teams. This is the asset that should be in compliance officer Slack channels.]
[NEEDS INPUT: One worked calibration session walk-through with screenshots from Lovelaice. Domain: TBD with input from Madalina and Catalina — recommend healthtech triage as the most visceral demonstration.]
Enjoyed reading? Add us as a preferred source in Google to support us.
Add Lovelaice as a preferred source
