This is the document we wish someone had handed us in 2024.

The AI evaluation category has produced a flood of blog posts, comparison tables, framework documentation, and SDK tutorials. None of them are written for the domain expert — the clinician on the healthtech team, the compliance officer at the neobank, the underwriter at the insurer, the lawyer reviewing the legaltech assistant. The category assumes the domain expert hands their judgment to an AI engineer who then encodes it into a scoring function.

That assumption is the bug.

This playbook is for the people who hold the judgment. It does not assume you know what a prompt is. It does not assume you have access to a Python REPL. It does not assume you have ever written a pytest fixture. It does assume you know your domain — clinical accuracy, regulatory compliance, financial correctness, legal precedent — better than anyone else in your organization. The job of this playbook is to give you a workflow for encoding that judgment into evaluations that the rest of the AI team can run, audit, and prove.

Twelve chapters. No marketing CTAs. Read it once, cite it forever.

Chapter 1: Why the domain expert owns the rubric

Evaluation is not a measurement problem. It is a judgment problem.

Every AI feature has to make decisions that range from trivially correct (the text contains the word "Tuesday") to deeply context-dependent (the clinical summary correctly identifies prior-authorization criteria for a specific payer). The trivially correct decisions can be evaluated by code. The deeply context-dependent decisions require domain judgment. There is no model, no LLM-as-judge, no benchmark, no automated scoring framework that can encode that judgment without input from someone who holds it.

When AI teams skip the domain expert, the evaluations end up measuring proxies. Faithfulness as a stand-in for clinical accuracy. Relevance as a stand-in for regulatory citation correctness. BLEU as a stand-in for legal-prose precision. Every proxy is wrong in the cases that matter most — the edge cases, the ambiguous ones, the ones a regulator would flag.