Eval
Definition
A repeatable test that scores AI output against a definition of good, run across many cases in your dataset. The closest thing to a unit test for a probabilistic system — without it, every iteration is a guess dressed up as a decision.
An eval is a repeatable, automated test that scores AI output against a written definition of 'good,' applied across every case in your dataset. Evals come in two shapes: deterministic checks that a computer can verify unambiguously, and LLM-as-judge scores for aspects that require judgment. A working eval suite is mostly the first, with a few of the second layered on top.
Why it matters
Evals are what turns AI development from opinion into engineering. Without them, teams argue over which output is better; with them, they compare two runs on the same set and read the diff.