Deterministic eval
Definition
A check with one right answer that a computer can verify: exact match, valid JSON, a number within tolerance, a required field present, a forbidden phrase absent. Cheap, instant, reproducible, and it never disagrees with itself.
Deterministic evals are checks with a single verifiable right answer. Exact match. Valid JSON. A number within tolerance. A required field present. A forbidden phrase absent. Use them for everything that can be checked mechanically before reaching for anything fancier.
Why it matters
They handle roughly 40% of what a real eval suite needs to cover, at essentially zero cost per run. The limit is obvious: they don't work when the aspect of quality requires judgment — 'is it friendly,' 'did it prioritize correctly.' Those are what LLM-as-judge is for.
Related terms