Golden dataset
Also known as: Test set, Eval set
Definition
A curated, stable set of real inputs paired with ground truth — the reference set you re-run after every change. Usually 50–200 cases, deliberately including the ugly edge cases rather than the demo-friendly ones.
A golden dataset is a curated, stable set of real inputs paired with their ground truth: the reference set you re-run after every change. 'Golden' means it's the version everyone agrees is correct, and it doesn't quietly drift. It's usually 50–200 cases, not thousands, and it should include the ugly edge cases rather than the demo-friendly ones. Every experiment — every prompt tweak, every model swap — is scored against this set.
Why it matters
Building one is the highest-leverage day of work most teams do. It's the artifact that turns 'seems fine' into a number, and every iteration decision — model, prompt, tool — is settled against it. Teams without a golden dataset argue in circles; teams with one just read the diff between two runs.