Experiment
Also known as: Run
Definition
One full pass of a given prompt, model, and configuration over your whole golden dataset — scored and saved. The unit of progress in AI development. Comparing two experiments is the only honest way to say an iteration made things better.
An experiment (or 'run') is one complete evaluation of a specific configuration — a chosen model, prompt, and settings — against the golden dataset, with scores saved alongside inputs and outputs. Every meaningful decision in AI development happens by comparing experiments: this prompt vs that one, this model vs that one, this retrieval strategy vs that one.
Why it matters
If iterations aren't versioned experiments, they're anecdotes. 'It seems better now' isn't a decision; 'run 34 beats run 33 by 8 points on the same set, with three regressions in the extractive cases' is.
Related terms