A document extraction test dataset can contain 200 invoices and still test only one supplier layout. Add another hundred from that supplier and the new total may hide the same gap: the team has never checked what happens when the table, labels, or date format changes.
The useful question is what those documents let you decide. Are you checking a supported layout, comparing candidate extractors, or deciding whether unfamiliar files can enter the workflow? Each needs a different sample.
What belongs in a document extraction test dataset?
Begin with the product's scope: document types, sources, languages, required fields, and the conditions that need review. Ask the domain expert which files take care during manual processing.
That conversation often produces better cases than a generic list of “edge cases.” A figure may sit beside two reporting periods. A handwritten correction may conflict with a printed amount. An appendix may describe a different entity.
Record the variation you intend to cover. For an invoice workflow, that could include digital files and scans, familiar and unfamiliar suppliers, single-page headers and long line-item tables. A technical-document assistant needs its own categories, such as revision, component, unit, and source section.
The document automation evaluation workflow connects those cases to checks and downstream handling.
The source file is only half the case
An evaluation case needs an agreed interpretation. Store enough for another reviewer to reach or challenge that decision.
| Record | Purpose |
|---|---|
| Stable case ID | Connect results across runs |
| Exact file and version | Preserve the evidence assessed |
| Category and layout family | Reveal coverage and slice results |
| Expected fields or response | State what success means |
| Page or passage evidence | Make the reference inspectable |
| Matching policy | Specify normalization and allowed alternatives |
| Expected handling | Process, abstain, reject, or request review |
| Label review | Record ownership and unresolved questions |
Keep “not labelled yet” separate from “absent in the source.” Otherwise, incomplete review turns into false failures or false passes.
Retain intermediate inputs when they affect the task. If the model receives OCR text or retrieved passages, preserve what it received. A cleaned reference transcription can help reviewers, but it should not silently replace production-like input in the experiment.
Coverage and stress tests answer different questions
A representative sample helps estimate how a configuration behaves on the workload. A deliberately difficult set helps diagnose weaknesses.
Keep both, with their purpose visible. If you construct a set dominated by damaged scans, its overall pass rate describes that set. It should not be presented as the expected rate for a workload dominated by clean digital files.
Report results by useful categories. A stable average can conceal a new failure on a less common format.
Preserve ordinary successful cases too. They tell you whether an edit that fixes a rare document has damaged behavior worth keeping. The golden dataset definition explains the role of these retained references across versions.
How many test documents do you need?
Enough to cover the decision is a better starting point than a universal count. A small varied set can reveal a specification mistake quickly. Establishing a low failure rate, especially for an uncommon consequential error, requires more evidence.
Start by listing uncovered conditions and observed error rates. Add cases where another document is likely to answer an unresolved question. Report the number tested alongside the result and keep the uncertainty visible.
A set can be useful before it is sufficient for a broad release claim. Treat those as different decisions.
Hold out the kind of document you expect to meet later
If unfamiliar suppliers are the challenge, keep some suppliers out of prompt development. Splitting near-identical pages from one template between development and evaluation creates a much easier test.
Other tasks may need a split by layout family, source, or time period. Pick the unit that matches the generalization question.
Once a failed case has repeatedly shaped the prompt, keep it for regression testing. Its result provides less independent evidence about unfamiliar cases. Separate those jobs in the dataset record.
Before launch and after the first real failures
You can start with mock documents that exercise specific rules: an absent date, conflicting totals, or instructions embedded in a PDF. Have a domain expert check that the documents and expected handling are plausible.
When real documents become available, use them to challenge those assumptions. A model-generated source and model-generated answer can repeat the same mistake.
Version changes to cases and labels. Rerun the baseline and candidate on the same version before comparing them. A new score should reflect a product change you can explain, rather than a quiet change in what the test counts.
If you are starting without production records, the first-eval guide gives a smaller initial session to build from.
Enjoyed reading? Add us as a preferred source in Google to support us.
Add Lovelaice as a preferred source
