AI evals for document automation need to catch a result that looks ready to use: a supplier invoice returned as valid JSON, with the subtotal entered as the amount payable. Every required key can be present. The response can arrive quickly. The next system can still receive the wrong amount.

A useful evaluation follows the value through the workflow. It asks whether the source was readable, whether the right field was selected and what happens when the result reaches a record, a reviewer or an automated action. That gives the team several places to detect the mistake before it becomes someone else's problem.

Start at the action the document enables

Take an invoice workflow that prepares records for an operations reviewer. Its job is to return the supplier, invoice number, currency, amount payable and due date, with evidence the reviewer can inspect.

That description already contains decisions. The amount payable must refer to the final amount due. A missing date needs an agreed representation. The system prepares a record; it has no authority to approve payment.

Write these decisions before choosing a score. If one reviewer expects copied text and another expects normalized values, their labels will disagree even when they read the same source. If engineering interprets a null as “no review needed,” the workflow can fail after the extraction itself behaved as requested.

The task contract should identify the required result, allowed transformations, missing-information policy and permitted next action. It becomes the common reference for the prompt, checks and receiving application.

How do you choose cases for AI evals in document automation?

Begin with documents that differ in ways the workflow will encounter: supplier layout, number of pages, language, scan quality and table structure. Repeating one clean template tests repetition of that condition. It supplies little evidence about a different template.

Add cases that force a decision. An empty file tests intake. A revised invoice tests document identity. Two totals test field selection. An unreadable date tests whether uncertainty survives into the review record. Text asking the model to ignore its instructions tests whether document content can redirect the task.

For each case, save the exact file, the expected behavior and the evidence supporting the reference. A constructed document is useful when it exposes a missing requirement; review its contents and expected answer before relying on it.

Keep case counts and coverage together. A diverse first set helps discover failures, while estimating a rare error rate needs additional evidence. Record what the set cannot yet tell you.

Document automation checks belong at several handoffs

The invoice subtotal error suggests a field-selection check, but it should not be the only check in the run.

HandoffRequirementEvidence of a failure
File to processing stepExpected pages are present and readableThe final page is missing
Model to parserResponse follows the agreed schemaAmount is returned as free text
Source to extracted fieldValue has the requested meaningSubtotal becomes amount payable
Extracted field to evidenceFile, version and passage support the valueQuote comes from an older invoice
Result to review recordUnresolved fields stay visibleAn unreadable date becomes an empty, unflagged cell
Record to actionOnly permitted actions occurA preparation task initiates an approval

Report these results separately. Several easy format passes should not conceal a wrong payable amount.

Use code where an explicit rule can decide the requirement. A reviewed reference can establish whether a known test value matches. A judgment about ambiguous evidence may need a domain reviewer or a validated model judge. The choice follows the failure you need to detect.

Read the failure before extending the prompt

Put the output beside its source and record the exact mismatch. “Selected the subtotal row” is a useful note. “Bad extraction” leaves the team guessing.

Then locate the step that introduced the defect. If the source text already lost a decimal separator, investigate document processing. If the correct rows reached the model but it selected the wrong one, inspect the field definition and instructions. If the extraction was right and the receiving application altered it, fix that handoff.

This prevents prompt revisions from becoming a collection of instructions for problems caused elsewhere. Group recurring notes into categories that support a decision: change the source preparation, clarify a requirement, add a check or route a case for review.

The field-level evaluation guide shows how to turn those comparisons into counts without losing the individual field failures.

A release record should explain the exceptions

Run the current and candidate configurations on the same dataset version. Include prompt, model, parsing steps and checker versions in the result. Inspect newly failed cases as well as improvements.

Decide what blocks release and what can enter an explicit review path. For the invoice workflow, an unresolved currency conflict might require review regardless of the overall pass rate. That policy needs to be tested on cases where a conflict is present.

A compact release record names the task, cases covered, remaining failure categories, review rules and person accepting the result. Add elapsed time and operating cost, including retries and review. The cheapest model call may create more repair work downstream.

The golden dataset preserves the references for the next comparison. Keep a separate record when the expected behavior changes, so a policy revision cannot masquerade as a model improvement.

After launch, inspect apparent successes too

Reported failures are only part of the evidence. Sample records that passed the automated checks and compare them with their source files. A checker for required keys will not discover that the wrong total was copied into one of them.

Retain verified new failures as regression cases. Add unfamiliar document conditions to the coverage record and reassess the review load as usage changes.

The workflow is ready for another iteration when the team can name a specific defect, show where it entered and rerun a check for it. That is a more useful outcome than a single green percentage with no explanation of what it measured.