An invoice extraction returns valid JSON with the wrong payment term. A schema check passes it because the field is a string, exactly as required. Choosing deterministic evals vs LLM judges for document extraction means deciding which requirement each check can actually establish.

Suppose the source says payment within 45 days and the output says 30. If your offline case has a reviewed reference, a direct comparison may catch the error. A broader question about whether unfamiliar wording preserves the contractual term may need interpretation.

Deterministic evals vs LLM judges: the decision table

Failure to detectFirst check to considerBoundary of that check
Response cannot be parsedParse the complete responseEstablishes syntax
Required keys or types are wrongValidate the agreed schemaEstablishes structural compliance
Currency is outside the allowed setCheck permitted valuesDoes not establish the right currency for this document
Field differs from a reviewed answerCompare under approved normalizationApplies to that reference and matching rule
Values violate a stated relationshipApply the documented business ruleCan miss a wrong value that still satisfies the relationship
Wording changes the source's meaningCompare a reference where sufficient; otherwise assess a narrow judgeRequires adequate source evidence and validated criteria

The table maps failures to checks. One green result does not cover the remaining requirements.

For a fixed reference comparison, agree how dates, whitespace and decimal punctuation are normalized. Preserve the raw values. A transformation that removes every non-digit character can make distinct amounts look identical. The ground-truth guide covers establishing the expected answer.

Which document-extraction failures need an LLM judge?

Use a judge when the requirement needs interpretation your rules and available reference cannot adequately express. Give it a defined failure, relevant source content and exceptions.

A judge checking payment-term support needs the source and the extraction. The final JSON alone cannot establish what the document says. Conflicting sections, missing pages and unreadable text also need a stated route; a persuasive explanation cannot supply absent evidence.

Validate its verdicts against reviewed cases. Inspect false accepts, false rejects and some agreements, because the judge can fail an output for an unrelated reason. The LLM judge validation process supplies that wider loop.

Interpretation may still belong with a domain reviewer while the policy is unresolved. Automating an ambiguous requirement simply moves the ambiguity into another component.

The validator needs examples that try to fool it

A workshop participant asked whether generated evaluation code could be trusted. A section-presence check might accept all required headings while ignoring extra prose or wrong ordering.

Test the check on deliberately passing and failing outputs: JSON surrounded by commentary, missing fields, wrong types, permitted alternatives and plausible incorrect values. State the expected result before running the validator.

Then attach an action to failure. Invalid syntax might lead to a retry; unreadable source material might require a clearer file; an unsupported field might go to review. Repeated retries should not substitute for a policy on missing evidence.

The document automation evaluation workflow combines these checks with field accuracy and downstream behavior. Keep their results separate so the team can see which requirement improved.