An invoice extraction returns valid JSON with the wrong payment term. A schema check passes it because the field is a string, exactly as required. Choosing deterministic evals vs LLM judges for document extraction means deciding which requirement each check can actually establish.
Suppose the source says payment within 45 days and the output says 30. If your offline case has a reviewed reference, a direct comparison may catch the error. A broader question about whether unfamiliar wording preserves the contractual term may need interpretation.
Deterministic evals vs LLM judges: the decision table
| Failure to detect | First check to consider | Boundary of that check |
|---|---|---|
| Response cannot be parsed | Parse the complete response | Establishes syntax |
| Required keys or types are wrong | Validate the agreed schema | Establishes structural compliance |
| Currency is outside the allowed set | Check permitted values | Does not establish the right currency for this document |
| Field differs from a reviewed answer | Compare under approved normalization | Applies to that reference and matching rule |
| Values violate a stated relationship | Apply the documented business rule | Can miss a wrong value that still satisfies the relationship |
| Wording changes the source's meaning | Compare a reference where sufficient; otherwise assess a narrow judge | Requires adequate source evidence and validated criteria |
The table maps failures to checks. One green result does not cover the remaining requirements.
For a fixed reference comparison, agree how dates, whitespace and decimal punctuation are normalized. Preserve the raw values. A transformation that removes every non-digit character can make distinct amounts look identical. The ground-truth guide covers establishing the expected answer.
Which document-extraction failures need an LLM judge?
Use a judge when the requirement needs interpretation your rules and available reference cannot adequately express. Give it a defined failure, relevant source content and exceptions.
A judge checking payment-term support needs the source and the extraction. The final JSON alone cannot establish what the document says. Conflicting sections, missing pages and unreadable text also need a stated route; a persuasive explanation cannot supply absent evidence.
Validate its verdicts against reviewed cases. Inspect false accepts, false rejects and some agreements, because the judge can fail an output for an unrelated reason. The LLM judge validation process supplies that wider loop.
Interpretation may still belong with a domain reviewer while the policy is unresolved. Automating an ambiguous requirement simply moves the ambiguity into another component.
The validator needs examples that try to fool it
A workshop participant asked whether generated evaluation code could be trusted. A section-presence check might accept all required headings while ignoring extra prose or wrong ordering.
Test the check on deliberately passing and failing outputs: JSON surrounded by commentary, missing fields, wrong types, permitted alternatives and plausible incorrect values. State the expected result before running the validator.
Then attach an action to failure. Invalid syntax might lead to a retry; unreadable source material might require a clearer file; an unsupported field might go to review. Repeated retries should not substitute for a policy on missing evidence.
The document automation evaluation workflow combines these checks with field accuracy and downstream behavior. Keep their results separate so the team can see which requirement improved.
Enjoyed reading? Add us as a preferred source in Google to support us.
Add Lovelaice as a preferred source

