Suppose a judge agrees with reviewers on 70 of 80 outputs, yet approves six defective extractions. The 87.5% agreement score is correct. An LLM judge confusion matrix shows what that score leaves out: which mistakes passed, which acceptable outputs were rejected, and what those decisions would do to the workflow.

Before reading the matrix, check the label convention. Here, positive means an error is present. A false negative is a defective output that the judge passes; we call it a false accept. A false positive is an acceptable output that the judge rejects, or a false reject. Some tools use the opposite convention. Put the meanings beside the report.

The LLM judge confusion matrix, filled in

Take a judge checking whether an extracted payment term is supported by the invoice. A reviewer has labeled 40 outputs acceptable and 40 defective. The judge produces these decisions:

Reviewed referenceJudge says failJudge says passTotal
Defective output34 correctly rejected6 false accepts40
Acceptable output4 false rejects36 correctly accepted40
Total384280

The agreement calculation counts the diagonal: 34 correct rejections plus 36 correct acceptances, divided by 80. That gives 70 ÷ 80 = 87.5%.

An agreement report should retain all four cells. The six false accepts may let unsupported payment terms continue into another system. The four false rejects may create unnecessary review work. Combining them into ten disagreements makes their consequences disappear.

This matrix evaluates the judge against the reviewed labels. It does not establish the extractor's overall quality. That needs a separate measure of the extraction outputs and their failures.

Which false-accept rate are you calculating?

Two useful calculations use the same six errors with different denominators.

Among the defective outputs, the judge misses 6 ÷ 40 = 15%. This measures its failure to catch the error on the labeled set.

Among outputs the judge passes, 6 ÷ 42 = 14.3% are defective. This describes the contamination of its pass queue on this set. Calling both figures the false-accept rate without a denominator invites confusion.

The false-reject rate among acceptable outputs is 4 ÷ 40 = 10%. The operational effect depends on what rejection triggers. Sending an output to a quick review differs from blocking a customer task or running an expensive retry.

Write counts beside percentages, especially for small slices. One changed label can move a small-set rate substantially. If reviewers disagree about whether a payment term is acceptable, resolve the field policy before using that case to score the judge. The ground-truth guide describes how a label becomes an accepted reference.

A balanced set is a diagnostic, not a production forecast

The example deliberately contains equal numbers of acceptable and defective outputs. Production may have a very different mix.

That difference changes the share of passed outputs that are actually defective. You cannot take the 14.3% pass-queue result and present it as the expected production rate. Even the error-detection rate may change if production contains document types or failure patterns absent from this set.

Keep the balanced diagnostic useful by reporting what it covers: source quality, document families, task requirements and the failures represented. Use a representative sample to examine the actual workload, with breakdowns for difficult document types.

Also read a sample of agreements. A judge may reject the wrong payment term because it claims an unrelated field is missing. The label matches, but the explanation does not support the assigned check. Inspect the explanation against the source and test nearby cases before counting the example as evidence of reliable error detection.

The worksheet behind a release decision

For each judge version, complete this record:

ItemWhat the decision owner records
Target failureOne defect, its exceptions and the required evidence
ReferenceDataset version, labeling policy and reviewed case counts
Class conventionWhether positive means error or acceptable output
MatrixCorrect passes, correct failures, false accepts and false rejects
RatesEach calculation with its denominator
RoutingWhat a pass and a fail actually trigger
LimitsUnsupported document types, uncertain labels and small slices
DecisionAllowed use, required review and reason for that choice

Finish with the cases, not just the table. Locate the six false accepts, identify the evidence the judge missed and decide which routes remain acceptable. Retain them in the golden dataset for later regression checks.

Use the wider LLM judge guide when the result calls for a prompt or model change. The worksheet gives the person approving that change a concrete record of which errors remain and how the workflow handles them.