Suppose a judge agrees with reviewers on 70 of 80 outputs, yet approves six defective extractions. The 87.5% agreement score is correct. An LLM judge confusion matrix shows what that score leaves out: which mistakes passed, which acceptable outputs were rejected, and what those decisions would do to the workflow.
Before reading the matrix, check the label convention. Here, positive means an error is present. A false negative is a defective output that the judge passes; we call it a false accept. A false positive is an acceptable output that the judge rejects, or a false reject. Some tools use the opposite convention. Put the meanings beside the report.
The LLM judge confusion matrix, filled in
Take a judge checking whether an extracted payment term is supported by the invoice. A reviewer has labeled 40 outputs acceptable and 40 defective. The judge produces these decisions:
| Reviewed reference | Judge says fail | Judge says pass | Total |
|---|---|---|---|
| Defective output | 34 correctly rejected | 6 false accepts | 40 |
| Acceptable output | 4 false rejects | 36 correctly accepted | 40 |
| Total | 38 | 42 | 80 |
The agreement calculation counts the diagonal: 34 correct rejections plus 36 correct acceptances, divided by 80. That gives 70 ÷ 80 = 87.5%.
An agreement report should retain all four cells. The six false accepts may let unsupported payment terms continue into another system. The four false rejects may create unnecessary review work. Combining them into ten disagreements makes their consequences disappear.
This matrix evaluates the judge against the reviewed labels. It does not establish the extractor's overall quality. That needs a separate measure of the extraction outputs and their failures.
Which false-accept rate are you calculating?
Two useful calculations use the same six errors with different denominators.
Among the defective outputs, the judge misses 6 ÷ 40 = 15%. This measures its failure to catch the error on the labeled set.
Among outputs the judge passes, 6 ÷ 42 = 14.3% are defective. This describes the contamination of its pass queue on this set. Calling both figures the false-accept rate without a denominator invites confusion.
The false-reject rate among acceptable outputs is 4 ÷ 40 = 10%. The operational effect depends on what rejection triggers. Sending an output to a quick review differs from blocking a customer task or running an expensive retry.
Write counts beside percentages, especially for small slices. One changed label can move a small-set rate substantially. If reviewers disagree about whether a payment term is acceptable, resolve the field policy before using that case to score the judge. The ground-truth guide describes how a label becomes an accepted reference.
A balanced set is a diagnostic, not a production forecast
The example deliberately contains equal numbers of acceptable and defective outputs. Production may have a very different mix.
That difference changes the share of passed outputs that are actually defective. You cannot take the 14.3% pass-queue result and present it as the expected production rate. Even the error-detection rate may change if production contains document types or failure patterns absent from this set.
Keep the balanced diagnostic useful by reporting what it covers: source quality, document families, task requirements and the failures represented. Use a representative sample to examine the actual workload, with breakdowns for difficult document types.
Also read a sample of agreements. A judge may reject the wrong payment term because it claims an unrelated field is missing. The label matches, but the explanation does not support the assigned check. Inspect the explanation against the source and test nearby cases before counting the example as evidence of reliable error detection.
The worksheet behind a release decision
For each judge version, complete this record:
| Item | What the decision owner records |
|---|---|
| Target failure | One defect, its exceptions and the required evidence |
| Reference | Dataset version, labeling policy and reviewed case counts |
| Class convention | Whether positive means error or acceptable output |
| Matrix | Correct passes, correct failures, false accepts and false rejects |
| Rates | Each calculation with its denominator |
| Routing | What a pass and a fail actually trigger |
| Limits | Unsupported document types, uncertain labels and small slices |
| Decision | Allowed use, required review and reason for that choice |
Finish with the cases, not just the table. Locate the six false accepts, identify the evidence the judge missed and decide which routes remain acceptable. Retain them in the golden dataset for later regression checks.
Use the wider LLM judge guide when the result calls for a prompt or model change. The worksheet gives the person approving that change a concrete record of which errors remain and how the workflow handles them.
Enjoyed reading? Add us as a preferred source in Google to support us.
Add Lovelaice as a preferred source

