What document extraction actually costs across 15 models

Two numbers from our extraction benchmarks that shouldn't be allowed to coexist: an 11× cost gap between models on the identical task, and a 43-point accuracy gap on the identical model driven by nothing but prompt structure. Model choice and prompt choice each move the needle by more than most teams think either does.

The economics of LLM-powered document extraction get pitched to product teams as a straight line: pick the strongest model, pay for it per token, and scale. The line is wrong on both ends. On the same extraction task on real customer documents, we've measured cost differences between models of 11×, and accuracy differences on the identical model of 86% vs 43% driven purely by prompt structure. The two effects compound. A team that picks the expensive model and writes the wrong prompt loses both ways.

This is what benchmarking 15 models on real extraction documents actually surfaces, and how to price the choice honestly.

The 11× cost gap

Take the same batch of documents (invoices, contracts, purchase orders, whatever your feature ingests) and run the same prompt across the current mid-2026 model lineup. Extract the same 12 to 20 fields. Measure token cost per document.

The cheapest end of the spectrum lands around $0.001 per document on standard-length invoices. The most expensive end lands around $0.011 on the same set. That's 11× on a metric that scales linearly with your document volume.

For a team running 50,000 documents a month, that's the difference between $50 and $550 per month of model cost, on outputs a customer cannot tell apart if you picked the right pairing. For a team at 500,000 documents (the volume where extraction actually becomes a product line, not a feature) the gap is $500 vs $5,500. Every month. Because someone picked the model at the top of a vendor blog post instead of measuring.

Two things make the 11× worse in practice:

Reasoning-model overhead. The newer reasoning models emit hidden thinking tokens you're billed for and never see in the output. On extraction tasks where the reasoning adds nothing (the fields are on the page; there's no puzzle), you're paying for tokens that don't contribute to accuracy. On our runs, the overhead can 2–4× the effective per-document cost of certain models. The invoice cost per document, on paper, matches the vendor's advertised token price. In practice, it doesn't, and you find out on the bill.

The premium tier does not correlate with extraction accuracy. Which is the sharper finding.

The 43-point gap that has nothing to do with the model

Same model, same documents, same 15 fields. The only variable we changed was the prompt structure.

Two implications teams miss:

Your "the model isn't accurate enough" complaint is often a prompt problem wearing the model's coat. Before you spend two weeks trialing a more expensive model, spend two hours restructuring the prompt on the current one. On this dataset, that single change closed almost the entire accuracy gap teams typically try to solve by upgrading.

Prompt structure interacts with the model. The 43→86 jump held on some models. On others it didn't, or moved only a few points. Which is why the honest recommendation isn't "always use XML", it's "measure the prompt × model pair, because either half by itself misleads you."

The same lesson we found in our LLM judge experiment: the pairing is the unit. Prompt alone doesn't predict. Model alone doesn't predict.

What the two numbers do together

The 11× cost gap and the 43-point accuracy gap sit on top of each other in most teams' extraction stack.

The default configuration for a team that picked a model on cost efficiency alone: a mid-tier model, a bullet-list prompt written the day the feature shipped, no measurement. Accuracy: somewhere around 50–60%. Cost: middle of the pack.

The same team, after two afternoons of measurement: the same mid-tier model, a structured prompt, validated on 100 real documents. Accuracy: 85%+. Cost: identical to before.

The same team, if they'd upgraded to the premium model without touching the prompt: accuracy 55–65% (a few points up, if any), cost 5–8× higher. This is the direction most teams move when a customer complaint hits, and it's the wrong direction.

That's what "compare LLM models" produces when it's a benchmarking process rather than a vendor selection: the outcome you thought you needed a new model for, on the model you already have.

Why the vendor benchmarks don't help

The public benchmarks on document extraction accuracy (the ones that appear in every model launch post) use canonical document sets. Clean PDFs. English-only. The kinds of invoices that would embarrass a startup if they submitted them.

Your feature does not see those documents. Your feature sees:

• •Multilingual documents with domain vocabulary the base model has never trained on
• •Scanned PDFs at variable quality, sometimes rotated, sometimes handwritten
• •Long-tail formats (obscure invoice layouts, custom contract templates, non-standard purchase orders) that break the pattern the model expects

The benchmark rankings do not port. A model that ranks #1 on the public suite can rank #7 on your document set, and the model that ranks #4 on the public suite can be your best pairing, at a quarter of the cost. Every extraction team we've measured has found this. The rankings that matter are the ones from your data.

For the same reason, no cost table copy-pasted from another team's write-up will predict yours. Document length, output length, retry rate, error-mode overhead: all of it varies by product. The 11× number above is what we saw across our benchmarks; your ratio might be 6× or 15×.

What to actually measure

If you're building the extraction feature, or already running one you're not sure about, the measurement protocol is short:

1. •

   Pull 100 real documents from production. Weighted toward the failure modes you already suspect. Not the demo set.

2. •

   Label the ground truth yourself, or have your domain experts do it. The 12–20 fields you extract, with the correct value for each, on all 100 documents. This is the annotation step and it's the load-bearing hour of the whole process.

3. •

   Run at least 5 models on the same prompt. Include one low-cost, one mid, one premium. Include at least one reasoning model to measure the token overhead honestly. Track accuracy per field, not just per document, because some fields are easy for every model, and the overall number hides where the failures actually live.

4. •

   Rewrite the prompt into a structured format (XML tags with field names, explicit type hints, one worked example per non-trivial field). Re-run the same 5 models. Compare.

5. •

   Now compute effective cost per accurate extraction, not per API call. If model A costs $0.002 per document at 60% accuracy and model B costs $0.005 per document at 90%, model B is cheaper per correct answer ($0.0056 vs $0.0033) even though it looks more expensive per call. This is the math almost nobody does, and it inverts the model-selection decision more often than not.

6. •

   Pick the pairing with the best cost-per-accurate-extraction, not the best accuracy or the best cost alone. Then re-run the top pairing on a fresh sample of 30 documents. If accuracy holds inside a few points, ship it. If it drops sharply, your first 100 were unrepresentative.

That's a two-afternoon process. In every extraction engagement we've run, it's paid for itself inside the first month of production traffic.

The economics, restated

Model cost per document, in isolation, is a rounding error on the decision. The two numbers that actually drive extraction economics:

• •Accuracy per field, on your documents. Because a low-accuracy extraction that ships costs you rework, customer complaints, and the trust you were trying to build. Those costs are 10–100× the model bill.
• •Prompt structure on the pairing. Because 43 points of accuracy hiding inside a prompt rewrite is a bigger lever than any model swap.

The 11× cost gap between models is real. The 43-point prompt gap on the same model is bigger. The teams that measure both ship extraction features that hold up. The teams that measure neither ship features that pass the vendor demo and fail on customer document #34.

---

For the parallel finding on LLM judges (where the same prompt scored 76.5% on one model and 50% on another) see The battle of the LLM judges and which model should you run your LLM judge on?.

Lovelaice runs multi-model benchmarks on your real documents in an afternoon. Talk to us.