There's a reflex in every AI team: a new flagship model drops, it tops the public benchmarks, and the assumption is that switching to it makes your feature better. So you switch, you pay more, and you move on, confident you're using the best tool available.

That confidence is usually untested. The model that wins on generic benchmarks is answering generic questions — math problems, trivia, reasoning puzzles. Your feature isn't doing any of that. It's extracting fields from a specific kind of contract, or summarizing claims in a specific regulatory context. Performance on someone else's task tells you very little about performance on yours.

What benchmarks measure, and what they miss

Public leaderboards rank models on standardized tasks so they can be compared across the industry. That's their job, and it's a reasonable signal for "is this model broadly capable." It is a poor signal for "is this model good at the one narrow thing I'm shipping." A model can sit near the top of the public charts and still be middle of the pack on your documents, because your documents were never in the test.

The only benchmark that predicts how your feature performs is one run on your data, with your prompt, against your definition of a good answer. Everything else is borrowed evidence.

The cost trap nobody priced in

The default-to-newest habit has a second problem, and it's financial. With traditional software, your margins improve as you add users. With AI, every interaction costs money, and your most engaged users cost the most. The model you pick is multiplied across every call, forever. A model that's marginally better but several times more expensive can quietly turn a healthy feature into one that loses money at scale.

We've seen the gap run the wrong way entirely: the latest, most-hyped model scored around 20% accuracy on a real client task, at roughly 100 times the cost of a less fashionable alternative that did the job better.

How to actually choose

Choosing a model should be a measurement instead of a default. Run the same prompt and the same test set across several models — the latest one, a couple of cheaper ones, and at least one you'd normally dismiss — and read three numbers for each: accuracy on your task, cost per call, and latency.

What usually comes out of that comparison surprises people. Often a mid-tier or older model clears your accuracy bar at a fraction of the cost and half the latency, and the only thing the flagship was winning was the marketing. Now you have a model choice you can defend on price as well as quality, which is exactly the case a CFO wants to see before a feature scales.

The business case writes itself

The reason this matters beyond engineering hygiene: a model comparison is most of a business case. Accuracy tells product the feature works. Cost per call tells finance whether it's sustainable. Latency tells design whether the experience holds up. Run that comparison once and you can walk into the room with a recommendation backed by all three — instead of "we're using the best model," which, until you've measured, you don't actually know.

The newest model might be your best one. You find out by testing on your task, not by reading a leaderboard built for someone else's.


Want to run your own structured model comparison on your real data? Our Maven course, Ship AI features with confidence for PMs, walks PMs through the full workflow — same prompt, same test set, several models, three numbers.

Related reading: The model selection blind spot · Tokens, explained