The AI cost trap

Let me show you what I mean.

The economics are backwards

There's advice everywhere on Linkedin right now on how to build AI products.

"Don't optimize costs early. Build first. Solve for retention and adoption. Then optimize your AI costs."

This is gospel in traditional SaaS. And for traditional SaaS, it's correct.

Here's why it worked:

In traditional software, there's zero incremental cost per additional user.

Your infrastructure costs are mostly fixed. You pay for capacity, not usage.

So your power users, the ones who use your product 10x more than everyone else, are your most profitable customers. They extract maximum value, they stay longer, they refer others, and they don't cost you significantly more to serve.

The economics work in your favor. You can focus entirely on building something people want, then optimize your infrastructure when you reach massive scale.

AI breaks this model completely.

In AI products, there is an incremental cost with every single request.

Every time a user uses you AI feature, you pay. Every time they generate content, you pay. Every time they run an analysis, you pay.

The more your users love your product and use it, the more money you lose.

Your power users, the ones who love your product most, who get the most value from it, are now the ones who are least profitable.

The economics are inverted.

Let me make this concrete with a real example.

The CEO of Loveable shared this story a few months ago:

A user vibecoded on their platform for 30 hours straight. 1,500 prompts in the first day.

With an average LLM cost of $0.07 per request (and this is conservative—it could easily be $0.70 with the wrong setup), the cost for supporting this one user for one day was $105.

The revenue? Probably a $20 monthly subscription.

The AI costs alone exceeded 5x the entire month's subscription revenue in a single day.

This user loved the product. They were deeply engaged. They were getting massive value.

And they cost the company money with every interaction.

Now scale this.

We ran an experiment recently on a hypothetical AI feature for Airbnb: personalized property descriptions based on user interests, travel history, and preferences.

Great use case. Clear value hypothesis. Could improve conversion.

After experimentation, we discovered that GPT-5 (the frontier model everyone defaults to) was 10x more expensive than GPT-4.1—with lower accuracy.

Imagine the cost difference at Airbnb's scale.

Millions of users. Millions of properties. Daily.

This is the difference between AI costs in the hundreds of thousands versus millions per month.

The "optimize later" advice that works everywhere else? For AI, it means you're optimizing after you've already committed to an architecture that costs 10x what it should.

The two types of "cost optimization"

Here's where people get confused.

When I tell teams to experiment with models and test costs before shipping, they hear "premature optimization."

They think I'm saying: "Spend weeks comparing API pricing sheets. Obsess over whether to fine-tune an open-source model or use a frontier one."

That's not what I'm saying at all.

There are two completely different activities that both get called "cost optimization"—and only one of them is premature.

Type 1: paper optimization (this is useless)

You open pricing pages for OpenAI, Anthropic, Gemini.

You compare: GPT-5 costs X per million tokens. Claude-Opus-4.5 costs Y. Gemini 3 pro costs Z.

You make a spreadsheet. You pick the cheapest one.

You've never run your actual prompt. You've never tested with your data. You've never seen how any of these models actually perform for your specific use case.

This is premature optimization. And it's worthless.

Why? Because you're optimizing a variable you don't understand yet. You don't know which model will actually work for your use case. You don't know if a "cheaper" model will cost you more because it requires 3x the requests to get acceptable results.

You're making decisions in a vacuum.

Type 2: hands-on experimentation (this is discovery)

You write your actual prompt.

You create 10-20 test cases, the real scenarios your AI will face, including the messy edge cases.

You run these test cases across 5-10 different models.

You evaluate the results blindly, without knowing which model produced which output.

And you discover things you could never learn from a pricing page:

• •An older model outperforms the newest one on your specific task
• •A "weaker" model with a better-structured prompt beats a "stronger" model with a generic prompt
• •The cost difference between your options is 20x, for identical accuracy

This isn't premature optimization.

This is discovery.

This is how you learn what actually works before you commit engineering resources to building it.

The distinction matters because in traditional software, you optimize after you've proven something works. In AI, experimentation is how you prove it works in the first place.

If you skip experimentation and go straight to production with your first attempt, you're locking in costs you don't understand for a solution that probably doesn't work as well as alternatives you haven't tested.

The systematic experimentation framework

Here's how you run discovery on your AI feature to understand user value and costs:

Step 1: Define success

What does "works" mean for your specific feature?

For a customer research synthesis tool: Quotes must be real, themes must be accurate.

For a product recommendation feature: Suggestions must match user preferences, handle edge cases (conflicting requirements), return results in the expected format.

For a medical test interpreter: Units must be correct, ranges must be appropriate, missing data must be handled safely.

Be specific.

---

Step 2: Build your test dataset

Start with 10-20 test cases.

Don't just test happy paths. Those are the easy ones. Spend 70% of your effort on edge cases:

Happy paths:

• •Clear, complete information
• •Obvious correct answer
• •Perfect formatting

Edge cases (where AI actually breaks):

• •Conflicting requirements ("I want it fast AND heavily cushioned")
• •Missing information ("user left some fields blank")
• •Ambiguous requests ("I want something good")
• •Unusual combinations
• •Typos and real-world messiness

This is where you'll discover your AI's actual failure modes.

---

Step 3: Run multi-model experiments

Test your prompt across 5-10 models. Experiment with multiple prompting techniques.

Don't assume you know which model is "best." Let the data tell you.

If you test 2 prompt variations across 5 test cases with 4 models, you get 40 responses to evaluate. This takes a few hours, not weeks.

---

Step 4: Evaluate and document

For each inadequate response document why it failed:

• •"Ignored the format instructions"
• •"Recommended the wrong product because it prioritized speed over cushioning"
• •"Hallucinated a feature that doesn't exist"
• •"Couldn't handle missing data, gave generic recommendation instead of asking for clarification"

This is where you learn the failure patterns.

---

Step 5: Analyze your data

Before you deploy anything to production, you should be able to say "based on our 100+ test on 5+ models, 5+ iterations on the prompt, we expect an average accuracy of 85% and an average cost of $0.03, with model X".

To get to this level of understanding, you need to experiment. A lot. And do so in a systematic way, such that you are able to assess progress, communicate performance clearly, gain confidence in your approach.

A data backed approach to AI development, means you deeply understand what changes the needle for your specific use case.

---

Step 6: Iterate your prompt and model combo

Look at your failure patterns. Adapt your prompt instructions to better account for them. Look at model perfromance evolution and how it evolves through different iterations.

Run the experiments again with your improved prompt and your new learnings. Iterate until you no longer learn and uncover new patterns

Step 7: Establish your baseline

Keep iterating until you hit your success threshold.

At this point, you know:

• •Which model works best for your use case
• •What your actual cost per request will be
• •What your accuracy baseline is

Now you're ready to build.

The counter-intuitive part

Here's what makes this approach feel wrong to many product people:

It feels slower.

It feels like you're not "shipping fast."

It feels like you're optimizing too early.

But here's what actually happens:

You move faster. Because you're not rebuilding features that fail in production. You're not chasing silent failures you don't understand. You're not burning engineering time on models that don't work for your use case.

You waste less. That Loveable user who vibecoded for 30 hours on a €20 subscription? With experimentation, you'd know your cost structure so you can account in your pricing for extreme cases.

You maintain trust. With your users, with your stakeholder, with your team.

The illusion is that skipping validation saves time.

The reality is that skipping validation means you learn the expensive lessons in production, with real users, burning real money, eroding real trust.

---

Systematic experimentation isn't about being cautious. It's about being strategic. It's about learning the right things at the right time.