What does "the AI is working" actually mean?

"Is our AI working?" is a question most product teams cannot answer — not because they lack data, but because they never defined what "working" means.

They shipped the feature. It returns outputs. No one is screaming. So it's working, right?

That is not a definition. That is the absence of a complaint.

The Cost of Not Defining "Working"

Without a clear definition of success, the same pattern plays out every time. Leadership asks, "How's our AI doing?" The PM says, "Good question." The monitoring is Slack threads. The dashboard they need doesn't exist. The only feedback loop is churn — and by the time churn shows up, the damage is months old.

This is Ship and Pray dressed up as product management.

AI doesn't fail loudly. There is no error log, no crash, no red alert. The feature quietly underperforms while users quietly leave. A traditional software bug throws an exception. An AI feature that misclassifies 30% of inputs just looks like a feature that works — until you measure it.

One team we've seen upgraded their model. Then the complaints started. They found out a month later, from users. There was no alert, no comparison, no process. Just inbox messages and a very uncomfortable sprint review.

That is what happens when "working" means "nobody has yelled yet."

"Working" Has Three Components, Not One

The definition most teams are missing is not complicated. It is specific. "Working" for an AI feature means three things, measured together:

1. Accuracy on your actual use cases — not three happy-path examples.

Most teams test a handful of examples they already know work, then ship. That is vibe-checking. Your three happy test cases are not enough. "Working" means you have run evaluation across your full dataset — hundreds of real inputs — and you know the accuracy number. Not a feeling. A number.

The difference matters. A team that tested three examples might believe accuracy is 90%. Run it across 200 real inputs and you discover it is 52%. That is not a rounding error. That is a product that does not work.

2. Cost at scale — not cost for one demo.

In traditional SaaS, a new user costs you almost nothing. In AI, every user has incremental cost. Every interaction costs money. A feature that "works" at demo scale can bankrupt your unit economics at production scale.

One Fintech team discovered they could switch away from GPT-4.1 after structured model comparison — same task, 10x lower cost, higher accuracy. Before that comparison, they were paying $847/month on a model they never validated. "Working" includes knowing your cost per use case before you commit.

3. Consistency across edge cases — not just the median.

AI is non-deterministic. The same input does not guarantee the same result. Ask the same question five times, get five different answers. When you integrate AI into a product, you are solving one problem at scale — the same set of instructions applied to inputs that are completely outside your control.

A customer support classifier needs to handle polite requests, angry rants, one-word messages, and everything in between. "Working" means you have tested across the distribution of real inputs, not the center of it. It means you have identified failure categories — not just whether individual outputs look right, but where and how the feature breaks.

Why Teams Skip the Definition

Two reasons. Both fixable.

Reason one: they treat AI like traditional software.

With deterministic software, you write the code and it functions the same way everywhere. Same input, same output. Bugs are visible. You can write a test suite and trust it.

AI does not work this way. Results are probabilistic. Context changes behavior. You cannot prove correctness from a hundred passing cases — you only know it works on the cases you tested. Teams that apply the traditional software mental model to AI features end up with false confidence. They tested, it passed, they shipped. But passing three tests does not mean it passes the three hundred they did not run.

The mindset shift is fundamental: with AI, you are not verifying correctness. You are measuring performance. That requires a different definition of "working" — one built on statistical evidence, not pass/fail assertions.

Reason two: the people who know what "good" looks like are locked out of the process.

Engineering knows the technology. But product managers and domain experts know what a correct output actually is. The doctor knows what a good diagnosis looks like. The lawyer spots the gap in the contract review. The analyst understands the failure.

When AI quality is owned entirely by engineering, the definition of "working" becomes technical: latency is under 200ms, the API returns 200, the output is valid JSON. Those are necessary. They are not sufficient. They tell you the plumbing works. They do not tell you the answer is right.

The definition of "working" must include domain accuracy — and that means the people closest to the problem need to be evaluating outputs directly. Not through a requirements document. Not through a Jira ticket. Directly.

How to Build the Definition Before You Ship

This is not a six-month project. It is a one-day exercise.

Step 1: Write down what "correct" means for your specific use case.

Be concrete. If you are classifying customer support emails, list the categories. Define the edge cases. Write down what a wrong answer looks like for each category. This is domain work, not engineering work. The PM or the subject matter expert does this.

Step 2: Build a test dataset from real inputs.

Not synthetic data. Not happy-path assumptions. Real inputs from your actual users. If you have 50, start with 50. If you have 500, use 500. Each input needs a known-correct answer — the ground truth that your AI output gets compared against.

Step 3: Define your accuracy threshold before running the evaluation.

This is the step almost everyone skips. If you evaluate first and then decide the number is "good enough," you are anchoring to whatever you got. Set the bar before you see results. Is 80% acceptable for this use case? Is it 95%? The answer depends on the stakes — classifying blog tags is not the same as classifying medical symptoms. But the point is: you have a number, in writing, before the experiment runs.

Step 4: Run evaluation across the full dataset, not a sample of three.

Compare outputs against your ground truth. Group failures by category. Identify where the feature breaks and why. This is where vibe-checking dies and evidence begins.

Teams that follow this process go from "I think this works" to "87% accuracy on 150 test cases." That is a number you can show to leadership. That is a number you can defend. That is a number you can improve.

One team went from 50% to 93% accuracy in a single iteration, in under one hour — because they had a clear definition of correct, a real dataset, and a structured evaluation process. Without the definition, that improvement does not happen. You cannot fix what you cannot measure.

"But We'll Evaluate After Launch"

This is the most common pushback, and it is the most expensive mistake.

"We'll ship, collect traces, and evaluate later." But by then, you have already lost users. You have already burned money. You have already created the impression — with real customers — that your product does not work. User trust is not something you get back with a prompt fix.

The economics make this particularly painful for AI features. Traditional features have near-zero marginal cost per user. AI features have real cost per interaction. If you are running an unvalidated model at scale, you are gambling with your margins every day the feature is live.

"We'll iterate later" is not a strategy. It is a postponement of the decision you should have made before deployment.

"Our Engineers Handle Quality"

They handle technical quality. Uptime, latency, integration stability — that is engineering's domain and they are good at it.

But output quality — whether the AI's answer is actually right for your users — requires the knowledge that lives outside engineering. A prompt written by an engineer who does not deeply understand the domain will produce generic output. Teams that applied field knowledge directly to evaluation and prompt iteration saw 40% accuracy improvements. Not because the engineering improved. Because the right people were evaluating the right thing.

Quality ownership for AI features belongs to the person who can look at an output and say, with authority, "That is wrong and here is why." That person is usually not the one writing the API integration.

Define It or Discover It From Users

Those are the two options. You define what "working" means before you ship, or your users define it for you — through complaints, through churn, through that very uncomfortable sprint review.

"Working" is not the absence of errors. It is accuracy you measured, cost you validated, and consistency you proved — across real data, before deployment.

Product analytics taught teams that gut-feel feature decisions were wrong on almost every count. AI quality is at that same inflection point. The teams that define "working" with data will build products that last. The rest will ship and pray.

Data beats gut feel. Always.