The objection comes up in almost every conversation we have with product teams: "We'd love to evaluate our AI feature properly, but we don't have any production data. We haven't shipped. There's nothing to measure yet."

It sounds reasonable, and it's the moment most teams talk themselves into shipping on vibes — write a prompt, check a few happy-path examples, release, and find out what breaks from the support queue.

You don't need production data to start. You need test cases, and you can write those today.

Where the "no data" belief comes from

The mental model people borrow is from traditional analytics. Amplitude and Mixpanel only work once users are clicking around — no traffic, no funnel, nothing to look at. Teams then assume AI evaluation works the same way: ship first, collect interactions, then analyze.

Evals don't work like that. An eval is a test case plus a definition of what a good answer looks like, and both come from your domain knowledge rather than your users. A compliance analyst already knows what a correct contract summary contains. A fintech PM already knows what a KYC check must never miss. That knowledge is the dataset, and it exists before launch.

How to build a test set from zero

Start with the inputs you already understand. For a document-extraction feature, that's a handful of real documents — anonymized contracts, invoices, claim forms, whatever the feature will actually process. Ten to twenty is enough to begin. The goal on day one is not statistical representativeness, it is coverage of the cases that matter and the cases that scare you.

Three categories to seed:

  • •The obvious ones. The clean, typical input the feature was designed for. If it fails here, nothing else matters.
  • •The edge cases you already worry about. The malformed PDF, the contract in two languages, the field that's sometimes blank. You know these exist because you know the domain.
  • •The cases where a wrong answer is expensive. In regulated work this is the whole game — a missed clause, an incorrect figure, a hallucinated entitlement. Weight your set toward these.

For each input, write down what a good output contains. Not a vibe — a specific, checkable description. "Extracts the renewal date in ISO format, flags if absent, never invents one." That sentence is the start of an eval.

Synthetic data, used carefully

When you genuinely can't get real inputs — the feature is too new, the data is too sensitive — you can generate synthetic test cases. Describe the shape of a realistic input to an LLM and ask for variations: different formats, missing fields, adversarial phrasing. Treat the output as a starting scaffold, not ground truth. A domain expert still reads each one and corrects it, because synthetic data inherits the model's blind spots, and those are exactly the blind spots you're trying to catch.

What this gets you before launch

Run your prompt across your seed set and read the outputs. You'll find failure patterns immediately, long before a user ever does. You'll also see how different models handle the same inputs, which tells you something useful about cost and accuracy before you've committed to one. By the time you ship, you're not guessing whether the feature works. You have evidence, and a test set that keeps paying off every time you change the prompt.

Teams that wait for data ship blind and learn from churn they can't see. Teams that start with domain knowledge ship with a baseline and a safety net.

You have the knowledge today. That's the only input the first eval needs.


Want a guided path from a blank seed set to a working evaluation suite? Join our Maven course, Ship AI features with confidence for PMs — the same approach we use with product teams who don't have production data yet.

Related reading: The regression trap · Can you trust an LLM judge?