
In 2025, many product teams added an AI chatbot to their product with little validation. Quality checks were superficial and based on a few manual tries. In 2026, users have much less patience for generic or mediocre AI output, and the quality of the AI decides whether a feature gets used.
The question teams ask has changed accordingly. "Which model should we use?" has been replaced by "How do we know our AI feature is good?" Answering that question means deciding how AI quality is measured, who owns the measurement and what happens with the result. That is what evals are, and most of that work sits with product.
This guide covers AI evals for product managers from first principles to a working eval suite. It is based on more than 1,500 experiments we ran with product teams in fintech, HR tech, procurement, logistics, health tech and sustainability. One running example, an AI shopping assistant for an e-commerce platform, carries through every section so each concept is applied to something concrete.
It is written for PMs who own an AI product or feature and need to answer "How good is it, and how do you know?" with data. Evals have become a core part of AI product management, and this guide shows the workflow in practice.
What this guide covers
- •What an AI eval is and where its limits are
- •Why evals belong to the product manager
- •Why AI breaks the testing habits from traditional software
- •The three types of evals: manual review, code-based checks and LLM-as-judge
- •The PM's eval workflow, with two ways to start and one shared loop
- •How to turn failures into metrics, with worked examples
- •Offline and online evals, and a golden dataset that evolves with the product
- •Metrics that mislead, common mistakes and the tools to run evals
Key takeaways
- •An AI eval is a repeatable test suite that scores AI output against your own success criteria. Public leaderboards, quick manual tries and production monitoring each answer different questions.
- •Evals are a product responsibility. Judging whether an output is good requires domain and user knowledge, which usually sits with the PM and subject-matter experts.
- •AI fails silently. It always returns an answer. Without evals, users end up doing your quality assurance, and they rarely file bug reports before they churn.
- •Start with manual annotation and automate later. Teams that build an LLM judge before labeling a single output usually build a judge for the wrong failures.
- •You can start with a live feature or with an idea. With a live feature you analyze real traces, with an idea you run an experiment on best-guess inputs. Both paths lead to the same step: reading the outputs and annotating them.
- •The difference shows up in time and cost. Teams that run structured evals before launch get from idea to validated feature in 3 to 7 days. Teams that iterate through production complaints need 8 to 14 weeks at around ten times the cost.
What is an AI eval?
An AI eval (short for evaluation) is a structured, repeatable test suite that measures the quality of an AI feature's output against criteria you define. You give the AI a set of representative inputs, score each output on several dimensions with quantitative metrics, usually on a scale from 0 to 1, and track how those scores change across iterations.
An eval answers one question: how good is the AI output for my users, measured against my own definition of good? Because the definition is written down, the result can be repeated, shared and trusted.
Three things are often mistaken for evals:
- •Public benchmarks. A model topping a leaderboard says nothing about how it performs on your data, your task and your edge cases.
- •Production monitoring. Monitoring confirms that the feature responded. A wrong answer never throws an error, so monitoring cannot tell you that the content was wrong.
- •A one-time demo. "It worked when I tried it" is an anecdote. An eval measures the full range of inputs the feature will actually receive.
| Demo or manual try | Public benchmark | Production monitoring | AI eval | |
|---|---|---|---|---|
| Uses your real data | Sometimes | No | Yes, after launch | Yes |
| Repeatable | No | Not applicable | Not applicable | Yes |
| Catches silent wrong answers | No | No | No | Yes |
| Runs before users see the output | Yes | Not applicable | No | Yes |
| Gives you a number you can track | No | Only for the benchmark task | Indirectly | Yes |
Why evals are a product manager's job
AI quality is a product problem that often gets treated as an engineering problem.
An engineer can build the pipeline that runs the model, captures the output and computes a score. Deciding what the score should measure requires different knowledge: noticing that a contract summary missed the liability clause, recognizing which compliance rule applies to a user question, or spotting that a transaction landed in a category that will break a customer's bookkeeping.
That judgment about what good looks like comes from people who know the domain and the users: the PM, the subject-matter expert, the person who handled these cases manually for years.
When prompt and eval ownership moves entirely to engineering, three things tend to happen:
- •The criteria measure what is easy to measure. Engineers check whether an answer is returned and whether the JSON is valid. Whether the answer is complete, relevant and aligned with the product goal rarely gets measured.
- •The feedback loop slows down. If a PM needs a ticket to change a prompt or an eval criterion, the process becomes the bottleneck.
- •Quality stops improving early. The largest accuracy gains we see come from domain experts rewriting prompts and sharpening eval criteria, much less from infrastructure changes.
In 2026, the PM owns the eval and engineering builds the infrastructure that runs it. We go deeper on this split in who should own AI features in product teams.
Why AI breaks the testing you already know
Product managers who have shipped traditional software bring testing habits with them, and most of them do not transfer to AI.
Traditional software is deterministic: the same input produces the same output. Unit tests and regression tests check fixed expectations, and if a login flow works in 100 tests, case 101 will work too. AI is probabilistic: the same input can produce different outputs on different runs. One hundred successful tests give no guarantee for case 101.
This difference changes how testing works:
- •Exhaustive testing is impossible. There is no finite set of code paths. Users will send empty fields, typos, contradictory instructions and requests nobody anticipated.
- •"It worked in ChatGPT" is misleading. ChatGPT is a finished product with its own system prompt and guardrails. When you integrate an LLM into your product, you work with the raw model and supply the instructions, the edge-case handling and the output format yourself.
- •The most expensive failures are silent. A bad AI output does not raise an error. It returns a confident, well-formatted answer that is wrong. Monitoring records a success while the user receives a mistake, and when the problem finally shows up as churn months later, nobody can reconstruct the cause.
AI rarely says "I don't know how to handle this." It produces a structured, articulate answer that may be too generic, mediocre or simply wrong for the user. Evals find those answers before users do.
One team ran a structured evaluation on its full dataset before deployment and found five distinct failure categories across 36 model runs in 14 minutes. Every one of those failures would have reached production under their previous approach of testing three examples and shipping. The failures had always been there, and the team's testing method had no way to see them. We describe the case in why ship and learn does not work for AI features.
The 3 types of AI evals every PM should know
Once you measure AI quality systematically, every eval falls into one of three types. In practice you combine all three.
The running example. We follow one feature through the guide: an AI agent on an e-commerce platform that helps shoppers find the right products and get to checkout. A shopper describes a need in plain language, for example "educational toys for my child under 2" or "a laptop and accessories under 3,000 EUR". The assistant searches the catalog, compares options, recommends products and keeps the context across several turns.
1. Human (manual) evaluation
A person reviews AI outputs and annotates them. This is the first and most important type, because every automated check builds on it. Only manual review tells you which failures the AI actually makes, as opposed to the failures you expect it to make.
Three rules make manual evaluation work:
- •Describe each failure precisely. "Missed the key clause", "addressed the user too formally" or "missed the root cause of the metric drop" can later be grouped and measured. "Wrong answer" gives you nothing to fix or automate.
- •Keep all annotations in one place. PMs, financial analysts, compliance and legal experts often review the same outputs. If their notes are spread across documents, chats and spreadsheets, nobody can see patterns across iterations.
- •Compare several models, blind. Run the same system prompt and test inputs on several models and providers, for example OpenAI, Anthropic and Google. Side-by-side answers make it easier to say which is better and why, and they often reveal context the instructions were missing. Hide the model names: as soon as a reviewer knows which model produced an answer, bias enters. Blind review regularly overturns a team's assumption about which model is best.
Worked example: annotating one conversation. A parent asks the shopping assistant for educational toys for a child under two. In the second turn, the parent adds a budget of 40 EUR. In the third turn, the parent asks for "something with magnets". The assistant answers:
"Here is a great option: the MagnaBuild 64-piece set (ages 3+, 49 EUR). Unfortunately we don't carry other magnetic toys, so you may want to check a specialist store."
The annotation for this answer:
- •Drops the age constraint from turn one after two turns and recommends a 3+ product.
- •Drops the budget constraint from turn two.
- •Calls the external search tool although the catalog database already holds the answer.
- •Misrepresents the catalog: two magnetic toys for under-twos are in stock, yet the assistant sends the shopper to another store.
Each line is specific enough to become a metric later.
2. Code-based (deterministic) checks
Code-based metrics are small functions that take the model response, the test input and optionally an expected output, and score the answer on one dimension. They are rules that pass or fail without ambiguity: format validation, JSON schema validation, exact match against an expected value, length limits, required fields, forbidden content. Use them for everything that can be checked objectively. They cost almost nothing, run instantly and are easy to set up. Our glossary entry on deterministic evals has more examples.
Code-based checks should be the first automated evals you put in place, especially while you are still exploring architecture and implementation options. They are also the fastest way to scale manual testing: automate the checks for failures you found during manual annotation, then review by hand only the answers that pass, looking for new failure patterns. Each new pattern can be automated the same way.
Worked example: designing deterministic metrics. Most shopping-assistant checks turn out to be deterministic, because the catalog has structured fields to compare against:
- •Right products for the query? Compare the recommended product IDs with the catalog products that match the shopper's filters (category, age, price). Any product outside that set fails.
- •Right category? Check each recommended product's category against the request: toys, lifestyle or computers.
- •Stock reported correctly? Check each product's
in_stockfield. If the assistant claims a product is unavailable when it is in stock, or says only one product matches when several do, the check fails. - •Unnecessary tool call? Compare the tool calls the AI made with the expected output. If the answer did not require a tool call and the AI made one, the metric returns 0.
3. LLM-as-judge
An LLM-as-judge is the most demanding eval type to set up. A second AI model reads the output and checks whether one specific error is present, returning a score between 0 and 1. Set up LLM judges last, and only for high-impact failures that require judgment and that you have observed during manual annotation. They need more discipline, more maintenance and add cost per call, which deterministic checks do not. Our guide to building an LLM judge you can trust covers the setup in detail.
Two rules separate a useful judge from a useless one:
- •One failure per judge. "Does this answer contain this specific failure?" is a question a judge can answer reliably. "Is this answer good?" cannot be validated or acted on.
- •Validate the judge against your human labels. A judge has to agree with your manual ratings before it replaces them. Your manual annotations now pay off a second time: they become the test dataset for the judge. The judge goes through the same eval process as the main AI feature, with several prompt iterations and a comparison of models. A judge that scores below 90% agreement on your validation dataset will not produce results you can base decisions on.
Worked example: an LLM judge for dropped constraints. One failure has no structured field to check against, because it depends on reading the shopper's intent. It gets a narrow judge:
You are an evaluation judge for an AI shopping assistant on an e-commerce marketplace.
Your task is to assess whether the assistant commits the error "dropping the user
constraint": recommending products that violate constraints the user has expressed.
- Score 1 (PASS): The output respects all constraints the user expressed.
- Score 0 (FAIL): The output drops at least one constraint and recommends products
that do not match the user's stated preferences or needs.
# Rules
1. Types of constraints: budget, shipping time, product specifications, seller
preferences, personal preferences. They can be stated as preferences, numbers or lists.
2. Multi-turn conversations: a constraint can be stated in the first message or in any
later turn. It applies from the turn in which it was stated.
3. Dropping a constraint is justified (score 1) if one of these is true:
- the user explicitly asked to drop the constraint
- the user changed or replaced the constraint
- the user moved to a different product category unrelated to the original request
# Output format
Respond with valid JSON only:
{
"score": <0 or 1>,
"justification": "<1-2 sentences explaining your assessment>"
}
Then run the judge on the turns you already annotated by hand. Where it agrees with you, you can trust it. Where it disagrees, tighten the definition and run it again.
| Eval type | Best for | Cost | Watch out for |
|---|---|---|---|
| Human (manual) | Error analysis, building the foundation | Time-intensive | Does not scale, needs domain expertise |
| Code-based | Format, structure, comparison with expected output, objective rules | Close to zero | Cannot score subjective qualities |
| LLM-as-judge | Subjective quality attributes at scale | Per call | Must be validated against human judgment |
The order matters: manual annotation first, automation second. The most common mistake is building an LLM judge before anyone has read a single output, which produces a judge that encodes untested assumptions.
The PM's eval workflow: two ways to start
Most guides assume an AI feature is already in production. In practice, PMs come to evals from two directions, and both are valid:
- •Path A: You have a live feature and want to know how it actually performs.
- •Path B: You have an idea and want to validate it before anything ships.
Both paths lead to the same place: a set of AI outputs in front of you that you read and annotate. Everything after that, the analysis loop, is shared.
Path A: You have a live feature, analyze your traces
In production you already have the most valuable test data there is: real user inputs and the real answers your AI gave. A trace is the complete log of one interaction, including the user input, the model response, the tools it called, latency and token usage.
- •If you can access your traces, import a representative sample into an evaluation tool. Include the messy cases and inputs nobody designed for. A sample of only the good conversations hides the problems you are looking for.
- •If you cannot access traces yet, collect a handful of inputs users actually sent. Set up an experiment with your system prompt, those inputs and the model you use in production, plus two or three others for comparison, and run it. The answers will differ slightly from what users saw, but most of the failures users hit will show up again.
The advantage of this path is that your test set reflects reality from the first day.
Path B: You have an idea, run an experiment on your best guess
Without a live feature, you can build your eval suite while you design the feature. You create the test data instead of importing it.
- •Make your best guess at the user input. Sketch the inputs you expect, based on the data you can collect from users and how you expect them to interact with the AI. A reasonable guess is enough to learn whether AI can solve the task at all, at what accuracy and at what cost. Ten to fifteen diverse test cases are enough to start.
- •Write your best version of the system prompt. Describe the task, the tools the AI can use and the output format as clearly as you can. Keep it simple at this stage. The right improvements become obvious once you read the answers and see the failure patterns.
- •Run the experiment on several models. Send the same test cases and system prompt to several LLMs and capture the outputs. Even if your company is tied to one provider, the comparison shows what a good answer can look like.
- •Annotate the results. Read each answer, mark it pass or fail and write down why.
This path lets you validate a feature before committing engineering effort, so the build starts with evidence.
Where both paths converge: the shared analysis loop
Whether the outputs came from real traces or from an experiment, the next steps are identical.
1. Human review first. A domain expert reviews the actual AI responses, the answers the model generated from your instructions, architecture and inputs. Everything else builds on this step, and it does more for the reliability of your evals than any other activity.
2. Group failures into patterns. Fixing outputs one by one does not scale. Cluster your free-text annotations into categories that share a root cause. This step, often called error analysis or axial coding, is the most valuable eval skill a PM can learn. Each category should describe one root cause and not overlap with the others. Make categories broad enough to avoid dozens of tiny ones, and specific enough that each can be detected and measured, either deterministically or with an LLM judge. For each category, record severity and frequency, because a rare cosmetic issue and a frequent reasoning error deserve different priority. You can try this on your own outputs with our free Failure Patterns tool: paste in annotated outputs, and it groups them and suggests metrics you can automate.
Worked example: failure patterns for the shopping assistant. The annotations collapse into four patterns. None of them would be caught by a generic metric:
| Failure pattern | Root cause | Shows up as |
|---|---|---|
| Wrong recommendations | Drops a hard constraint (age, budget) when the conversation moves on, or searches the wrong department | Recommends a 3+ magnetic set to a parent shopping for a child under two |
| Stock and availability errors | Claims a product is unavailable, or that only one matches, when the catalog says otherwise | "We only have one matching set" while three are in stock |
| Talking instead of acting | Describes a search instead of running it, or loops on confirmations | "I can search for that…" with no search executed |
| Misread intent | Misunderstands what the shopper wants next | Treats "ok, go ahead" as the end of the conversation |
3. Turn failures into metrics. Make each failure measurable so you no longer have to reread every output. For each category, define one or more metrics and choose the cheapest type that can catch it:
- •Deterministic checks for anything objective and rule-based: JSON schema, value matching, format, required fields, comparison with an expected value. Use these first.
- •LLM-as-judge only for what code cannot check, such as tone, respecting user constraints or missing important information. Keep each judge narrow, one failure per judge, and validate it against your manual labels.
Keep metrics granular. You can combine them into one aggregate score to track over time, but the individual metrics show what actually changed. Be careful with generic judge templates for relevance, coherence, hallucination or toxicity. They were built without knowledge of your high-impact failures and were never validated against your labels, so the numbers they return rarely help you improve. Our article on building an AI evaluation rubric from real outputs walks through this step with a second example.
4. Prioritize and choose a fix. There are five levers:
- •System prompt: sharper instructions, domain context, examples, explicit handling of edge cases. Try this first.
- •Metrics: the failure may reveal a quality dimension you were not measuring yet. Add the eval.
- •Test cases: expand coverage for the failure you just found.
- •Models: compare alternatives on your own metrics.
- •Architecture: split the agent, change the data or tools the AI can access, or change how users interact with it. This is the biggest lever and can require reworking existing evals.
Start with the cheap fixes. Exhaust prompt improvements before you change the architecture, and move to bigger changes only when several iterations bring no progress.
5. Re-run and decide whether it is ready. Run the new setup with the improved prompt, new models and new test cases. Check whether each failure pattern improved, pattern by pattern, in addition to the aggregate score. Because the most important checks are now automated, you can test many combinations in a few hours.
Offline vs. online evals: building and evolving your golden dataset
Evals also differ in when and where they measure quality. Offline evals run before users see an output, online evals run while the feature is live. You need both, and they serve different purposes.
Offline evals: your golden dataset
An offline eval runs against a golden dataset: a curated set of test cases where you have defined the ideal output for each input. That ideal output is the ground truth the eval compares against. You do not need it upfront, it can emerge from your first experiments.
For the shopping assistant, the golden dataset is a collection of representative shopper questions and multi-turn conversations, each paired with the response the assistant should give. The eval metrics then compare the actual output with that ideal: Did it recommend the right products? Did it miss any?
Offline evals are your controlled environment, built on human annotations and validated expected outputs. Because you have a ground truth, you can run many metrics and get a numeric read on quality, including whether a change improved or worsened it. Run the offline suite whenever something changes that could affect quality:
- •when you update the system prompt
- •when you test a new model
- •when you change the architecture of the AI system
- •when you expand to a new use case or customer segment
This is your regression net. A regression is a change that quietly makes quality worse, and the offline suite shows it numerically before the change ships.
Online evals: measuring quality in production
Once the feature is live, there is no expected output. Users send inputs that are not in your golden dataset, and nobody has written the ideal answer for them, so comparison metrics cannot run.
Metrics that judge quality from the input and the actual output alone can run. You build them as part of the offline suite, and they work in production as well. They give you a live signal on real user inputs, show quality trends on a dashboard, alert you when quality drifts, and surface inputs you never thought to test, which you can then add to the golden dataset.
| Offline eval | Online eval | |
|---|---|---|
| When | Before a change ships | Live, on real traffic |
| Needs an expected output | Yes, from the golden dataset | No, only input and actual output |
| Best for | Regression testing, comparing prompts, models and architectures | Drift detection, finding unseen failures |
| Based on | Curated human annotations | Real user inputs |
Your golden dataset is a living thing
The golden dataset changes with your product. At the start you have no realistic test data, so you guess the inputs, exactly as in Path B. Once the feature is live, real production cases replace those guesses, including the ones the AI struggled with.
The loop works like this: online evals or a manual review of fresh traces show where the AI fails in production. You take those cases, write the ideal output for each and add them to the golden dataset. The next offline run then tests against real usage. Repeated continuously, new use cases, customer segments and edge cases flow back into your evals.
The golden dataset has to match production
One risk deserves a warning, because it makes teams feel safe shortly before something breaks: the golden dataset has to reflect the distribution of real usage.
A dataset filled with easy cases, one user profile and one type of question will score close to 100%. The team concludes the AI is ready and ships a feature that fails on the real range of inputs. A dataset of easy cases measures the team's expectations instead of quality. Cover the spread of profiles, question types, edge cases and messy inputs in roughly the proportions they appear in production.
The metrics that matter and the ones that mislead
Some common metrics give a false sense of quality:
- •A single accuracy number without breakdown. "72% accurate" leaves open what was tested, how the AI fails and which errors are acceptable. Always pair the number with the failure breakdown.
- •"The AI returned an answer." AI always returns an answer, so this measures nothing.
- •A confidence score from the same model. Asking the model to rate its own output is an unreliable quality signal.
- •A second AI rating "overall quality". A judge scoring broad dimensions like "good quality", without validation against human labels, produces numbers nobody can act on.
Metrics that hold up are defined on your own terms:
- •Task accuracy against your expected outputs and human ratings.
- •Failure rate by category, so you know how the AI fails and how often.
- •Format compliance: did it follow the required structure? A cheap code-based check.
- •Cost per request at scale, which can quietly break unit economics.
- •Latency, which can decide the product experience: 4.9 seconds and 47.6 seconds are two different products.
Cost is the eval PMs forget most often. AI changes software economics, because every request costs money and your most active users cost the most. In one extraction task across 15 models, GPT-5 reached 60% accuracy while GPT-4o reached 100% at a fraction of the cost, with 4.9 seconds of latency against 47.6 seconds. Without cost in your eval metrics, cost projections are guesswork. We explain the mechanics in the AI cost trap.
Tools for running AI evals as a product manager
A spreadsheet is enough for the first 20 to 50 cases: inputs in one column, outputs in the next, annotations in a third. It stops working once you compare several models and prompt versions, run blind reviews with domain experts, or re-run the same golden dataset after every change.
At that point, look for an evaluation tool in which a PM can do the work without an engineering ticket: import traces or upload test cases, run experiments across models, annotate outputs in one place and turn failure categories into automated metrics. We compared eight tools, including Confident AI, LangSmith, Braintrust and Langfuse, in our ranking of the best AI evaluation tools for product managers. Lovelaice is the AI evaluation platform built for product managers and covers the workflow in this guide from the first experiment to the golden dataset.
Common AI eval mistakes PMs make in 2026
- •Validating on three happy-path examples. The demo works, production fails, and the team learns about it from a churned customer or a lost deal.
- •Building the LLM judge before reading real responses. A judge cannot be validated without labeled outputs to compare against.
- •Handing the eval to engineering or QA based on acceptance criteria. Engineering and QA usually lack the domain expertise to spot high-impact failures beyond format and structure.
- •Treating the prompt as engineering work. "You are a helpful assistant" is the AI equivalent of a spec that says "make it good". In one test, every model except GPT-5 scored 0% with a basic prompt, and a structured prompt brought several models to 90%. For a PM, the prompt is the strongest lever on AI behavior. More on this in 3 mistakes PMs make when they let engineers own the AI prompt.
- •Skipping cost modeling. Many teams do not know what an average AI call costs them, which leaves major decisions about the feature without data.
- •Relying on generic metrics from observability templates. Template metrics such as relevance or consistency produce a number without showing what went wrong or how to fix it.
- •Relying only on thumbs up and thumbs down from users. Few users click the button, and those who do rarely explain why. As the only eval strategy, it fails. Our article on moving past vibe checks shows how to scale manual testing into systematic evaluation.
Evals and regulation: the EU AI Act raises the floor
If your AI feature touches a regulated domain, evals become evidence. The EU AI Act requires documentation, traceability and risk management, and a disciplined eval process produces exactly that as a by-product: a record of what was tested, what failed, how it was fixed and why the feature was judged ready.
A team that ships after a few manual tries has nothing to show a regulator or an enterprise procurement team. A team with structured evals has an audit trail.
The bottom line
Evals are the core product skill of 2026, because they answer the question that matters most about an AI feature: is it good enough for my users, and how do I know?
The answer comes from a suite of quantitative metrics you defined yourself, based on what a good output looks like and how you have seen the AI fail for your use case. Confidence scores, QA sign-offs and thumbs-up rates cannot provide that answer.
You need no permission, technical background or perfect setup to start. The first step takes an afternoon: put 20 real outputs in front of you and annotate them. If the feature is live, pull your traces. If it is not, run an experiment on your best guess. If you want to run that first experiment without engineering support, see how Lovelaice works for product managers.
The teams that succeed with AI bring evidence to AI decisions in the same way good product teams brought data to product decisions when usage analytics became standard.
Proof that an AI feature works comes from a structured eval across the full range of inputs your users send.
Enjoyed reading? Add us as a preferred source in Google to support us.
Add Lovelaice as a preferred source