AI experimentation for product teams

An AI experiment is a controlled comparison designed to answer a product decision. It might ask whether a revised claims-extraction prompt is safer than the current version, whether a smaller model can meet the same quality threshold at lower cost, or whether giving an agent a calculator reduces policy errors. The decision comes first. The models, prompts and scores exist to inform it.

A useful experiment contains a stated decision or hypothesis, a representative dataset, a baseline, one or more controlled variants, and quality criteria written before the team reviews aggregate results. It uses human or automated evaluation to judge real outputs. When the feature will run at meaningful volume, it also records cost and latency. It ends with limitations and a decision: select, reject, iterate or collect more evidence.

Sending one prompt to several chatbots and choosing the answer that looks best is not an experiment. It is a useful first encounter with the problem, but the choice is exposed to cherry-picking, brand preference and an input that may flatter one configuration. A controlled comparison runs every plausible variant against the same cases and rubric. It makes an improvement visible across the set and makes regressions difficult to ignore.

Teams can enter this workflow at very different points. In Lovelaice’s work with product teams, one team may arrive with a two-sentence idea and no expected outputs. Another may already operate an OCR, computer-vision or chatbot workflow and need to compare an LLM-based replacement. The first team needs rapid learning; the second needs careful baseline preservation. Both need an explicit decision and evidence proportionate to its consequences.

Four nearby practices, separated

Traditional A/B testing measures what users do after they see a variant. AI experimentation must also inspect what the system says or does before users are exposed. A conventional interface variant is normally deterministic: the button is blue every time. An AI feature can return a fluent, plausible and wrong result, then return a different result for the same input. That makes output quality a pre-release product concern.

“Ship and learn” becomes risky when the failure is silent. A user may not report that a recommendation was mediocre, a contract summary omitted a clause, an extracted invoice moved a decimal, or a lead classifier assigned the wrong tier. The interface can work, the request can return successfully and the product analytics can look normal. Trust erodes without a neat error event. In high-consequence workflows, the user may act on the answer before anyone notices.

This does not make online experiments obsolete. Offline AI experimentation and A/B testing answer different layers of the same product question. First verify that candidate outputs meet a product-specific quality and safety threshold on controlled cases. Then, where responsible, measure whether the qualified variant improves completion, adoption, retention or another user outcome. A variant that drives clicks while quietly making worse decisions is not a product win.

Nor is the goal certainty before release. No offline set recreates every production input. The goal is to replace “ship and hope” with a bounded claim: on these representative cases, under these conditions, this version cleared the threshold, improved on the baseline, and retained these known weaknesses. Production monitoring then tests whether the claim continues to hold.

An AI feature is a system of choices, not a model name. Teams can vary the prompt, model, retrieval, tools, workflow, generation settings, context, output format and evaluator pairing. A clean experiment changes the variable needed to answer the decision while keeping other influential conditions fixed. When two elements must change together, document the combined configuration rather than pretending the result isolates one cause.

The table is a design aid, not a demand to test everything. If the decision is whether retrieval source A or B answers support questions more accurately, there is little value in simultaneously changing the model, response format and agent instructions. If the decision is which production configuration to ship, testing a short list of complete pairings can be appropriate—but each pairing and its differences must be named.

The loop turns an uncertain AI idea into a repeatable product decision. Each step has a concrete output: a decision statement, test set, rubric, baseline, controlled change, reviewed outputs, failure analysis, decision record and regression run. A small exploratory experiment can complete the loop quickly; a high-risk release will deepen every step.

1. 01

   Define the decision.

   State what the team must decide and why it matters to users or the business.

2. 02

   Build the test set.

   Combine representative inputs, edge cases, and high-cost failure cases.

3. 03

   Define “good.”

   Write the domain-specific rubric and acceptance threshold before seeing aggregate results.

4. 04

   Create the baseline.

   Record the current prompt, model, output quality, cost, latency, and known failures.

5. 05

   Change one variable.

   Keep the dataset and evaluation conditions constant wherever the question permits.

6. 06

   Review real outputs.

   Use blind side-by-side review when model identity could bias judgment; record why each output fails.

7. 07

   Analyse failure patterns.

   Group errors by frequency, severity, and user impact.

8. 08

   Decide and document.

   Select, reject, or iterate; retain the evidence and known limitations.

9. 09

   Repeat against the baseline.

   Re-run the evaluation suite before shipping each material change.

A representative dataset is a deliberately selected model of the product’s work. It includes real inputs, or realistically reconstructed inputs when privacy prevents direct use, and preserves the variation that could change the decision. It is not a random export assumed to be representative, and it is not a polished demo set made entirely of cases the feature already handles.

Begin with common cases so the experiment reflects the experience most users will have. Add known edge cases from support, QA, domain experts and production incidents. Add adversarial or stress cases that challenge instruction boundaries. Identify errors with asymmetric consequences: a false positive may be tolerable in one workflow while a false negative creates compliance exposure in another. Segment by the dimensions that change meaning—user type, language, document type, jurisdiction, conversation length or workflow stage.

Each case needs either an expected output or acceptance criteria. Exact expected values work for extraction fields and classifications. Open-ended recommendations usually need a rubric: required facts, prohibited claims, acceptable tone and a written account of what would make the output useful. Record uncertainty rather than forcing false precision. If experts legitimately disagree, preserve the disagreement and clarify which product policy resolves it.

Treat privacy as part of experiment design. Use the minimum data needed, redact identifiers consistently, restrict access, document transformations and verify that redaction does not remove the clues the product depends on. Synthetic cases can fill a coverage gap, but label them as reconstructed and do not present their results as production prevalence.

Worked example: expense-policy decisions

Suppose a team is deciding whether an AI agent can approve routine employee expenses. A weak set contains five ordinary restaurant receipts below the limit. A useful exploratory set also includes a receipt just above the limit, a missing attendee, a permissible exception, a duplicated charge, an ambiguous currency and a senior employee who should not receive different policy treatment. Each case includes the policy clause, expected approve/reject/escalate decision, and what evidence the explanation must cite.

Run the current prompt as baseline, then compare one structured prompt change on the same cases. Review decisions and reasons separately: the right decision for a fabricated reason is not trustworthy. The documented Lovelaice expense-policy experiment used seven cases, three prompt versions and six models—126 runs. Its best documented configuration reached six correct decisions out of seven, but seven cases remain exploratory evidence, not production readiness. Read the expense-policy experiment and limitations.

Product-specific quality describes the decision your product exists to make. Generic “helpfulness,” “relevance” or “hallucination” scores can be useful prompts for discussion, but they often miss the operational distinction users care about. A lead-qualification agent can sound helpful while assigning the wrong priority. A contract summary can be relevant while omitting the termination clause. A policy answer can be factually grounded while choosing the wrong exception.

Start in the language of domain expertise. Ask reviewers what would make the output unsafe, unusable or misleading; what must always be present; which mistakes deserve escalation; and whether some failures cost more than others. Turn those answers into observable rubric items. “Good answer” is difficult to apply consistently. “Names the controlling policy clause, reaches the correct decision and does not invent an exception” is reviewable.

Lovelaice’s battle of the LLM judges shows why the rubric must come from the product. The case study covered one lead-qualification product, 34 real responses with 17 correct and 17 incorrect human labels, eight judge prompts, ten models and two complete runs, producing more than 5,000 verdicts. The best purpose-built prompt × model pairing reached 76.5% agreement with the human expert. Seven generic templates averaged 52–58%, and approximately 8% of identical verdicts changed between runs.

Those numbers are not a universal benchmark for judge vendors, model families or other domains. They are detailed evidence from one product, one primary failure mode and one expert annotator. Their useful conclusion is methodological: a rubric built from real product errors, validated against human labels and tested as a prompt × model pairing gave this team more decision-relevant evidence than generic templates. Another team must validate its own rubric on its own labelled cases.

Quality criteria should also encode severity. Report false positives and false negatives separately when they have different consequences. Review results by segment, not just in aggregate. Define hard release gates for unacceptable failures and softer targets for preferences. That lets a team say, for example, “no unsupported compliance claims, at least 90% field completeness, and no regression for German-language invoices,” instead of chasing one blended score.

The unit of comparison is often the prompt × model pairing. The same instruction can be interpreted differently across models, while a prompt designed for one model may underuse another. Record both versions and judge the pairing as deployed. When you want to attribute an effect to the prompt alone, keep the model and settings fixed; when choosing a production configuration, compare named pairings under the same dataset and rubric.

Fair comparisons use identical cases, the same evaluation criteria, recorded model versions and controlled settings. Run variants close enough together that provider or dataset changes do not quietly alter conditions. Keep system prompts, retrieval context and tool access visible. Track failures at case level so an aggregate gain cannot hide a regression on the segment that matters most.

Blind review reduces preference for newer or better-known model brands. This is a practical lesson from Lovelaice’s founder experience, not an independent behavioural study: when reviewers know which response came from the fashionable model, reputation can influence how they interpret polish. Hide model identity during quality review, then reconnect the scores to cost, latency and operational constraints after the judgments are recorded.

Side-by-side review also reveals tacit requirements. Domain experts often notice that one answer prioritizes correctly, asks a safer follow-up, uses a clearer structure or avoids a phrase that could confuse users—details they did not think to put in the initial prompt. Capture these observations as candidate rubric items, then re-run the complete set. Do not simply crown the answer that inspired the most preferences.

The newest or most expensive model cannot be assumed to be best for a specific task. The Lovelaice document-extraction comparison found an 11× per-document cost range across 15 models on its documented invoice workload. That result is task-specific and pricing is time-sensitive; it supports benchmarking plausible models on the real workload, not a general ranking. The prompt-formatting comparison likewise treats Markdown, XML and JSON structure as variables to measure, not universal style rules.

Manual review is the starting point when expected outputs do not yet exist. It is not an immature workaround. Reading outputs is how a team discovers what its product actually gets wrong, which failures matter and how experts distinguish acceptable from unacceptable. Automating a generic score before doing this work creates precise-looking measurements with no proven relationship to the product decision.

Begin with a small set and ask reviewers to leave a written reason for each pass, fail or preference. Compare notes. Where reviewers disagree, clarify the rubric rather than averaging away the disagreement. Group recurring reasons into a failure taxonomy. Objective patterns—invalid JSON, missing fields, forbidden phrases or length limits—can become deterministic checks. Nuanced judgments can become a carefully written LLM judge only after enough human-labelled examples exist to validate it.

An automated evaluator earns trust by reproducing expert judgment on held-out cases, not by returning a confident score. Measure agreement, missed real errors, false alarms, reason quality and repeat-run consistency. Revalidate after changing the judge prompt or model. Keep sampled human review in the lifecycle because product behaviour, user inputs and expert standards evolve.

This workflow gives non-technical experts a concrete route through AI overwhelm. They do not need to know every model architecture. They need access to the prompt, representative cases and real outputs, plus a way to express why something works or fails. Engineering remains essential to integration and operations; experimentation lets product and domain experts contribute their unique context directly instead of treating AI as a magical component behind a ticket queue.

A shipping decision needs four layers of measurement: product quality, failure profile, operational performance and reliability. A single average accuracy number cannot describe all four. Keep the layers separate long enough to understand the trade-off, then make the product decision explicit.

The business case spans acceptable quality, cost, and latency at expected scale. A configuration with the highest quality score may make the product unusably slow. The cheapest model may require enough retries or human correction to cost more in practice. State minimum quality gates first, then compare operational trade-offs among the configurations that clear them. Do not trade away a critical safety requirement for an attractive blended average.

Confidence should be proportionate to risk. A three-case run can decide which prompt direction deserves another afternoon. It cannot justify automating a regulated decision. A larger set does not fix poor coverage, and statistical significance is not one mandatory threshold for every offline experiment. Ask whether the evidence could change the decision, whether important segments have enough coverage, whether repeated runs remain acceptable, and which unknowns are left.

Report denominators and scope beside every number. “86% accuracy” means more when the reader can see that it was six of seven expense cases in an exploratory reconstruction. “76.5% agreement” should sit beside the 34 balanced lead-qualification responses and one-product limitation. Clear scope makes evidence useful; hiding it turns a case study into marketing.

The experiment is complete when the team can make and defend the stated decision. The deliverable is not a leaderboard. It is a short record connecting the selected configuration to the baseline, dataset, threshold, remaining risks and operating plan. Someone revisiting the feature months later should understand what was known at release and what requires revalidation.

“Ship” can mean a limited rollout with human review rather than unrestricted automation. If a variant clears ordinary cases but retains a rare severe failure, route the risky segment to an expert or keep the existing workflow. Define rollback conditions before launch. Monitoring should capture the same quality concepts used offline so production evidence can enter the next experiment instead of living in a disconnected dashboard.

If the threshold is missed, document that too. A rejected configuration prevents the same weak approach from being rediscovered and shows what evidence would reopen the decision. Sometimes the right outcome is a narrower feature, more context, a deterministic rule around the model, or no AI at all. Experimentation earns its cost when it finds those answers before a production commitment.

AI quality is cross-functional ownership. Product defines the user outcome and the decision the experiment must support. Domain experts define nuanced quality, acceptable exceptions and failure severity. Engineering ensures technical feasibility, integration, version control and operational measurement. Design assesses usefulness, interaction, tone, transparency and recovery. Legal or compliance defines applicable constraints in high-risk domains.

These responsibilities overlap by design. Product should not invent a compliance rubric alone; engineering should not select a model solely from a public benchmark; domain experts should not be asked to approve an unexplained aggregate score. A review session where each discipline sees the same cases creates a shared language for trade-offs. The decision record makes the ownership durable after the meeting.

Giving non-technical colleagues direct access to experimentation does not replace engineering. It reduces lossy translation. A product manager can compare outputs and sharpen the intended outcome; a claims expert can label a subtle omission; a designer can identify a recovery failure. Engineers can then implement against specific evidence and focus on system constraints that require their expertise.

Lovelaice’s experience is that teams range from an early idea to complex existing pipelines, so ownership should match maturity. At the idea stage, product and domain experts may lead a small manual comparison with engineering advising feasibility. Near release, engineering and risk owners deepen dataset controls, operational measurements, fallback behaviour and monitoring. The loop stays the same; the rigor expands with the decision.

The four records below are first-party Lovelaice experiments. They are examples of method and documented evidence, not universal model rankings. Each record states the decision, dataset, result and limitation so you can judge whether its conclusion transfers to your context. Use the source article for full methodology before relying on a number.

These concise answers cover the decisions teams most often face when starting or formalising AI experimentation. Each answer uses the same definitions and thresholds as the guide above.