The Lovelaice glossary

A six-step methodology for AI evaluation where teams earn each step before automating, ending with deterministic checks and a surgical, validated LLM-as-judge built on documented failure patterns.

An AI feature is one capability inside a product that already does a job. An AI product is the whole thing built around the model. The distinction decides what a mediocre model output actually costs — being ignored, or eroding trust in the surrounding product.

Productivity use is you using an LLM to move faster in your own work, where a human reads every output. AI at scale is the same model running inside your product thousands of times a day with nobody checking. The techniques that work for one break silently in the other.

The pattern of deploying AI features based on a handful of happy-path demos, without systematic validation, and hoping production matches the demo. The default mode for most teams shipping their first AI feature.

AI quality issues that produce confident-looking but useless or wrong output. They don't trigger errors or user reports — users just lose trust and quietly stop using the feature, so the failure never appears in your metrics.

Eyeballing a handful of AI outputs and concluding the model 'seems fine.' The dominant industry practice for AI quality assessment — and the practice that systematic evaluation replaces.

The person who knows what 'good' looks like for your users — the PM, underwriter, lawyer, clinician, support lead. The model brings general intelligence; the domain expert brings what makes an answer useful in a specific context.

The agreed correct answer for a given input, written down by someone who actually knows. Without ground truth, you can't score anything — you can only vibe-check.

A curated, stable set of real inputs paired with ground truth — the reference set you re-run after every change. Usually 50–200 cases, deliberately including the ugly edge cases rather than the demo-friendly ones.

The quality drop that happens as you give a model more context. A million-token window doesn't use all of it equally well — accuracy falls, instructions in the middle get ignored, and the model leans on the wrong parts. It degrades gradually, so nobody notices the day it started.

Testing an AI feature against your golden dataset before it reaches users, so you know how it behaves at scale rather than how it behaved in a demo. The step almost nobody does — and the only one that prevents releasing an AI that disappoints your users.

Watching what your AI actually did in production — traces, inputs, outputs, latency, cost, errors. Necessary, and not the same as evaluation. Observability tells you what happened; it can't tell you whether the answer was any good.

The full record of one AI interaction: input, retrieved context, model calls, tool calls, output. Without traces you can only see that something failed, never why.

Something that worked yesterday and doesn't today, because of a change you made somewhere else. In AI, one extra instruction can reroute everything around it — so re-testing only the case you fixed is how you ship regressions and find out from a customer.

A repeatable test that scores AI output against a definition of good, run across many cases in your dataset. The closest thing to a unit test for a probabilistic system — without it, every iteration is a guess dressed up as a decision.

A check with one right answer that a computer can verify: exact match, valid JSON, a number within tolerance, a required field present, a forbidden phrase absent. Cheap, instant, reproducible, and it never disagrees with itself.

Using a model to score another model's output against written criteria, for things you can't check deterministically — tone, completeness, whether the answer addresses the real question. Only as good as the criteria you write, and only trustworthy after you've validated it against human-scored examples.

One full pass of a given prompt, model, and configuration over your whole golden dataset — scored and saved. The unit of progress in AI development. Comparing two experiments is the only honest way to say an iteration made things better.

Reading real failures and sorting them into named categories, rather than just counting them. '82% accurate' tells you nothing you can act on. 'Fails on scanned documents older than 2015' tells you exactly what to build next.

A recurring, nameable way your feature breaks. Patterns are fixable. Individual bad outputs are anecdotes — and anecdotes are what your team is currently trading in Slack threads.

The public leaderboards — MMLU, GPQA, SWE-bench, LMArena. Standardized tests that rank models on general tasks. Useful for narrowing a shortlist, useless for picking a winner: a model topping the chart can still be the worst option for your specific task.

The same idea as a model benchmark, run on your data, your constraints, your users' language: your golden dataset, scored across candidate models and prompts. The number that should decide anything — and it routinely disagrees with the public leaderboards.

In classic SaaS, margins improve as usage grows. In AI, every call costs money — so your most engaged users are your most expensive ones. Quality and cost are one conversation, not two.