A live AI feature,
without a measurement baseline.
A Series A B2B software company serving industrial customers. Their AI feature ingests source documents from their customers and converts them into structured JSON that end users execute inside the UI.
Live in production. Inputs span multiple languages and carry dense, domain-specific vocabulary alongside complex visual elements.
Where the gaps were
The team could see that the output had room to improve. End customers were manually editing the AI-extracted data inside the UI before passing it downstream. Improvements were on the roadmap for the quarter, but no clear action plan. Without a baseline, each prompt change was hard to evaluate.
If you're running an AI feature in production, some of these may sound familiar:
- No measured baseline yet. No single accuracy number — only subjective quality assessment.
- Quality discussed as one broad concept rather than a set of named, fixable failure modes.
- The default lever is trying a newer model, even when the gains are somewhere else.
- The rarer input formats fail in ways no one catches until a customer reports it.
What Lovelaice did.
Three weeks, ~120 minutes of their team's time, seven steps:
- 1.Ingested 18 real, very complex (10–100+ pages) documents spanning their full format mix.
- 2.Ran a 30-minute working session with their CTO and PM to name what “good” actually looked like and embed their expertise.
- 3.Turned that into a 20-point automatic eval rubric. Each point captured one precise, named failure mode. The team could finally answer questions like “how well does the AI feature manage required sections?” with a number.
- 4.Ran a baseline on their production model and prompt against the new rubric. Result: 52% accuracy. The first concrete number for the status quo.
- 5.8 iterations across 7 prompt versions, each targeting named failure patterns.
- 6.Benchmarked 18 models across 4 providers (OpenAI, Azure, Anthropic, Gemini) on the same 18-document dataset. The most expensive model in the field cost ~200× more per document than the cheapest. Several of the cheapest models landed within 2pp of the winner on accuracy.
- 7.60-minute review of the full report with their team.
Before and after.
| Metric | Before | After |
|---|---|---|
| Accuracy, production model | 52% | 88% · +36pp |
| Leading models within 2pp of winner | 1 | 5 · choose by cost or latency |
| Failure modes measurable | 0 | 20-point automatic eval rubric, each point tied to a precise failure mode |
| Named failure patterns completely fixed | 0 | 11 patterns |
| Reusable golden test set | None | 18 documents covering their format mix. Theirs to keep. |
| Images & diagrams in the output | Often dropped or attached to the wrong step | Anchored to the correct step automatically |
| Models benchmarked on domain data | 1 | 18 across 4 providers |
What they walked away with.
- A measured baseline. 52% accuracy, the first time anyone had a number.
- A named taxonomy of ~20 failure modes, with a check behind each one. Future iteration is targeted.
- An 18-document golden test set covering their full format mix. Theirs to keep and reuse forever.
- A 20-point automatic eval rubric that runs live at runtime and against the golden dataset, on every future change.
- A model benchmark across 18 candidates and 4 providers. The most expensive model cost ~200× more per document than the cheapest, and the team now sees exactly where each one sits on the cost-quality curve.
- A production-ready prompt and JSON schema hitting 88% on the model they were already running.
The cost to them, in time, was ~120 minutes.
As a bonus, they now know which of the 5 leading models is the right cost-quality choice for their use case, with the data to defend that choice.
What this would have costed in-house.
Eight iterations of measure, fix, and remeasure. Here is the time that would have taken in-house, against the 3 weeks and ~120 minutes of their team's time we delivered it in.
| Scenario | Engineering time | PM / Leadership time | Calendar time | vs. Lovelaice |
|---|---|---|---|---|
| Best case focused engineer, no interrupts | ~16 days 8 iterations × 2 days | minimal | ~3–6 weeks focused sprint | ~2× longer, unbounded team time |
| Realistic stop-start, no eval infrastructure | ~24 days 8 iterations × ~3 days | ~16 days 8 iterations × ~2 days | ~4 months stop-start work | ~6× longer, ~200× more team hours |
| Lovelaice eval infrastructure, on their behalf | handled | ~120 minutes total 1 kickoff + 1 review | 3 weeks end-to-end | baseline |
Best case. The fastest we've ever seen a focused AI engineer run a single iteration of this kind is roughly 2 days. Eight iterations × 2 days = ~16 days of dedicated engineering time, which translates into a 3–6 week sprint once you factor in reviews and calendar friction. And at the end of that sprint you still wouldn't have a model benchmark across 18 candidates, a golden test set, or a hardened pipeline for the trickier inputs.
Realistic. Teams without eval infrastructure tend to move slower. Build a change, review outputs by eye, debate whether it's better, ship, then learn from downstream signals. In our experience each cycle absorbs ~3 engineering days plus ~2 PM/leadership days, and stretches across ~14 calendar days. Across 8 iterations that adds up to ~40 person-days of team effort, spread over ~4 months of stop-start work. Almost exactly how long this had been on their roadmap before we started.
Lovelaice. 3 weeks end-to-end. ~120 minutes of the team's time. Everything else — the ingestion, the rubric build, the 8 iterations, the 18-model benchmark — happens outside their calendar.
Neither in-house figure counts the customer-success time spent on extraction issues downstream, the opportunity cost of a 4-month feature delay, or the deliverables you'd still be missing at the end.
If any of these feel familiar.
- You don't yet have a single accuracy number you'd be ready to defend
- Quality is still discussed as one broad problem rather than a set of named, fixable failure modes
- Your iteration loop leans on eyeballing outputs more than on automated checks
- You've been told “just use a newer model” and you suspect the real answer is somewhere else
- Your customers, or your CS team, are catching things downstream that the model should have handled
- You've compared models on public benchmarks but not on your data
- Your trickier inputs fail in ways you can't easily track
Then the same engagement applies: three weeks, ~120 minutes of your team's time. You keep the golden dataset, the eval rubric, the schema, and any new capabilities we stand up along the way.
More results from real projects.
ESG screening agent — 18% to 93% accuracy, 25 experiments
All five top-tier providers failed equally. The production answer turned out to be a cheaper model with a clean rubric and a corrected ground-truth set.
Read the case study →Healthtech recommendations — the failure no one was looking for
Running the golden set systematically surfaced a consistent bias one-by-one review can’t see — and a cheaper model that measured better on their data.
Read the case study →