— Who they are
A live AI feature,
without a measurement baseline.
A Series A B2B software company serving industrial customers. Their AI feature ingests source documents from their customers and converts them into structured JSON that end users execute inside the UI.
Live in production. Inputs span multiple languages and carry dense, domain-specific vocabulary alongside complex visual elements.
Where the gaps were
The team could see that the output had room to improve. End customers were manually editing the AI-extracted data inside the UI before passing it downstream. Improvements were on the roadmap for the quarter, but no clear action plan. Without a baseline, each prompt change was hard to evaluate.
If you're running an AI feature in production, some of these may sound familiar:
- No measured baseline yet. No single accuracy number — only subjective quality assessment.
- Quality discussed as one broad concept rather than a set of named, fixable failure modes.
- The default lever is trying a newer model, even when the gains are somewhere else.
- The rarer input formats fail in ways no one catches until a customer reports it.
— What it cost them
Customers were cleaning up almost every document by hand.
Each manual edit was a small tax on every document processed. Without a measurement in place, the team had no way to tell which prompt changes actually reduced that tax, and which ones quietly made it worse.
— The engagement
What Lovelaice did.
Three weeks, ~120 minutes of their team's time, seven steps:
- 1.Ingested 18 real, very complex (10–100+ pages) documents spanning their full format mix.
- 2.Ran a 30-minute working session with their CTO and PM to name what “good” actually looked like and embed their expertise.
- 3.Turned that into a 20-point automatic eval rubric. Each point captured one precise, named failure mode. The team could finally answer questions like “how well does the AI feature manage required sections?” with a number.
- 4.Ran a baseline on their production model and prompt against the new rubric. Result: 52% accuracy. The first concrete number for the status quo.
- 5.8 iterations across 7 prompt versions, each targeting named failure patterns.
- 6.Benchmarked 18 models across 4 providers (OpenAI, Azure, Anthropic, Gemini) on the same 18-document dataset. The most expensive model in the field cost ~200× more per document than the cheapest. Several of the cheapest models landed within 2pp of the winner on accuracy.
- 7.60-minute review of the full report with their team.
— The result
Before and after.
+36pp
Accuracy lift. 52% to 88%, on the same model family.
5
Leading models within 2% of the winner. Pick by cost or latency.
~20
20-point automatic eval rubric, running on every future change.
| Metric | Before | After |
|---|
| Accuracy, production model | 52% | 88% · +36pp |
| Leading models within 2pp of winner | 1 | 5 · choose by cost or latency |
| Failure modes measurable | 0 | 20-point automatic eval rubric, each point tied to a precise failure mode |
| Named failure patterns completely fixed | 0 | 11 patterns |
| Reusable golden test set | None | 18 documents covering their format mix. Theirs to keep. |
| Images & diagrams in the output | Often dropped or attached to the wrong step | Anchored to the correct step automatically |
| Models benchmarked on domain data | 1 | 18 across 4 providers |
— Same dataset, same rubric
The accuracy win came on the same model family they were already running.
We didn't migrate models, switch providers, or rewrite the architecture. The unlock was a structured and detailed prompt, an eval framework, a named taxonomy of failure modes, and targeted iteration against it.
— Deliverables
What they walked away with.
- A measured baseline. 52% accuracy, the first time anyone had a number.
- A named taxonomy of ~20 failure modes, with a check behind each one. Future iteration is targeted.
- An 18-document golden test set covering their full format mix. Theirs to keep and reuse forever.
- A 20-point automatic eval rubric that runs live at runtime and against the golden dataset, on every future change.
- A model benchmark across 18 candidates and 4 providers. The most expensive model cost ~200× more per document than the cheapest, and the team now sees exactly where each one sits on the cost-quality curve.
- A production-ready prompt and JSON schema hitting 88% on the model they were already running.
— Lovelaice
Without measurement, every prompt change is hard to evaluate.
They had this improvement on their roadmap for months. We delivered it in a few weeks.
The cost to them, in time, was ~120 minutes.
As a bonus, they now know which of the 5 leading models is the right cost-quality choice for their use case, with the data to defend that choice.
— The math
What this would have costed in-house.
Eight iterations of measure, fix, and remeasure. Here is the time that would have taken in-house, against the 3 weeks and ~120 minutes of their team's time we delivered it in.
— Important context
In practice, this path wasn't available to them. No dedicated AI engineer on staff, and no in-house eval methodology of this shape. The numbers below assume they could hire or borrow that capacity internally.
| Scenario | Engineering time | PM / Leadership time | Calendar time | vs. Lovelaice |
|---|
Best case
focused engineer, no interrupts | ~16 days
8 iterations × 2 days | minimal | ~3–6 weeks
focused sprint | ~2× longer, unbounded team time |
Realistic
stop-start, no eval infrastructure | ~24 days
8 iterations × ~3 days | ~16 days
8 iterations × ~2 days | ~4 months
stop-start work | ~6× longer, ~200× more team hours |
Lovelaice
eval infrastructure, on their behalf | handled | ~120 minutes total
1 kickoff + 1 review | 3 weeks
end-to-end | baseline |
Best case. The fastest we've ever seen a focused AI engineer run a single iteration of this kind is roughly 2 days. Eight iterations × 2 days = ~16 days of dedicated engineering time, which translates into a 3–6 week sprint once you factor in reviews and calendar friction. And at the end of that sprint you still wouldn't have a model benchmark across 18 candidates, a golden test set, or a hardened pipeline for the trickier inputs.
Realistic. Teams without eval infrastructure tend to move slower. Build a change, review outputs by eye, debate whether it's better, ship, then learn from downstream signals. In our experience each cycle absorbs ~3 engineering days plus ~2 PM/leadership days, and stretches across ~14 calendar days. Across 8 iterations that adds up to ~40 person-days of team effort, spread over ~4 months of stop-start work. Almost exactly how long this had been on their roadmap before we started.
Lovelaice. 3 weeks end-to-end. ~120 minutes of the team's time. Everything else — the ingestion, the rubric build, the 8 iterations, the 18-model benchmark — happens outside their calendar.
Neither in-house figure counts the customer-success time spent on extraction issues downstream, the opportunity cost of a 4-month feature delay, or the deliverables you'd still be missing at the end.
— Where this fits
If any of these feel familiar.
- You don't yet have a single accuracy number you'd be ready to defend
- Quality is still discussed as one broad problem rather than a set of named, fixable failure modes
- Your iteration loop leans on eyeballing outputs more than on automated checks
- You've been told “just use a newer model” and you suspect the real answer is somewhere else
- Your customers, or your CS team, are catching things downstream that the model should have handled
- You've compared models on public benchmarks but not on your data
- Your trickier inputs fail in ways you can't easily track
Then the same engagement applies: three weeks, ~120 minutes of your team's time. You keep the golden dataset, the eval rubric, the schema, and any new capabilities we stand up along the way.
— Next step
Book a demo, and we will scope your engagement against the same playbook.
Let's unlock the next level together.
— Related case studies
More results from real projects.