— Who they are
A live AI screening agent,
stuck below 20% accuracy.
A sustainability-tech company building an automated business-screening tool. Their AI agent reads public information about a business and decides whether it meets a defined set of sustainability criteria. The end customer is a financial institution or a procurement team that needs to filter thousands of businesses against ESG rules.
Live in production. Inputs span multi-source unstructured web content per business, with a structured rubric the agent has to apply to deliver a yes/no decision the customer can defend.
Where the gaps were
The team knew accuracy was the problem. They had tried five different model providers — Anthropic, OpenAI, Gemini, DeepSeek, Perplexity. None of them moved the needle. The most expensive run cost $74 for 226 calls and scored 0% reliability. The cheapest cost less than a cent and scored the same. They were on iteration 24 of model selection with no measured progress.
If you have shipped an AI feature that “kind of works,” some of this will sound familiar:
- No measured baseline. An accuracy number was missing, and “let's try a newer model” was the default lever.
- Quality treated as one fuzzy concept rather than a set of named, gradable criteria.
- Cross-provider iteration without a control. Each new model was a hope rather than a comparison.
- Source documents and evaluation criteria were drifting apart, so even a perfect model would have hit the same ceiling.
— The real cost
Every “wrong” answer hit the end customer’s decision pipeline.
An ESG screening tool that is wrong 80% of the time is not a tool the financial customer can defend. The product was live, but the team could not honestly recommend it at scale.
— The engagement
What Lovelaice did.
Three weeks, ~120 minutes of their team's time, seven steps:
- 1.Ran the baseline experiment on their production setup. Result: 18% accuracy across all five model providers. The first measured number anyone on the team could point to.
- 2.Held a 30-minute working session with their product and domain leads to name what “correct” actually meant on a per-criterion basis. Found two sources of ambiguity in the ESG criteria themselves that no model could have resolved.
- 3.Rebuilt the evaluation rubric against the cleaned-up criteria. Re-labelled a focused test set so two domain experts agreed on every row before any model was retested.
- 4.Ran a targeted prompt iteration loop. The hypothesis was no longer “find a better model” — it was “find the prompt structure that lets the strongest model actually do the task.”
- 5.Benchmarked the new prompt across the same five providers, then narrowed to three. Cost per call dropped from $0.12 on the best earlier variant to a fraction of a cent on the eventual winner.
- 6.Found the production winner: GPT-5 Mini at 93% accuracy. Not GPT-5. Not Claude Sonnet. The cheaper “one grade down” model — once the task was framed correctly.
- 7.60-minute review with their team. Production deployment plan in hand at the end of the meeting.
— The result
Before and after.
+75pp
Accuracy lift. 18% to 93%, on a cheaper model than they had been testing.
5 → 3
Provider field narrowed to the three worth a serious test. Two ruled out by data, not gut.
~100×
Cost-per-call gap from worst to winning model. The winner was on the cheap end.
| Metric | Before | After |
|---|
| Top accuracy across all models tested | 18% | 93% · +75pp |
| Best per-call cost on a passing model | $0.12 (failing at 20.8%) | Fractions of a cent on GPT-5 Mini at 93% |
| Production candidate | None — every provider scored under 30% | GPT-5 Mini, production-ready, error patterns named for follow-up |
| Iteration approach | Swap the model, hope it works | Targeted prompt iteration against a stable rubric |
| Ground-truth dataset | Inconsistent labels across team members | Re-labelled, two-expert-agreed test set. Theirs to keep. |
| Total spend to find the answer | ~$210 across 24 prior iterations, no answer | ~$25 in the final productive iterations |
| Providers benchmarked | 5 (with conflicting signal) | 5, with the per-prompt × per-model performance grid as data |
— Same task, same providers
The accuracy win came on a cheaper model from a provider already in their bench.
We didn't introduce a new vendor, switch architectures, or upgrade to a frontier model. The unlock was a clean evaluation rubric, a corrected ground-truth set, and prompt iteration aimed at the actual task — not at the model.
— Deliverables
What they walked away with.
- A measured baseline at 18% accuracy — the first time the team had a number they could iterate against.
- A re-labelled ground-truth dataset with two-expert agreement on every row. Theirs to keep and reuse forever.
- A cleaned-up evaluation rubric mapped directly to their ESG criteria, with the two sources of ambiguity resolved at the source.
- A five-provider × multi-prompt benchmark grid. The team now sees exactly where each model lands on the cost-quality curve for this specific task.
- A production-ready prompt and model selection: GPT-5 Mini at 93% accuracy, on a cheaper tier than any earlier attempt.
- A shortlist of three remaining error patterns with the exact cases that triggered them. Next iteration is targeted, not exploratory.
— Lovelaice
Without measurement, every model swap is a hope.
The team had been on iteration 24 of model selection before we started. They moved from “all five providers fail” to “GPT-5 Mini is the production answer” in a focused engagement.
The bonus: the eval rubric and golden dataset stay with them. Every future model that comes out, they can re-run the benchmark on their data in an afternoon — without us.
— The math
What this would have costed in-house.
25 iterations is the number that matters. The team had already burned 24 of them before we arrived. Here is the time that would have taken in-house, against the 3 weeks and ~120 minutes of their team's time we delivered it in.
— Important context
The team's first 24 iterations didn't get them to a production answer. The math below assumes a team with the same eval discipline we bring — most teams iterating without it stall exactly the way this team did.
| Scenario | Engineering time | PM / Leadership time | Calendar time | vs. Lovelaice |
|---|
Best case
focused engineer, no interrupts | ~10 days
dedicated work | minimal | ~3–6 weeks
focused sprint | ~2× longer, unbounded team time |
Realistic
stop-start, no eval infrastructure | ~30 days
25 cycles × ~1–2 days | ~20 days
25 cycles × ~1 day | ~1 year
stop-start work | ~17× longer, ~250× more team hours |
Lovelaice
eval infrastructure, on their behalf | handled | ~120 minutes total
1 kickoff + 1 review | 3 weeks
end-to-end | baseline |
Best case. A focused AI engineer with eval methodology in hand could land the production answer in roughly 10 days of dedicated work, which stretches into a 3–6 week sprint once you factor in reviews and calendar friction. At the end of that sprint the team would still need to build the rubric, the golden dataset, and the benchmark grid from scratch. Those stay with you when we leave.
Realistic. This is what actually happens. Teams iterating on AI features without an eval framework move at roughly two weeks per cycle: try a model, eyeball outputs, debate quality, ship a tweak, learn from production. Across 25 cycles that adds up to ~50 person-days of team effort, spread over a year of stop-start work — which is roughly the trajectory this team was on before we started.
Lovelaice. 3 weeks end-to-end. ~120 minutes of the team's time. The 18-model benchmark, the rubric build, the iteration loop — all happen outside their calendar.
Neither in-house figure counts the opportunity cost of selling a 20%-accurate ESG screening tool to financial customers, or the trust capital spent every time a wrong answer reaches a customer's compliance review.
— Where this fits
If any of these feel familiar.
- Your AI feature is live, and the team is on its third or fourth model swap with no measured progress
- Two engineers will defend different models in the same meeting, both with conviction, neither with a comparable number
- “Just use a newer model” has been the default lever for more than one quarter
- Your domain criteria live in a Notion doc, three Slack threads, and one expert's head — and no two team members would grade an output the same way
- You have tested models on public benchmarks but never on your data, in your task, against your criteria
- The cost of being wrong on an answer is high enough that “kind of works” is not a deployable position
- You suspect the answer is a cheaper model than the one you have been throwing at the problem
Then the same engagement applies: three weeks, ~120 minutes of your team's time. You keep the golden dataset, the eval rubric, the schema, and the per-model performance grid for your task.
— Next step
Book a demo, and we will scope your engagement against the same playbook.
Reference call with the customer in this case study available on request.
— Related case studies
More results from real projects.