A live AI screening agent,
stuck below 20% accuracy.
A sustainability-tech company building an automated business-screening tool. Their AI agent reads public information about a business and decides whether it meets a defined set of sustainability criteria. The end customer is a financial institution or a procurement team that needs to filter thousands of businesses against ESG rules.
Live in production. Inputs span multi-source unstructured web content per business, with a structured rubric the agent has to apply to deliver a yes/no decision the customer can defend.
Where the gaps were
The team knew accuracy was the problem. They had tried five different model providers — Anthropic, OpenAI, Gemini, DeepSeek, Perplexity. None of them moved the needle. The most expensive run cost $74 for 226 calls and scored 0% reliability. The cheapest cost less than a cent and scored the same. They were on iteration 24 of model selection with no measured progress.
If you have shipped an AI feature that “kind of works,” some of this will sound familiar:
- No measured baseline. An accuracy number was missing, and “let's try a newer model” was the default lever.
- Quality treated as one fuzzy concept rather than a set of named, gradable criteria.
- Cross-provider iteration without a control. Each new model was a hope rather than a comparison.
- Source documents and evaluation criteria were drifting apart, so even a perfect model would have hit the same ceiling.
What Lovelaice did.
Three weeks, ~120 minutes of their team's time, seven steps:
- 1.Ran the baseline experiment on their production setup. Result: 18% accuracy across all five model providers. The first measured number anyone on the team could point to.
- 2.Held a 30-minute working session with their product and domain leads to name what “correct” actually meant on a per-criterion basis. Found two sources of ambiguity in the ESG criteria themselves that no model could have resolved.
- 3.Rebuilt the evaluation rubric against the cleaned-up criteria. Re-labelled a focused test set so two domain experts agreed on every row before any model was retested.
- 4.Ran a targeted prompt iteration loop. The hypothesis was no longer “find a better model” — it was “find the prompt structure that lets the strongest model actually do the task.”
- 5.Benchmarked the new prompt across the same five providers, then narrowed to three. Cost per call dropped from $0.12 on the best earlier variant to a fraction of a cent on the eventual winner.
- 6.Found the production winner: GPT-5 Mini at 93% accuracy. Not GPT-5. Not Claude Sonnet. The cheaper “one grade down” model — once the task was framed correctly.
- 7.60-minute review with their team. Production deployment plan in hand at the end of the meeting.
Before and after.
| Metric | Before | After |
|---|---|---|
| Top accuracy across all models tested | 18% | 93% · +75pp |
| Best per-call cost on a passing model | $0.12 (failing at 20.8%) | Fractions of a cent on GPT-5 Mini at 93% |
| Production candidate | None — every provider scored under 30% | GPT-5 Mini, production-ready, error patterns named for follow-up |
| Iteration approach | Swap the model, hope it works | Targeted prompt iteration against a stable rubric |
| Ground-truth dataset | Inconsistent labels across team members | Re-labelled, two-expert-agreed test set. Theirs to keep. |
| Total spend to find the answer | ~$210 across 24 prior iterations, no answer | ~$25 in the final productive iterations |
| Providers benchmarked | 5 (with conflicting signal) | 5, with the per-prompt × per-model performance grid as data |
What they walked away with.
- A measured baseline at 18% accuracy — the first time the team had a number they could iterate against.
- A re-labelled ground-truth dataset with two-expert agreement on every row. Theirs to keep and reuse forever.
- A cleaned-up evaluation rubric mapped directly to their ESG criteria, with the two sources of ambiguity resolved at the source.
- A five-provider × multi-prompt benchmark grid. The team now sees exactly where each model lands on the cost-quality curve for this specific task.
- A production-ready prompt and model selection: GPT-5 Mini at 93% accuracy, on a cheaper tier than any earlier attempt.
- A shortlist of three remaining error patterns with the exact cases that triggered them. Next iteration is targeted, not exploratory.
The bonus: the eval rubric and golden dataset stay with them. Every future model that comes out, they can re-run the benchmark on their data in an afternoon — without us.
What this would have costed in-house.
25 iterations is the number that matters. The team had already burned 24 of them before we arrived. Here is the math, against the Eval Setup & Optimization package.
| Scenario | Engineering | PM / Leadership | Fully-loaded | vs. €4,999 |
|---|---|---|---|---|
| Best case focused AI engineer, no interrupts | ~10 days × €450 = €4,500 | minimal | ~€4,500 | roughly even |
| Realistic stop-start, no eval infrastructure | ~30 days × €450 = €13,500 | ~20 days × €450 = €9,000 | ~€22,500 | ~4.5× more |
Best case. A focused AI engineer with eval methodology in hand could land the production answer in roughly 10 days of dedicated work. At a fully-loaded ~€450/day that is ~€4,500 — roughly the same as the engagement, but the team would still need to build the rubric, the dataset, and the benchmark grid from scratch. Those stay with you when we leave.
Realistic. This is what actually happens. Teams iterating on AI features without an eval framework move at roughly two weeks per cycle: try a model, eyeball outputs, debate quality, ship a tweak, learn from production. 25 cycles at that pace is the better part of a year of stop-start work — which is roughly the trajectory this team was on before we started.
Neither figure prices in the opportunity cost of selling a 20%-accurate ESG screening tool to financial customers, or the trust capital spent every time a wrong answer reaches a customer's compliance review.
If any of these feel familiar.
- Your AI feature is live, and the team is on its third or fourth model swap with no measured progress
- Two engineers will defend different models in the same meeting, both with conviction, neither with a comparable number
- “Just use a newer model” has been the default lever for more than one quarter
- Your domain criteria live in a Notion doc, three Slack threads, and one expert's head — and no two team members would grade an output the same way
- You have tested models on public benchmarks but never on your data, in your task, against your criteria
- The cost of being wrong on an answer is high enough that “kind of works” is not a deployable position
- You suspect the answer is a cheaper model than the one you have been throwing at the problem
Then the same engagement applies: three weeks, ~120 minutes of your team's time. You keep the golden dataset, the eval rubric, the schema, and the per-model performance grid for your task.
More results from real projects.
Document extraction — 52% to 88% accuracy in three weeks
A 20-point eval rubric, 8 prompt iterations, and a model benchmark across 4 providers — on the same model family they were already running.
Read the case study →Healthtech recommendations — the failure no one was looking for
Running the golden set systematically surfaced a consistent bias one-by-one review can’t see — and a cheaper model that measured better on their data.
Read the case study →