Evals in CI are only as smart as the engineer who wrote the assertion

There is a clean, appealing idea making the rounds: put your LLM evals in CI. Write them as ordinary tests with pytest or Vitest, run them on every commit, and let a red build stop a bad model from shipping. It is a good engineering practice. It is also a trap if it becomes the whole evaluation strategy, because a CI test only checks the thing the person who wrote it already knew to worry about.

That person is almost always an engineer. And the failures that sink AI features in fintech, healthtech, and insurance are precisely the ones an engineer did not know to worry about.

A green build is not a correct answer

CI is built on a promise that has held for decades of deterministic software: if the assertions pass, the behavior is correct. assert response.status == 200. assert total == sum(line_items). The test encodes a rule, the rule is either satisfied or it is not, and a passing suite means the code does what the author specified.

AI breaks the second half of that promise. The model returns a fluent, well-formatted, confident answer every time. The HTTP status is 200. The JSON parses. The output "looks like" a good answer to anyone reading it quickly. An engineer writing an eval-as-test asserts on the things engineers can assert on: schema validity, length, the presence of a required field, a keyword match, maybe a similarity score against a reference. Those checks go green. And the answer is still wrong in the way that matters: it recommended a treatment where it should have deferred to a clinician, it cited the wrong policy clause, it answered a question it was supposed to refuse.

None of those failures trips an assertion, because nobody wrote the assertion. You cannot write a test for a failure mode you have not imagined. The engineer imagines the failure modes an engineer would have. The clinician, the underwriter, and the compliance officer imagine the ones that end up in front of a regulator.

The rubric is the real test, and the CI job just runs it

Here is the distinction that gets lost when evals collapse into "just write them as tests." There are two separate jobs, and the industry keeps treating them as one.

The first job is deciding what "correct" means for your domain. What a competent practitioner would have done. Which errors are cosmetic and which are disqualifying. When the AI should refuse outright. This is the rubric, and it is a judgment problem. Only the person who holds the domain judgment can author it.

The second job is running that rubric automatically on every change so nothing ships without being scored. This is a plumbing problem, and CI is a perfectly good place to solve it.

"Evals in CI" quietly assumes the first job is trivial: that whoever writes the test also knows what to test for. In a coding agent, fine: the engineer is the domain expert, and the two jobs really are one person's job. Everywhere else, the person who can write a pytest fixture and the person who knows what a correct answer looks like are two different people, and the CI framing hands the whole thing to the one who can write the fixture. The rubric becomes whatever an engineer could think to assert on a Tuesday afternoon. The domain expert never touched it.

Flip the order. The domain expert writes the rubric first, in plain language, with examples: "the summary must cite an ICD-10 code when the source names a diagnosis, and the code must match." "Any output containing a treatment recommendation must include the human-in-the-loop disclaimer." "A request that falls under KYC review must be flagged, not answered." Then the platform enforces that rubric, in CI if you like, on every commit, blocking the merge when a criterion fails. The engineer still owns the pipeline. The domain expert owns the definition of correct. The build stays green only when the answer is right by the standard of the person who actually knows.

What the engineer-authored eval misses in practice

The gap is not theoretical. When we ran error analysis on our own insights feature, we expected hallucination to be the dominant failure. It barely registered. The real top failures were superficial insights, recommendations made confidently on too little data, and format violations that broke the downstream report. An engineer setting up eval-as-test would have written a hallucination check and a schema check and called it covered. Both would have passed. The feature would have shipped with the three failures that were actually costing quality, none of which an engineer had a reason to anticipate.

The fix was not a better assertion. It was putting the people who understood the domain in charge of reading the outputs and naming the failures. One sustainability team we worked with started at under 40% accuracy with an engineering-led setup. Once their domain experts, the people who had done these assessments by hand for years, owned the review and shaped the rubric, they reached over 90% in about three weeks, on a cheaper model. The technology did not change. The author of the rubric did.

Keep the CI job. Change who defines the pass.

None of this is an argument against running evals in your build pipeline. Automated, on every commit, blocking the merge on failure: that is exactly where evaluation should live once it exists. The argument is about the step before that, the one the CI framing skips: someone has to decide what the test is actually checking, and in a regulated domain that someone is not an engineer.

So the useful test for any "evals as tests" setup is one question: who wrote the criteria the build is checking against? If the answer is "the engineer who set up the pipeline," your CI is green on the engineer's definition of working, and silent on everything the engineer did not know to encode. If the answer is "the domain expert who owns the outcome, and the pipeline just runs their rubric on every change," now the green build means something a compliance officer can sign.

Evals belong in CI. The rubric belongs to the domain expert. Wire them in that order and the passing build finally tells you what you actually wanted to know: not that the code ran, but that the answer was right.