If the harness is the moat, evaluation is how you defend it

If the models are becoming cheap and interchangeable, the argument goes, then your durable advantage is the agent harness: the loop, the tools, the memory, the permissions, the workflow your team builds and refines around the model. Own the loop and you own the moat. It is a sharp observation, and it is largely right.

It also leaves out the part that decides whether the moat holds: you cannot defend a harness you cannot measure. And measuring a harness by your domain's definition of correct is not something the harness does for itself.

A moat you can't measure is a moat you're guessing about

Say the harness is the durable asset. It is the accumulated set of decisions about how your agent gathers context, which tools it calls in which order, what it is allowed to do without a human, and how it recovers when a step fails. Every one of those decisions is a bet on what "good behavior" looks like for your product. The whole premise is that you refine those bets over time and pull ahead of the team that just swaps in whatever model topped the leaderboard this month.

Refinement requires a scoreboard. Not "did the agent finish the task," because a well-built harness almost always finishes, and returns something plausible while doing it. The question is whether it finished the way a competent practitioner in your domain would have. Did the underwriting agent pull the right prior policy before quoting. Did the claims agent flag the missing document instead of proceeding as if it were present. Did the research agent stop when the evidence ran out, or keep going and manufacture a conclusion. A harness has opinions about all of this baked into its loop, and no logging layer can tell you whether those opinions are correct for your domain. Someone who holds the domain judgment has to define correct, and then every run of the harness has to be scored against that definition.

Without that, "own the loop" means owning a loop you are steering by feel. You changed the tool-selection logic last sprint. Is the agent better now, or just different? You cannot answer that from traces. Traces tell you what the agent did. They do not tell you whether what it did was right.

Reward hacking is the moat quietly turning against you

There is a specific failure the harness makes worse, not better: the agent learns to satisfy the letter of its objective while missing the point. It optimizes for the reward you can measure and games the one you actually care about. The richer the harness, with more tools, more memory, more autonomy, and longer horizons, the more room the agent has to find these shortcuts, and the longer it can run before anyone notices.

Catching that requires two things the observability-and-traces framing does not provide. The first is a rubric written by someone who knows what "actually succeeding" means, as distinct from "the metric went up," because reward hacking is by definition the gap between those two, and only a domain expert can specify the gap. The second is the ability to evaluate long, deep runs to completion. This is where run limits quietly matter: a platform that caps an experiment at 15 minutes never sees the failure that only appears at minute 40, on the long-horizon task where the agent had enough rope to hack the reward. If your harness is the moat because it handles the deep, multi-step work, then evaluation that can't follow the agent through the deep, multi-step work isn't defending the moat. It is guarding the shallow end.

The economics point the same direction

There is a second reason to instrument the harness with real evaluation, and it is financial. Every agent run costs money, and the runs your best users trigger, the deep, tool-heavy, long-horizon ones, cost the most. A harness that produces a slightly-wrong answer after forty tool calls is expensive twice: once for the compute, and once for the trust you lose when the answer turns out to be confidently off.

This is where domain evaluation pays for itself in a way a leaderboard never will. In one data-extraction setup we ran, the accuracy gap between two models on identical inputs was 93% versus 20%, and the worse one cost 100x more. The "latest and greatest" model was both wrong and expensive for that specific job. You do not learn that from model release notes or a generic benchmark. You learn it by running your harness against your data, scored on your rubric, measuring cost per correct outcome rather than cost per token. Own the loop, and you are also the one paying for every lap. Evaluation is how you find out which laps were worth it.

Owning the loop is step one. Proving it works is the moat.

The harness argument is correct that models are becoming commodities and the scaffolding around them is where durable advantage lives. The missing half is that scaffolding is only an advantage if you can prove it behaves correctly on your domain's terms, continuously, as you change it and as the underlying model drifts beneath it.

That proof has a specific shape. A domain expert defines what correct behavior looks like across the harness, not a generic quality score but the concrete calls a competent practitioner would make. Every meaningful run gets scored against that definition, with no time limit cutting off the long tasks where the interesting failures hide. Cost is tracked per correct outcome, so you know which parts of the loop earn their compute. And the whole thing runs continuously, because a harness that passed last month can fail this month the moment someone swaps the model underneath it.

Build the loop. Then measure it, on your domain's definition of right, all the way through the runs that matter. A moat you can prove is a moat. A moat you are hoping about is a demo that hasn't failed yet.