Observability
Definition
Watching what your AI actually did in production — traces, inputs, outputs, latency, cost, errors. Necessary, and not the same as evaluation. Observability tells you what happened; it can't tell you whether the answer was any good.
Observability for AI is the practice of capturing what your feature actually did in production: the traces, inputs, outputs, latency, cost, and errors of every model call. It's foundational infrastructure — you need it — but it isn't evaluation. Observability shows volume and anomalies. It can't tell you whether an answer was actually useful, because nothing in a log knows the correct output.
Why it matters
Teams often buy observability and think they've covered quality. They haven't. Observability tells you what happened after users were already affected; evaluation tells you what would happen before you ship. You need both, but they're not substitutes.
Related terms