A practical guide to turning recorded observability data into structured evaluation experiments

Your recorded LLM calls can become an evaluation dataset. If your logs already capture prompts, completions, token counts and latency, importing is a data-routing step rather than a new logging project. Check what was captured before choosing an import path.

This guide covers four external import paths, how to choose between them, and what to check after import. Lovelaice also accepts its own JSON experiment exports.

The evaluation data problem

Most teams accumulate LLM call data in one of two places: an observability tool (Langfuse, Datadog, Honeycomb) or the provider's own storage (OpenAI stored completions, Azure Foundry logs). Getting that data into a systematic evaluation experiment typically requires a custom ETL script per data source.

The alternative is a normalised import that understands each format, extracts the fields that matter for evaluation, and produces consistent evaluation records regardless of source.

The four import formats

OpenTelemetry (OTLP HTTP)

Best for: Apps already instrumented with OTel-compatible libraries (Traceloop, OpenLLMetry, Langfuse SDK with OTel configured)

OTLP is the streaming path covered here. Send supported spans to the ingestion endpoint. With an on-demand key, traces wait for you to promote them into an experiment. With a direct key, LLM spans become call records immediately, without an experiment assignment.