A practical guide to turning recorded observability data into structured evaluation experiments
Your recorded LLM calls can become an evaluation dataset. If your logs already capture prompts, completions, token counts and latency, importing is a data-routing step rather than a new logging project. Check what was captured before choosing an import path.
This guide covers four external import paths, how to choose between them, and what to check after import. Lovelaice also accepts its own JSON experiment exports.
The evaluation data problem
Most teams accumulate LLM call data in one of two places: an observability tool (Langfuse, Datadog, Honeycomb) or the provider's own storage (OpenAI stored completions, Azure Foundry logs). Getting that data into a systematic evaluation experiment typically requires a custom ETL script per data source.
The alternative is a normalised import that understands each format, extracts the fields that matter for evaluation, and produces consistent evaluation records regardless of source.
The four import formats
OpenTelemetry (OTLP HTTP)
Best for: Apps already instrumented with OTel-compatible libraries (Traceloop, OpenLLMetry, Langfuse SDK with OTel configured)
OTLP is the streaming path covered here. Send supported spans to the ingestion endpoint. With an on-demand key, traces wait for you to promote them into an experiment. With a direct key, LLM spans become call records immediately, without an experiment assignment.
The importer recognises OpenLLMetry-style prompt/completion attributes and events named gen_ai.content.prompt and gen_ai.content.completion. JSON and Protobuf content types are accepted. Check a sample from your instrumentation version; not every gen_ai convention shape maps to the same fields.
Langfuse CSV
Best for: Teams using Langfuse for observability who want to import a batch of historical traces or annotated generations with scores
Export the generations table, then check that the CSV has the required id, traceId, and input columns. Optional scores with source: HUMAN_ANNOTATION are saved as feedback notes. Automated scores do not become imported evaluation results.
When input is a messages array, the importer joins system/developer instructions and takes the last user message as the test input. Plain-string inputs are also accepted. See the Langfuse guide for score joins, schema compatibility, and trace-level deduplication.
OpenAI Stored Completions (JSONL)
Best for: Teams using Chat Completions with store: true who want to evaluate recorded calls
Export completion records as JSONL with their original messages attached as input_messages. The API list response alone is insufficient: it is a paginated envelope without those messages. The OpenAI guide includes an exporter that fetches both. Responses API objects use a different schema and are not supported by this JSONL importer. Primitive metadata values become tags.
Model date suffixes are stripped automatically (gpt-4o-mini-2024-07-18 → gpt-4o-mini) for consistent grouping and cost lookup. Content-filtered calls are imported as ERROR-level entries — they're useful evaluation data, not noise.
Azure AI Foundry (manifest + response bundle)
Best for: Teams running structured test suites through the Foundry Responses API, with known test inputs and agent responses
Assemble a manifest and saved response JSON into Lovelaice's bundle format. This is not a native Foundry export. The manifest supplies the test input; response text and refusals are extracted from selected message blocks. Tool types and declared resource IDs become tags, while reasoning effort becomes a model parameter. Raw tool arguments and binary outputs are not copied into the saved response.
Choosing a format
Situation
Format
Already using compatible OTel instrumentation
OTLP HTTP, then promotion for experiment evaluation
Using Langfuse with scored generations
Langfuse CSV (batch)
Using stored Chat Completions
OpenAI JSONL with input messages (batch)
Running Foundry test suites
Foundry bundle (batch)
Migrating from a previous evaluation export
Lovelaice JSON export
If you have compatible instrumentation, add Lovelaice as an exporter destination. Otherwise, set up instrumentation and verify content capture first. For historical data, choose the batch format that matches your source schema.
The normalised call schema
Batch imports produce normalised evaluation calls. OTLP uses a separate projection with some different field shapes. Not every source supplies every field:
Field
Description
External ID
Source identifier; deduplication uses the stored trace-ID tag for batch imports
Trace ID
Groups related calls
System prompt
Extracted from messages by role
Test input
User message, the per-call variable input
Full prompt
When available, richer message sequences are retained as conversation history
Response
Model's reply
Model
Date suffixes stripped for OpenAI JSONL and Foundry bundles
Model family
Provider (openai, anthropic, google, etc.)
Model parameters
Source-dependent parameters, such as temperature or reasoning effort
Input tokens / Output tokens / Total tokens
Usage
Cost (USD)
Source value or estimate; missing cost can be saved as zero
Latency (ms)
Level
DEFAULT or ERROR
Status message
Source error detail; retained fields vary by import path
Batch imports check an existing experiment only when you explicitly supply its name under the same agent. They compare incoming trace IDs with that experiment's trace_id:<source_id> tags. The source ID varies by format: Langfuse uses traceId, OpenAI uses request_id or id, and Foundry uses manifest requestId or the response ID. A new experiment does not deduplicate against earlier experiments.
Langfuse's check is at trace level, so later generations from an already imported trace may be skipped. OpenAI and Foundry records without their required source IDs are rejected. OTLP does not use this batch deduplication check; resending or promoting the same spans can create duplicate calls.
Cost calculation
Langfuse CSV uses calculatedTotalCost; missing costs are not recalculated. OpenAI JSONL and Foundry imports estimate cost using the OpenAI pricing table, which can use fallback rates for unknown names. OTLP uses gen_ai.usage.cost when present and otherwise attempts a price lookup that can also fall back. Missing costs may be saved as zero. Review source costs and model names before treating imported totals as billing data.
Import limits
Batch imports accept up to 5,000 parsed calls per upload. For larger datasets, split the source data into batches and specify the target experiment when appending. Review the format-specific deduplication rules before using overlapping ranges.
OTLP accepts batched span requests through a separate endpoint. It does not use the batch importer's 5,000-call check; request-size limits and exporter retries still apply.
From import to evaluation
Batch-imported calls and promoted OTLP traces land in an evaluation experiment. Direct OTLP ingestion records calls without an experiment assignment. Within an experiment, you can:
•Run automated LLM judges on specific quality dimensions
•Compare the same inputs run through a different model
•Filter by tag to focus on a specific feature or failure mode
•Add expected outputs to create a golden dataset for regression testing
The import is the first step. The goal is a closed loop: observe in production, evaluate systematically, adjust the prompt or model, verify the adjustment didn't break existing cases, and ship.
FAQ
Can I combine data from multiple sources in one experiment?
Yes. Promote OTLP traces into an experiment, then name that experiment and its agent when adding a batch export. Promoted OTLP calls do not carry the batch importer's trace_id:<id> tag, so a call present in both sources can appear twice even if its trace IDs match.
Does importing preserve the original trace structure?
OTLP retains spans in the trace record and projects eligible LLM spans into separate call records. Non-LLM spans remain trace context. Batch formats work differently: for example, each Foundry response becomes one call with extracted text and tool-type tags, rather than the complete raw trace structure.
Can I export my evaluation data back out?
Yes. Lovelaice exports experiments in its own JSON format, which is also an importable format. This lets you migrate data between evaluation environments or archive experiment data externally.
What about rate limits or throttling during import?
Batch imports are processed synchronously within the request. Duration depends on file size and persistence work; the 5,000-call limit is not a processing-time guarantee. Review the returned summary before starting the next batch. OTLP requests are batched and retried according to your exporter configuration; retries can create duplicate call records.
Is there a way to import from other observability tools?
An OTLP exporter or collector can send supported spans to the streaming endpoint. A CSV or JSON file is not automatically OTLP: use a matching batch format, convert the data explicitly, or instrument future calls. Check the source tool's export capabilities before choosing a path.
What happens to tool calls and function calls in the imported data?
Tool content is source-dependent. OTLP recognises some multimodal tool blocks and flattens them to markers such as [tool_use: function_name] and [tool_result]. Foundry stores tool call types as tags, not complete arguments or results. OpenAI JSONL copies only choices[0].message.content, so tool-only completions can have an empty response.
Enjoyed reading? Add us as a preferred source in Google to support us.
Define what good looks like for your AI feature, measure it on real data, and improve it with your domain experts in the loop, no code required. Teams have gone from 52% to 88% accuracy in under three weeks.