How to export your OpenAI API call history as JSONL and import it into a structured evaluation experiment

OpenAI's stored completions feature records Chat Completions calls made with store: true. That history contains production inputs and model outputs, with token counts, finish reasons, and request metadata attached. To evaluate those calls, export each completion together with its input messages.

This guide covers how to get the export, what fields it contains, and how they map to evaluation-ready call data.

Getting the export

The importer accepts one Chat Completions object per line, or pretty-printed objects separated by whitespace. OpenAI's list endpoint returns a paginated JSON envelope, not an import-ready JSONL file. It also omits the input messages. Fetch those from the messages endpoint and attach them as input_messages before importing.

This Python example uses the standard library and follows pagination for both completions and messages. Set OPENAI_API_KEY in your environment first.

import json
import os
from urllib.parse import urlencode
from urllib.request import Request, urlopen

BASE_URL = "https://api.openai.com/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"}


def records(path):
    after = None
    while True:
        params = {"limit": 20, "order": "asc"}
        if after:
            params["after"] = after
        request = Request(f"{BASE_URL}{path}?{urlencode(params)}", headers=HEADERS)
        with urlopen(request, timeout=30) as response:
            page = json.load(response)
        yield from page["data"]
        if not page["has_more"]:
            return
        after = page["last_id"]


with open("completions.jsonl", "w", encoding="utf-8") as output:
    for completion in records("/chat/completions"):
        messages = list(
            records(f"/chat/completions/{completion['id']}/messages")
        )
        for message in messages:
            if message.get("content") is None:
                message["content"] = message.get("content_parts") or ""
        completion["input_messages"] = messages
        output.write(json.dumps(completion) + "\n")

Inspect the resulting prompts and responses before importing. This example preserves text messages; the importer does not turn image, audio, or file payloads into equivalent evaluation inputs. For a large history, filter the completions by model or metadata, or split the exported JSONL into batches of at most 5,000 records. If the script stops on an API error, the output file contains only the records written before that error.

Field mapping

Each JSONL record maps to an evaluation call as follows:

JSONL fieldEvaluation fieldNotes
request_id or idExternal ID + Trace IDUses request_id when present, otherwise id; see Deduplication
modelModelDate suffix stripped (gpt-4o-mini-2024-07-18 → gpt-4o-mini)
input_messagesPrompt (full messages) + system prompt + user inputJoined system/developer instructions and the last user message; added by the export script above
choices[0].message.contentResponseFirst choice only
usage.prompt_tokensInput token count
usage.completion_tokensOutput token count
usage.total_tokensTotal token count
createdStarted at / Ended atUnix timestamp → datetime
metadataTagsKey:value pairs flattened as key:value tags
choices[0].finish_reasonLevelcontent_filter and error → ERROR; others → DEFAULT
temperature, top_p, seedModel parametersA seed of -1 is omitted; presence and frequency penalties are also kept

Cost is not included in the JSONL export. The importer estimates it from token counts and its OpenAI pricing table. Unknown model names can use fallback rates, so validate the estimate against your provider bill before using it for a cost comparison.

Model name normalisation

OpenAI includes the deployment date in model names (gpt-4o-mini-2024-07-18, gpt-4-turbo-2024-04-09). The importer strips the date suffix to produce a stable model identifier used for cost lookup and grouping. The original snapshot name is not retained in the imported call.

Content filtering and finish reasons

The choices[0].finish_reason field determines the call status:

finish_reasonLevelMeaning
stopDEFAULTNormal completion
lengthDEFAULTMax tokens reached
content_filterERRORContent policy triggered
tool_callsDEFAULTModel invoked a function
function_callDEFAULTLegacy function calling

Calls with content_filter are imported as ERROR-level entries. They're valuable evaluation data — they show where your prompts are hitting policy boundaries.

Metadata promotion to tags

You can attach string metadata to stored Chat Completions, within the API's key and value limits:

client.chat.completions.create(
    model="gpt-4o",
    messages=[...],
    store=True,
    metadata={
        "user_id": "usr_123",
        "feature": "invoice-summary",
        "version": "v2.1",
    }
)

Each metadata key-value pair becomes a tag on the evaluation call: user_id:usr_123, feature:invoice-summary, version:v2.1. This lets you filter your evaluation experiment by feature, version, or user segment.

Handling malformed records

The parser reads JSON objects one at a time. A malformed object is counted as a parse error, then the parser looks for the next line starting with { and resumes there. Review total_traces, parse_errors, and duplicates_skipped in the import summary; a successful request does not mean every record was accepted.

Deduplication

When an import names an existing experiment under the same agent, the importer scans that experiment's trace_id:<id> tags and skips matching source IDs. It uses request_id when available, otherwise the completion id. Without an explicit existing experiment name, the import creates a new experiment and does not deduplicate against earlier imports.

Records missing both request_id and id are rejected and counted as parse errors.

Import limits

A single upload accepts up to 5,000 records. For larger exports, split the JSONL file and import in batches.

Combining with evaluations

After import, each stored completion becomes an evaluation call you can:

  • •Run through LLM judges to score quality dimensions
  • •Compare against outputs from a different model (re-run the same input through Claude or Gemini, then compare side by side)
  • •Filter by tag to isolate a specific feature or user segment
  • •Add expected outputs manually to create a golden dataset

With the original messages included, your stored completion history becomes an evaluation dataset built from real production traffic. Check a sample of imported inputs and outputs before running evaluations.

FAQ

Does this work with the Responses API as well?

No. This importer expects Chat Completions fields such as choices, created, and usage.prompt_tokens. Responses API objects have a different shape and cannot be imported through this format.

Can I import completions that weren't stored with store: true?

No. The stored Chat Completions list returns only calls created with store: true.

What if I used Azure OpenAI instead of OpenAI directly?

This guide's exporter calls OpenAI directly. For Azure data, check the exported schema before importing: the importer requires the Chat Completions fields listed above, including input_messages. A deployment name may not match the pricing table, so review any cost estimate separately. For a Foundry Responses bundle, use the Foundry import guide.

Does the importer reconstruct the full conversation history?

Each record becomes one evaluation call. When input_messages contains a conversation, the importer keeps the full sequence as conversation history, joins system/developer instructions for the system-prompt field, and uses the last user message as the test input. It does not join separate completion records into a conversation.

Are function/tool call responses included?

No. The importer reads only choices[0].message.content; it does not copy tool_calls or function_call objects into the response. A tool-only completion can therefore have an empty response. Use an import format that preserves the tool behavior you need to evaluate.