How to export your OpenAI API call history as JSONL and import it into a structured evaluation experiment
OpenAI's stored completions feature records Chat Completions calls made with store: true. That history contains production inputs and model outputs, with token counts, finish reasons, and request metadata attached. To evaluate those calls, export each completion together with its input messages.
This guide covers how to get the export, what fields it contains, and how they map to evaluation-ready call data.
Getting the export
The importer accepts one Chat Completions object per line, or pretty-printed objects separated by whitespace. OpenAI's list endpoint returns a paginated JSON envelope, not an import-ready JSONL file. It also omits the input messages. Fetch those from the messages endpoint and attach them as input_messages before importing.
This Python example uses the standard library and follows pagination for both completions and messages. Set OPENAI_API_KEY in your environment first.
import json
import os
from urllib.parse import urlencode
from urllib.request import Request, urlopen
BASE_URL = "https://api.openai.com/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"}
def records(path):
after = None
while True:
params = {"limit": 20, "order": "asc"}
if after:
params["after"] = after
request = Request(f"{BASE_URL}{path}?{urlencode(params)}", headers=HEADERS)
with urlopen(request, timeout=30) as response:
page = json.load(response)
yield from page["data"]
if not page["has_more"]:
return
after = page["last_id"]
with open("completions.jsonl", "w", encoding="utf-8") as output:
for completion in records("/chat/completions"):
messages = list(
records(f"/chat/completions/{completion['id']}/messages")
)
for message in messages:
if message.get("content") is None:
message["content"] = message.get("content_parts") or ""
completion["input_messages"] = messages
output.write(json.dumps(completion) + "\n")
Inspect the resulting prompts and responses before importing. This example preserves text messages; the importer does not turn image, audio, or file payloads into equivalent evaluation inputs. For a large history, filter the completions by model or metadata, or split the exported JSONL into batches of at most 5,000 records. If the script stops on an API error, the output file contains only the records written before that error.
Field mapping
Each JSONL record maps to an evaluation call as follows:
| JSONL field | Evaluation field | Notes |
|---|---|---|
request_id or id | External ID + Trace ID | Uses request_id when present, otherwise id; see Deduplication |
model | Model | Date suffix stripped (gpt-4o-mini-2024-07-18 → gpt-4o-mini) |
input_messages | Prompt (full messages) + system prompt + user input | Joined system/developer instructions and the last user message; added by the export script above |
choices[0].message.content | Response | First choice only |
usage.prompt_tokens | Input token count | |
usage.completion_tokens | Output token count | |
usage.total_tokens | Total token count | |
created | Started at / Ended at | Unix timestamp → datetime |
metadata | Tags | Key:value pairs flattened as key:value tags |
choices[0].finish_reason | Level | content_filter and error → ERROR; others → DEFAULT |
temperature, top_p, seed | Model parameters | A seed of -1 is omitted; presence and frequency penalties are also kept |
Cost is not included in the JSONL export. The importer estimates it from token counts and its OpenAI pricing table. Unknown model names can use fallback rates, so validate the estimate against your provider bill before using it for a cost comparison.
Model name normalisation
OpenAI includes the deployment date in model names (gpt-4o-mini-2024-07-18, gpt-4-turbo-2024-04-09). The importer strips the date suffix to produce a stable model identifier used for cost lookup and grouping. The original snapshot name is not retained in the imported call.
Content filtering and finish reasons
The choices[0].finish_reason field determines the call status:
finish_reason | Level | Meaning |
|---|---|---|
stop | DEFAULT | Normal completion |
length | DEFAULT | Max tokens reached |
content_filter | ERROR | Content policy triggered |
tool_calls | DEFAULT | Model invoked a function |
function_call | DEFAULT | Legacy function calling |
Calls with content_filter are imported as ERROR-level entries. They're valuable evaluation data — they show where your prompts are hitting policy boundaries.
Metadata promotion to tags
You can attach string metadata to stored Chat Completions, within the API's key and value limits:
client.chat.completions.create(
model="gpt-4o",
messages=[...],
store=True,
metadata={
"user_id": "usr_123",
"feature": "invoice-summary",
"version": "v2.1",
}
)
Each metadata key-value pair becomes a tag on the evaluation call: user_id:usr_123, feature:invoice-summary, version:v2.1. This lets you filter your evaluation experiment by feature, version, or user segment.
Handling malformed records
The parser reads JSON objects one at a time. A malformed object is counted as a parse error, then the parser looks for the next line starting with { and resumes there. Review total_traces, parse_errors, and duplicates_skipped in the import summary; a successful request does not mean every record was accepted.
Deduplication
When an import names an existing experiment under the same agent, the importer scans that experiment's trace_id:<id> tags and skips matching source IDs. It uses request_id when available, otherwise the completion id. Without an explicit existing experiment name, the import creates a new experiment and does not deduplicate against earlier imports.
Records missing both request_id and id are rejected and counted as parse errors.
Import limits
A single upload accepts up to 5,000 records. For larger exports, split the JSONL file and import in batches.
Combining with evaluations
After import, each stored completion becomes an evaluation call you can:
- •Run through LLM judges to score quality dimensions
- •Compare against outputs from a different model (re-run the same input through Claude or Gemini, then compare side by side)
- •Filter by tag to isolate a specific feature or user segment
- •Add expected outputs manually to create a golden dataset
With the original messages included, your stored completion history becomes an evaluation dataset built from real production traffic. Check a sample of imported inputs and outputs before running evaluations.
FAQ
Does this work with the Responses API as well?
No. This importer expects Chat Completions fields such as choices, created, and usage.prompt_tokens. Responses API objects have a different shape and cannot be imported through this format.
Can I import completions that weren't stored with store: true?
No. The stored Chat Completions list returns only calls created with store: true.
What if I used Azure OpenAI instead of OpenAI directly?
This guide's exporter calls OpenAI directly. For Azure data, check the exported schema before importing: the importer requires the Chat Completions fields listed above, including input_messages. A deployment name may not match the pricing table, so review any cost estimate separately. For a Foundry Responses bundle, use the Foundry import guide.
Does the importer reconstruct the full conversation history?
Each record becomes one evaluation call. When input_messages contains a conversation, the importer keeps the full sequence as conversation history, joins system/developer instructions for the system-prompt field, and uses the last user message as the test input. It does not join separate completion records into a conversation.
Are function/tool call responses included?
No. The importer reads only choices[0].message.content; it does not copy tool_calls or function_call objects into the response. A tool-only completion can therefore have an empty response. Use an import format that preserves the tool behavior you need to evaluate.
Enjoyed reading? Add us as a preferred source in Google to support us.
Add Lovelaice as a preferred source