How the Foundry Responses API export format maps to structured evaluation calls, including tool tracking and refusal handling
Azure AI Foundry's Responses API returns structured output, tool calls, usage metrics, and timing when those fields are available. To import those responses into Lovelaice, pair the saved response JSON with a manifest recording the test inputs. That preserves what was asked alongside what the agent produced; you still need reviewed reference answers or acceptance criteria to evaluate correctness.
This guide covers the bundle format, how the importer validates it, and what gets extracted from complex multi-step responses.
The Foundry export format
Lovelaice accepts a bundle assembled from your saved Foundry responses and a test manifest. This is the importer's bundle structure, not a built-in Foundry export format:
{
"manifest": {
"agentName": "Invoice Processor",
"traces": [
{
"file": "trace-001.json",
"prompt": "Process this invoice: INV-2024-001...",
"responseId": "resp_abc123"
}
]
},
"traces": {
"trace-001.json": { }
}
}
The manifest records what your test harness sent to the agent. The traces object holds the raw API responses. The importer uses the manifest to supply test_input (what was asked) and checks supplied response IDs when matching records.
Manifest validation
The importer validates the manifest and matches its entries to response records:
- •For each manifest entry, it looks for a matching trace file by filename or by
responseId - •If neither the filename nor the
responseIdresolves to a response, that entry is reported as a parse error (the rest are still processed) - •If a filename match has both a manifest
responseIdand a responseid, they must agree or that entry is rejected
Include both IDs to catch swapped or misnamed files. An omitted ID cannot provide that check. The bundle also needs a non-empty manifest.traces array and at least one response record.
Response extraction
Foundry responses contain nested output blocks across multiple processing phases. The importer extracts the final answer from these blocks:
- •It looks for message blocks with
phase == "final_answer" - •If those blocks contain text or refusal content, that content is assembled in order as the response
- •Otherwise, it uses text and refusals from the other message blocks
Both text content (output_text) and refusal content are extracted. Refusals — cases where the model declined to answer — are evaluable outputs. Knowing when and why your agent refuses is as important as knowing when it answers correctly.
Tool tracking
Foundry agents often call tools (web search, code interpreter, function calls) as part of producing a response. The importer captures three dimensions of tool usage:
Declared tools: Tool types listed in the response record's tools array become available_tool:<type> tags. For a function tool, the tag is available_tool:function, not the individual function name.
Invoked tools: Output block types ending in _call become tool:<type> tags after removing that suffix. For example, web_search_call produces tool:web_search, while function calls produce tool:function.
Resource IDs: Declared file-search vector stores become vector_store_id:<id> tags. Files seeded in a code-interpreter container become code_interpreter_file_id:<id> tags. These describe resources available to the agent, not proof that it used them.
This distinction is useful during evaluation: an agent that had access to 5 tools but only called 2 may be over-specified. An agent that always calls every available tool may not be routing queries well.
Reasoning model support
For responses that expose reasoning effort, reasoning.effort is saved as the reasoning_effort model parameter. This lets you compare runs at different effort levels.
Field mapping
| Foundry field | Evaluation field |
|---|---|
id or manifest requestId | External ID |
manifest requestId, otherwise the external ID | Trace ID and deduplication key |
model (date suffix stripped) | Model |
manifest prompt | Test input |
instructions | System prompt |
| Assembled output blocks | Response |
usage.input_tokens | Input token count |
usage.output_tokens | Output token count |
created_at | Started at |
completed_at | Ended at |
(completed_at - created_at) × 1000 | Latency in ms |
status + error object | Level (DEFAULT / ERROR) |
reasoning.effort | reasoning_effort model parameter |
manifest agentName | agent_name tag |
tools[*].type | available_tool:<type> tags |
Output types ending in _call | tool:<type> tags |
| Declared vector store / seeded file IDs | vector_store_id:<id> / code_interpreter_file_id:<id> tags |
Cost is estimated from token counts using the importer's OpenAI pricing table, with fallback rates for unknown model names. It does not read your Azure billing configuration, discounts, or deployment-specific rates. Compare the estimate with your Azure bill before using it for budgeting.
A typical evaluation workflow with Foundry
- •Build a test manifest with representative inputs covering your key scenarios
- •Run those inputs through your Foundry agent and save each response JSON
- •Assemble the manifest and response files into the bundle above
- •Import to Lovelaice
- •Run LLM judges or human review on the imported calls
- •Adjust your agent (system prompt, tools, model) and re-run
- •Compare the new run against the previous one
Fill in prompt for every manifest entry. The response does not supply this field on its own; if the manifest omits it, the imported test input is blank.
FAQ
What if my Foundry runs don't use a manifest?
Create a manifest from your test-harness records. The importer rejects a bundle without a manifest or with an empty manifest.traces array. An entry without prompt can import with a blank test input, but it cannot support rerunning the original task from that input.
Can I import individual response JSON files, not a bundle?
The current import accepts the bundle format. Wrapping individual files in the bundle structure is straightforward: create a manifest with one entry per file and assemble them into the traces object.
Are multi-step agent runs captured as a single evaluation call or multiple?
Each response becomes one evaluation call. Tool tags record distinct call types in first-seen order, not the complete call sequence or arguments. The saved response contains extracted message text and refusals, not the raw output array. Retain the source JSON if you need to inspect individual tool calls.
Does refusal content appear in the response field?
Yes. Refusal blocks are extracted alongside text within the selected message blocks. If final-answer messages contain text or refusal content, that content takes precedence; otherwise the importer uses the other message blocks.
What about image outputs or file outputs from code interpreter?
Text and refusal blocks are extracted. Binary output content is not stored in the evaluation call. The file-ID tags described above cover seeded code-interpreter inputs, not generated output files.
Enjoyed reading? Add us as a preferred source in Google to support us.
Add Lovelaice as a preferred source