fix(providers): preserve Responses schema strict false

[VectorPeak]

Preserve explicit response_format.json_schema.strict: false when building OpenAI and Azure Responses API request bodies.

What Problem This Solves

OpenAI Responses and Azure Responses both accepted response_format objects that include json_schema.strict, but their request builders always emitted text.format.strict: true for JSON Schema formats.

That meant an explicit user configuration like strict: false was indistinguishable from an omitted value by the time promptfoo sent the request. This matters because provider APIs treat strict mode as a real schema-enforcement switch, and strict mode only supports a subset of JSON Schema.

Change

The Responses request builders now preserve the configured strict value using the same fallback shape already used by the Azure Foundry Agent provider:

const strict = responseFormat.json_schema?.strict ?? responseFormat.strict ?? true;

The default remains true when no strict value is provided. The change only affects OpenAI Responses and Azure Responses JSON Schema request body construction.

Evidence

Before this change, both request builders hardcoded strict: true, so strict: false could not reach text.format.

Added focused regression coverage for both provider paths:

• OpenAI Responses preserves body.text.format.strict === false
• Azure Responses preserves body.text.format.strict === false

Possible call chain / impact

promptfoo provider config
  -> response_format: { type: json_schema, json_schema: { strict: false, ... } }
  -> maybeLoadResponseFormatFromExternalFile(...)
  -> OpenAiResponsesProvider.getOpenAiBody(...) / AzureResponsesProvider.getAzureResponsesBody(...)
  -> request text.format.strict

Sibling surface checked: Azure Foundry Agent already preserves strict: false; this PR aligns the Responses providers with that behavior without changing json_object, plain text, tools, or schema file loading.

Validation

• npm ci
• npx @biomejs/biome check --write src/providers/openai/responses.ts src/providers/azure/responses.ts test/providers/openai/responses/responseFormat.test.ts test/providers/azure/responses.test.ts
• npx vitest run test/providers/openai/responses/responseFormat.test.ts test/providers/azure/responses.test.ts — 2 files, 50 tests passed
• npx vitest run test/providers/openai/responses test/providers/azure/responses.test.ts — 15 files, 237 tests passed
• git diff --check
• npm run tsc

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.72%. Comparing base (5469ee5) to head (a7411a2).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10139      +/-   ##
==========================================
- Coverage   80.73%   80.72%   -0.01%     
==========================================
  Files         929      929              
  Lines       76701    76703       +2     
  Branches    25154    25156       +2     
==========================================
- Hits        61923    61919       -4     
- Misses      14778    14784       +6     

Flag	Coverage Δ
backend	82.55% <100.00%> (-0.01%)	⬇️
site	5.35% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[promptfoo-scanner]

👍 All Clear

Reviewed changes to Azure/OpenAI response formatting that now honor an explicitly provided strict: false in json_schema configs, along with corresponding tests. The updates only affect request construction and do not introduce new tools, execution sinks, or untrusted input paths. No LLM security vulnerabilities were identified in this PR.

_{Minimum severity threshold: 🟡 Medium | To re-scan after changes, comment @promptfoo-scanner
Learn more}

---

_{Was this helpful?  👍 Yes  |  👎 No} 

[mldangelo-oai]

Release audit on 6d7ebb4: ran the focused Responses/schema suites (50 tests) and reviewed request construction. Explicit strict false is preserved while unset/true behavior remains intact; no release-blocking findings.

[mldangelo]

Thanks for this — tightly scoped fix, and it's correct. Verified it's safe to land.

Heads-up: I pushed a commit to your branch

So you're not surprised by a commit you didn't write: I added two tests to test/providers/openai/responses/responseFormat.test.ts. Test-only, no production code touched. The reason was that mutation testing showed the OpenAI half of your change wasn't pinned by anything:

• Delete ?? true from src/providers/openai/responses.ts:942 → Test Files 34 passed (34) / Tests 1197 passed (1197). Survived.
• Change it to ?? false → same, 1197 passed. Survived.
• The identical ?? false mutation on the Azure line (src/providers/azure/responses.ts:187) → Tests 1 failed | 534 passed, failing at test/providers/azure/responses.test.ts:157:

  -   "strict": true,
  +   "strict": false,

  Killed.

Your Azure test pinned the default; nothing pinned it on OpenAI. Separately, deleting the middle rung ?? responseFormat.strict from both files left all 1732 tests green — that shape was unexercised everywhere. The pushed tests cover both rungs, and each one fails against its corresponding mutant.

What's good

The three-rung resolution json_schema.strict ?? strict ?? true is the right shape, and it matches what src/providers/azure/foundry-agent.ts:396 already does (from #8059) — so this aligns the Responses providers with an existing precedent instead of inventing a fourth convention. Backward compat genuinely holds: a config with no strict key anywhere still emits text.format.strict: true. And you caught both clones, which is the easy thing to miss here.

Remaining items

1. xai:responses still hardcodes strict: true — P2

src/providers/xai/responses.ts:124, inside buildTextFormat, is the same block reading the same config shape:

const schema = responseFormat.schema || responseFormat.json_schema?.schema;
const schemaName = responseFormat.json_schema?.name || responseFormat.name || 'response_schema';
return { format: { type: 'json_schema', name: schemaName, schema, strict: true } };

After this PR, strict: false is honored on openai:responses and azure:responses and silently ignored on xai:responses — same docs page, same config shape, divergent behavior. Was leaving xAI out deliberate (e.g. their API rejects non-strict json_schema)? If so, a note in site/docs/providers/xai.md would save someone a debugging session. If not, it's the same one-liner.

2. Docs — P3

site/docs/providers/openai.md:1529-1547 shows the flattened Responses response_format with no strict key at all, while the Chat Completions example directly above (line 1524) does show "strict": true. Nothing in site/docs states the Responses default or that it's now configurable.

3. Optional: one helper instead of four copies

buildTextFormat at src/providers/xai/responses.ts:107 is already the extracted form of the block you edited. Lifting it into src/providers/responses/ (all three files already import from there) drops ~50 duplicated lines and fixes xAI for free. Totally fine as a follow-up.

Nit if you're in there anyway: the two tests you added are near-verbatim copies of the tests immediately above them — it.each([{ strict: true }, { strict: false }]) would avoid a 7th copy of the mockApiResponse blob.

Nothing blocking. Thanks again.