Skip to content

feat(bedrock): support responses API auth refresh #10123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jameshiester-oai wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(bedrock): support responses API auth refresh #10123

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions examples/amazon-bedrock/models/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ This directory contains several example configurations for different Bedrock mod

- [`promptfooconfig.claude.yaml`](promptfooconfig.claude.yaml) - Claude 4.6 Opus, Claude 4.1 Opus, Claude 4 Opus/Sonnet, Claude Haiku 4.5
- [`promptfooconfig.openai.yaml`](promptfooconfig.openai.yaml) - OpenAI GPT-OSS models (120B and 20B) with reasoning effort
- [`promptfooconfig.openai-responses.yaml`](promptfooconfig.openai-responses.yaml) - OpenAI GPT-OSS 120B through the Bedrock Responses API (requires `AWS_BEARER_TOKEN_BEDROCK`)
- [`promptfooconfig.openai-frontier.yaml`](promptfooconfig.openai-frontier.yaml) - OpenAI GPT-5.6 Sol, Terra, and Luna with reasoning, explicit prompt caching, and streaming
- [`promptfooconfig.grok.yaml`](promptfooconfig.grok.yaml) - xAI Grok 4.3 on the Bedrock Mantle endpoint (requires `AWS_BEARER_TOKEN_BEDROCK`)
- [`promptfooconfig.mantle.yaml`](promptfooconfig.mantle.yaml) - `bedrock:mantle:` Chat Completions endpoint for mantle-only models like GLM 4.6 and DeepSeek V3.1 (requires `AWS_BEARER_TOKEN_BEDROCK`)
Expand Down Expand Up @@ -244,12 +245,23 @@ The OpenAI example (`promptfooconfig.openai.yaml`) demonstrates OpenAI's GPT-OSS
- **OpenAI API Format**: Uses familiar OpenAI parameters like `max_completion_tokens`
- **Available in us-west-2**: Ensure you have model access in the correct region

For the OpenAI-compatible Responses API variant, use
`promptfooconfig.openai-responses.yaml`. It targets the shorter mantle model id
`openai.gpt-oss-120b` through `bedrock:responses:` and requires
`AWS_BEARER_TOKEN_BEDROCK`.
Comment on lines +248 to +251

Run the OpenAI example with:

```bash
promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.openai.yaml
```

Run the Responses API example with:

```bash
promptfoo eval -c examples/amazon-bedrock/models/promptfooconfig.openai-responses.yaml --no-cache
```

## OpenAI Frontier Models Example

The frontier example (`promptfooconfig.openai-frontier.yaml`) demonstrates OpenAI's GPT-5.x frontier models on Bedrock:
Expand Down
24 changes: 24 additions & 0 deletions examples/amazon-bedrock/models/promptfooconfig.openai-responses.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: 'OpenAI GPT OSS through the Amazon Bedrock Responses API'

# Export an Amazon Bedrock API key before running:
# export AWS_BEARER_TOKEN_BEDROCK="..."

Comment on lines +4 to +6
prompts:
- 'Answer concisely: {{question}}'

providers:
- id: bedrock:responses:openai.gpt-oss-120b
label: GPT OSS 120B (Responses API)
config:
region: us-east-1
reasoning_effort: medium
max_output_tokens: 1024
store: false

tests:
- vars:
question: What is the capital of France? Reply with only the city name.
assert:
- type: equals
value: Paris
Loading
Loading