feat(llm): add LLM profiles

[enyst]

HUMAN:
LLM Profiles behavior

• integrated with LLMRegistry
  • at any point, the registry knows all profiles and which is used for which usage_id
  • profile_id is set by the user, and corresponds to llm_profiles/profile_id.json in persistence_dir (if set)
  • defines a small API for profiles:
    • load_profile()
    • save_profile()
    • validate_profile()
    • list_profiles()
    • get_profile_path()
• uses LLM_PROFILES_DIR

---

Summary

• Integrate LLM profile persistence into LLMRegistry, exposing list/load/save/register/validate helpers with configurable profile directories
• Fix docs example checker to detect nested example paths (avoids false CI failures)

Testing

uv run pytest tests/sdk/llm/test_llm_registry_profiles.py
uv run pytest tests/sdk/conversation/local/test_state_serialization.py

Related

• Docs PR: docs(sdk): add LLM profiles guide docs#280
• Replaces feat(llm): add LLM profiles #795 (closed due to long conversation)

---

Agent Server images for this PR

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant	Architectures	Base Image	Docs / Tags
java	amd64, arm64	eclipse-temurin:17-jdk	Link
python	amd64, arm64	nikolaik/python-nodejs:python3.13-nodejs22-slim	Link
golang	amd64, arm64	golang:1.21-bookworm	Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:d1a4998-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-d1a4998-python \
  ghcr.io/openhands/agent-server:d1a4998-python

All tags pushed for this build

ghcr.io/openhands/agent-server:d1a4998-golang-amd64
ghcr.io/openhands/agent-server:d1a4998-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:d1a4998-golang-arm64
ghcr.io/openhands/agent-server:d1a4998-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:d1a4998-java-amd64
ghcr.io/openhands/agent-server:d1a4998-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:d1a4998-java-arm64
ghcr.io/openhands/agent-server:d1a4998-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:d1a4998-python-amd64
ghcr.io/openhands/agent-server:d1a4998-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-amd64
ghcr.io/openhands/agent-server:d1a4998-python-arm64
ghcr.io/openhands/agent-server:d1a4998-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-arm64
ghcr.io/openhands/agent-server:d1a4998-golang
ghcr.io/openhands/agent-server:d1a4998-java
ghcr.io/openhands/agent-server:d1a4998-python

About Multi-Architecture Support

• Each variant tag (e.g., d1a4998-python) is a multi-arch manifest supporting both amd64 and arm64
• Docker automatically pulls the correct architecture for your platform
• Individual architecture tags (e.g., d1a4998-python-amd64) are also available if needed

[enyst]

@OpenHands Why are we using profile_id, didn't we have a kind profile_ref? Please investigate the entire diff, and understand the code patterns and fix.

[openhands-ai]

I'm on it! enyst can track my progress at all-hands.dev

[openhands-ai]

Final update:

Summary (new since last update):

• No additional changes since the previous summary; branch is up to date and already pushed.

Checklist:

• Confirmed the request is fully addressed.
• Verified there are no extra or unrelated changes (diff vs. remote is empty).
• Changes are concise and already pushed to the PR branch.

If you want any further adjustments or additional tests run, just let me know.

View full conversation

[openhands-ai]

Looks like there are a few issues preventing this PR from being merged!

• GitHub Actions are failing:
  • [Optional] Docs example
  • Agent Server

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #1843 at branch `agent-sdk-18-profile-manager`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}

[all-hands-bot]

[Automatic Post]: It has been a while since there was any activity on this PR. @enyst, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

[all-hands-bot]

[Automatic Post]: It has been a while since there was any activity on this PR. @enyst, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

[enyst]

This PR has been the oldest experiment I think, with LLM profiles. Meanwhile, I've integrated profiles in a fun typescript version of the SDK:
LLM profiles spec

That is in a sense a full-featured version (switch on the fly, define on the fly, use multiple LLMs for tiny user-friendly functionality during the run). But also, a somewhat simplified design (e.g. always have a default profile, usage_ids are hardcoded, and some unused).

Maybe we should simplify here too, though maybe not so much, we'll see as we move forward. @VascoSch92 has added the first version of profiles, and we'll look at the next steps. Maybe this comment is something we'll get back to. We'll see!

[enyst]

@OpenHands Look at the llm profiles on main branch and understand the feature as it is today. (you can look for 'profile')

Look also for the PR adding switch llm. Read the PRs.

Make a plan of adding the behavior in those two PRs to main. Take your time to investigate and split into chunks that work all the time. Post your plan as a comment to this PR. Note that it will be rendered as markdown.

[openhands-ai]

I'm on it! enyst can track my progress at all-hands.dev

[enyst]

Investigation notes (current main)

What “LLM profiles” means on main today

On main, “profiles” exist as a standalone on-disk store:

• openhands.sdk.llm.llm_profile_store.LLMProfileStore
  • default dir: ~/.openhands/profiles
  • concurrency: FileLock + atomic write via temp file + Path.replace()
  • API: list(), save(name, llm, include_secrets=False), load(name), delete(name)
  • stored payload: full LLM JSON (secrets optionally omitted)
• The SDK does not currently:
  • store a profile_id on the LLM model
  • persist “profile references” in ConversationState (base_state.json)
  • integrate profiles into LLMRegistry

So: profiles are usable as a utility (and by fallback), but they’re not part of conversation persistence or runtime switching.

---

PRs investigated

PR #1843 (this PR): “feat(llm): add LLM profiles”

This PR’s intended behavior (from code + tests) is:

• Add LLM.profile_id and a profile reference persistence format (e.g. {kind: profile_ref, profile_id: ...})
• Add LLMRegistry.{list/load/save/validate}_profile* helpers and a configurable profile dir (via LLM_PROFILES_DIR)
• Update conversation persistence so base_state.json can store a profile ref rather than an inline LLM payload

“Switch LLM” PR(s)

There is a stacked “switch LLM” line of PRs:

• Switch LLM profiles #1049 “Switch LLM profiles” (base: agent-sdk-18-profile-manager)

  • SDK-local API: LocalConversation.switch_llm(profile_id)
  • enforces safety: only allow when conversation is idle
  • (in that PR) gating based on an inline/non-inline persistence mode

• [agent-sdk-j2s] agent-server runtime LLM switching #1544 “[agent-sdk-j2s] agent-server runtime LLM switching” (base: Switch LLM profiles #1049)

  • extends to agent-server REST:
    • POST /api/conversations/{id}/llm (inline payload or profile_id)
    • POST /api/conversations/{id}/llm/switch
  • changes persistence/restore semantics so a switched LLM can persist across restart

---

Proposed landing plan (split into always-green chunks)

The key to making this land safely is to port the behavior onto current main without regressing main’s existing LLMProfileStore, FallbackStrategy, LLMRegistry metrics-independence behavior, or the current conversation persistence lifecycle.

I propose landing this in a small stack of PRs (each rebased onto latest main, each CI-green):

PR 1 — “LLM: add profile_id + profile-ref encoding/decoding (no persistence wiring yet)”

Goal: introduce the model-level primitives with minimal surface area.

• Add LLM.profile_id: str | None (default None)
• Add LLM.to_profile_ref() and a pydantic serializer gated by context={"persist_profile_ref": True}
• Add a validator that can expand {kind: "profile_ref", profile_id: "..."} when context contains an llm_registry
• Backward compatibility:
  • old inline LLM payloads continue to validate unchanged
  • existing LLMProfileStore files (full LLM JSON) continue to load unchanged

CI gates: unit tests only (new tests for serialization + “expand ref requires context”).

---

PR 2 — “LLMRegistry: profile helpers that wrap existing LLMProfileStore (no new store class)”

Goal: avoid two competing profile stores.

• Keep openhands.sdk.llm.llm_profile_store.LLMProfileStore as the canonical on-disk implementation (locking, atomic write, default dir, etc.)
• Add thin convenience methods on LLMRegistry:
  • list_profiles(), load_profile(), save_profile(), get_profile_path()
  • directory selection rules:
    • explicit profile_dir arg wins
    • else LLM_PROFILES_DIR
    • else default to the current LLMProfileStore default (~/.openhands/profiles) to preserve behavior
• Preserve current LLMRegistry invariants on main:
  • usage_to_llm read-only mapping
  • metrics independence logic (don’t regress it while adding profile helpers)

CI gates: add/port tests similar to tests/sdk/llm/test_llm_registry_profiles.py but adapted to the existing store + directory rules.

---

PR 3 — “Conversation persistence: optionally persist profile refs + expand on restore”

Goal: teach ConversationState to round-trip profile refs safely.

• When saving base_state.json:
  • if agent.llm.profile_id is set, dump it as a profile ref (no inline secrets)
  • else keep existing inline behavior (still protected by cipher/redaction as today)
• When restoring:
  • pass llm_registry into model_validate(..., context=...) so {kind: profile_ref,...} can expand
  • make the registry available from LocalConversation (and agent-server) before state restore

Key design constraint to make behavior stable:

• Restoring should be deterministic even if the caller supplies a runtime Agent.
  • Option A (minimal): keep “runtime Agent is source of truth” and only use persisted agent for tool verification; switching will require the caller to re-supply the desired profile on restart.
  • Option B (what [agent-sdk-j2s] agent-server runtime LLM switching #1544 aims for): persisted LLM selection wins by default so a runtime switch persists across restart; runtime agent may override only when explicitly requested.

I recommend Option B if the goal is “switch LLM and it sticks”, especially for agent-server.

CI gates: state serialization tests + at least one end-to-end persistence lifecycle test.

---

PR 4 — “SDK runtime switching: LocalConversation.switch_llm() (+ optionally set_llm())

Goal: expose a safe API to mutate the active LLM at runtime.

• Add AgentBase._clone_with_llm(llm) helper so swapping doesn’t drop tool wiring
• Add ConversationState.switch_agent_llm(profile_id, registry=...)
  • enforce: conversation must not be RUNNING (idle/paused only)
  • validate profile exists / is loadable
  • persist base_state immediately after switch
• Add LocalConversation.switch_llm(profile_id) wrapper
• (If we want parity with [agent-sdk-j2s] agent-server runtime LLM switching #1544 remote flows) also add LocalConversation.set_llm(llm) for “client sends inline config”

CI gates: unit tests for:

• requires idle
• missing profile rejected
• persistence reflects switched profile ref

---

PR 5 — “agent-server switching API (from #1544)”

Goal: land the REST endpoints + persistence-across-restart behavior.

• Add server routes + service methods:
  • POST /api/conversations/{id}/llm/switch (profile_id)
  • POST /api/conversations/{id}/llm (either profile_id or inline llm payload)
• Ensure restart restores the switched LLM (requires PR 3 Option B or equivalent)
• Integration tests (wsproto) to validate:
  • switch endpoint updates active LLM
  • state persists across restart

CI gates: agent-server integration tests + docs/examples checks.

---

Why this chunking should stay green

• PR 1–2 are purely additive to model + registry, and should not touch conversation lifecycle.
• PR 3 introduces persistence wiring but can be done in a fully backward-compatible way (inline payloads still supported).
• PR 4–5 depend on PR 3 and are “feature surface” PRs that can be reviewed/rolled back independently.

---

Open questions / decisions to lock early

1. Profile directory + env var: should LLM_PROFILES_DIR default to ~/.openhands/profiles (current) or a new dir? I recommend preserving the current default for compatibility.
2. Profile ref shape: keep {kind: "profile_ref", profile_id: ...} (explicit) vs a minimal {profile_id: ...}. Explicit kind is safer for schema evolution.
3. Restore precedence: do we want “persisted LLM wins” (needed for agent-server “switch sticks”), or keep today’s “runtime agent wins” behavior?

If we align on these three, the implementation becomes mostly mechanical.

[openhands-ai]

I investigated the current “LLM profiles” feature on main and the two “switch LLM” PRs, then produced (and posted) a markdown plan for landing their combined behavior onto main in reliable, CI-green chunks.

What I did:

• Reviewed origin/main for existing profile support:
  • Found that main currently has openhands.sdk.llm.llm_profile_store.LLMProfileStore (default ~/.openhands/profiles, file locking + atomic writes) and FallbackStrategy using it.
  • Confirmed main does not have LLM.profile_id, conversation persistence of profile references, nor profile integration in LLMRegistry.
• Identified and read the relevant “switch LLM” PRs:
  • PR Switch LLM profiles #1049 “Switch LLM profiles” (local SDK switching, idle-only safety checks).
  • PR [agent-sdk-j2s] agent-server runtime LLM switching #1544 “agent-server runtime LLM switching” (adds REST endpoints and “switch persists across restart” semantics).
• Posted a detailed, markdown-rendered landing plan as a comment on this PR (feat(llm): add LLM profiles #1843), split into 5 always-green steps:
  1. Add LLM.profile_id + profile-ref serialization/expansion (no persistence wiring).
  2. Add LLMRegistry profile helpers wrapping the existing LLMProfileStore (avoid duplicate stores; preserve metrics-independence behavior on main).
  3. Update ConversationState persistence to optionally store profile refs and expand them on restore (decide restore precedence).
  4. Add SDK runtime switching (LocalConversation.switch_llm and optionally set_llm), with idle/paused-only safety checks.
  5. Add agent-server switching endpoints and restart-persistence integration tests.

Where the plan is posted:

• feat(llm): add LLM profiles #1843 (comment)

No code changes were made in the repository for this request; I only performed investigation and posted the plan comment.

[all-hands-bot]

[Automatic Post]: It has been a while since there was any activity on this PR. @enyst, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

[all-hands-bot]

[Automatic Post]: It has been a while since there was any activity on this PR. @enyst, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

[all-hands-bot]

[Automatic Post]: It has been a while since there was any activity on this PR. @enyst, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

[all-hands-bot]

[Automatic Post]: It has been a while since there was any activity on this PR. @enyst, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

[enyst]

@OpenHands understand this pr very well, then merge main into it carefully (fix conflcits, but also keep an eye for redundancies that the commits on main to this may have intro)

[openhands-ai]

I'm on it! enyst can track my progress at all-hands.dev

[github-actions]

Python API breakage checks — ✅ PASSED

Result: ✅ PASSED

Action log

[github-actions]

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

[openhands-ai]

Final summary of new work since the last summary:

• Merged origin/main into agent-sdk-18-profile-manager
• Resolved the PR branch conflicts carefully
• Checked for redundancies introduced by the merge and removed one real one:
  • duplicate self.llm_registry = LLMRegistry() in LocalConversation
• Preserved the intended LLM profile behavior while keeping newer main changes:
  • LLM: kept profile-ref expansion and newer secret handling
  • LLMRegistry: kept profile persistence helpers and main’s independent-metrics logic
  • wsproto test: kept main’s dual auth env var setup
  • removed the obsolete old workflow test path in favor of main’s relocated tests/cross/test_resolve_model_config.py
• Ran focused validation:
  • pre-commit on the directly edited files passed
  • focused pytest set passed: 105 passed
  • tests/agent_server/test_agent_server_wsproto.py still hung in this environment, but collection succeeded and a manual startup check for the updated auth env setup succeeded
• Committed and pushed the merge result to the PR branch
  • pushed commit: d1a49985

Conciseness check:

• The changes were kept focused to merge resolution plus one small AGENTS memory note.
• I did not add extra product code beyond what was needed to reconcile main with this PR.
• No obvious extraneous merge-resolution edits remain to revert.