Feat/cloud models#15
Open
dijix009 wants to merge 6 commits into
Open
Conversation
…e 1) First phase of optional hosted-model support: a self-contained, MLX-free CloudInference module that adapts Anthropic's Messages API to Interless's backend-agnostic TokenChunk stream. No wiring yet — local MLX remains the only active backend until phase 2 routes roles to it. - CloudModelClient protocol; CloudProvider + CloudModelResolver (the `provider/model` id convention — `anthropic/claude-opus-4-8` is cloud, bare / Hugging Face ids stay local); CloudKeyProvider with Keychain (SecretStore, account `anthropic.apiKey`) + ANTHROPIC_API_KEY env fallback. - HTTPTransport seam + URLSessionHTTPTransport (the app's first outbound HTTP), injectable so tests run with no network. - AnthropicModelClient: maps GenerationRequest → Messages (system hoisted, text- mode messages with same-role merge, tools→input_schema, max_tokens, temperature); parses streamed SSE → TokenChunk (text deltas, tool_use → ModelToolCall, terminal usage/stop_reason); maps non-2xx → InferenceError with a clear message; clear missing-key error. Module depends only on Shared + InterlessSecurity. Tests drive a fake transport with canned SSE (request mapping, text+tool_use+terminal stream, 401 error, missing key, resolver). Full suite green (370 tests).
Wire the Anthropic adapter into the inference seam without changing the controller or agents. - RemoteInferenceBackend (MLXEngine): an InferenceBackend over a CloudModelClient. Lifecycle is hollow — no local weights/KV/GPU/RAM, so load/unload/clearKVCache/ draft are no-ops and memory readings are zero; load validates the API key so a missing key surfaces at load time. generate delegates to the client using the model id on the handle (provider prefix stripped). embed throws (embeddings stay local). - RoutingInferenceBackend (MLXEngine): dispatches by model id — a `provider/model` id (anthropic/…) routes to the remote backend, everything else (bare / Hugging Face ids) to local MLX. No mutable routing table: load/generate/embed route by id, role-keyed calls forward to both sub-backends (no-op where absent), and countTokens/gpuMemory/footprint defer to local (correct for cloud roles too). - EngineBootstrap composes RoutingInferenceBackend(local: MLXBackend, remote: RemoteInferenceBackend(AnthropicModelClient())). Local-only behavior is unchanged; the remote backend is inert until a cloud id is loaded. MLXEngine now depends on CloudInference. Tests: remote backend delegates + strips the prefix, embed throws; routing dispatches local vs cloud by id. Full suite green (373 tests). Note: consent (allowCloudModels) + the loadModels/RuntimeConfigMapper wiring that lets a user actually select a cloud model land in phase 2b; UI in phase 3.
Make hosted models actually selectable per role, gated by explicit consent. - ModelSettingsViewState gains allowCloudModels (default false) — distinct from allowNetworkTools because it sends prompts + workspace context to the provider. A migration-tolerant init(from:) is added so older persisted settings (missing the new key) still decode. - AppDependencyFactory.loadModels validates cloud usage before loading: a cloud orchestrator/utility role is refused unless allowCloudModels is on, and cloud embedding models are rejected (embeddings stay local). Errors surface through the existing invalidModelSettings path. Cloud roles otherwise load through the routing backend with no download/quantization (RAM-free). AppCore now depends on CloudInference (for the provider/model resolver). Tests: local roles never need consent; cloud roles require it; cloud embeddings are unsupported even with consent. Full suite green (377 tests). UI (key field, consent toggle, cloud status) lands in phase 3.
Wire the cloud-model controls through the view chain so they're user-reachable: WorkspaceActions → WorkspaceView → SettingsHubView → ModelSettingsView, plus the session key handlers in the app composition root. This completes the Settings surface (the field/toggle live in ModelSettingsView and WorkspaceSessionModel from the prior phase): an Anthropic API-key field (Keychain, account anthropic.apiKey), an "Allow cloud models" consent toggle in the Danger Zone with a privacy warning that prompts + workspace context are sent to the provider, and model-id fields that accept anthropic/… ids. Full suite green (377 tests); app builds.
Add OpenAI as a sibling cloud provider behind the existing CloudModelClient abstraction, so a role can use openai/gpt-4o (mix freely with local MLX or Anthropic per role). - CloudProvider gains .openai (Keychain account openai.apiKey, OPENAI_API_KEY env fallback); OpenAIModelClient maps GenerationRequest → Chat Completions (Bearer auth, system kept inline, tool turns as user text + merge, tools via ToolDefinition.schema, max_tokens, stream + usage) and parses streamed chunks → TokenChunk (content deltas; incremental tool_calls accumulated by slot and flushed as ModelToolCall; usage/finish_reason → CompletionInfo). 4xx/5xx → a clear InferenceError. - RemoteInferenceBackend generalized from one client to [CloudProvider: CloudModelClient], resolving the provider from the model id; EngineBootstrap registers anthropic + openai. - Consent + resolution are provider-agnostic, so openai/... is gated by the same allowCloudModels consent (no new gate). - Settings gains an OpenAI API-key field mirroring the Anthropic one, wired through the view chain to WorkspaceSessionModel.save/deleteOpenAIAPIKey. Tests: OpenAI adapter (content+tool_calls stream, request mapping, 401, missing key, resolver) and multi-provider backend dispatch (anthropic vs openai by id). Full suite green (383 tests). Reasoning models (max_completion_tokens / no temperature) are out of scope for this first cut.
Two role-management decisions, written for the all-local world, blocked mixing models per role (e.g. openai/gpt-5.5 orchestrator + anthropic/claude-haiku-4-5 sub-agent, or a cloud orchestrator + local sub-agent). Fix both: - effectiveSingleAgentMode(settings:orchestratorID:utilityID:): single-agent collapse now only applies when small-RAM AND both roles are local. A cloud role costs zero local RAM, so mixing is allowed even on an 8 GB Mac. Used in loadModels (which model(s) to load) and makeAgent (route/prompt/agent selection) so both agree. - advertisesNativeTools(modelID:toolCallFormat:): cloud models do native tool-calling and no longer require a local toolCallFormat. The main path gates on the orchestrator id, the sub-agent path on the utility id (where read-only sub-agents run); local roles still need a configured format. The sub-agent runs on the Utility role's model; embeddings stay local. Tests: single-agent matrix (small/large RAM × local/cloud) and tool-advertise matrix (cloud/local × format). Full suite green (385 tests).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.