feat(service-automation, runtime, spec): connector credential store and open-tier credentialRef chain (#27237)

.changeset/27237-cli-ownership-inventory-connector-credential.md

@@ -0,0 +1,10 @@
+---
+'@objectstack/cli': patch
+---
+
+fix(cli): the organization-ownership inventory has a row for `sys_connector_credential`
+
+Clause-②: no
+Metadata-protocol: none
+
+The ADR-0131 D10 inventory now has a row for the new environment-level connector credential store (`systemFields.tenant: false`). Its fate is `column-drop`, the same fate the flow credential store has: the upgrade plan treats the table as deployment-level state with no organization column, so a database carrying it is not refused as an uninventoried platform table.

.changeset/27237-connector-credential-refs-routes.md

@@ -0,0 +1,12 @@
+---
+'@objectstack/runtime': patch
+---
+
+feat(runtime): `GET`/`PUT`/`DELETE /api/v1/automation/connectors/credential-refs` — list the connector credentials an environment's connectors name, and set or clear one
+
+Clause-②: yes (widening)
+Metadata-protocol: none
+
+Three new routes serve the automation service's connector credential store (#27237). `GET …/credential-refs` lists every `credentialRef` an installed connector names, plus stored references nothing names any more. Each entry has its `status` (`set` / `missing` / `rejected`), its `source` (`env` / `object` / `vault` / null) and the instances naming it, and never a value. `PUT …/credential-refs/:ref` with the closed body `{ "value": "…" }` stores a value, and the instances naming the reference re-materialize without a restart. `DELETE …/credential-refs/:ref` removes it.
+
+All three require the `manage_metadata` capability. Under the `group` and `isolated` tenancy postures they refuse every caller with a 403 that names the posture, because connector credentials are environment-level. They take no query parameters. The service's refusals keep their standard codes: 400 `VALIDATION_ERROR` (an empty value, or one starting with `secret:`), 403 `PERMISSION_DENIED` (the service's own walled refusal), 404 `RESOURCE_NOT_FOUND` for an undeclared reference, 409 `RESOURCE_CONFLICT` for one an environment variable supplies or a first write another replica made at the same moment, and 503 `SERVICE_UNAVAILABLE` with no crypto provider. A mounted service without these optional members answers 501.

.changeset/27237-connector-credential-rejected-marker.md

@@ -0,0 +1,18 @@
+---
+'@objectstack/spec': patch
+---
+
+feat(spec): `ConnectorCredentialRejectedError` and the connector credential members of `IAutomationService`
+
+Clause-②: yes (widening)
+Metadata-protocol: none
+
+Three additions for the connector credential store (#27237):
+
+- `@objectstack/spec/integration` exports `CONNECTOR_CREDENTIAL_REJECTED`, `ConnectorCredentialRejectedError` (with `status`: 401 or 403) and the structural check `isConnectorCredentialRejected`. They sit beside the upstream-unavailable marker. A connector provider factory throws it when the upstream refuses the credential it sent, and the materializer then degrades the instance instead of failing the boot. The code is registered in the error-code ledger under `@objectstack/spec`.
+- `ConnectorMaterialization` gains the optional `credentialVerified`. A provider factory sets it when the upstream accepted the instance's credential while materializing (the MCP handshake over http); the automation service then records the credential as verified, and a stored one marked `rejected` returns to `set`. A provider that first sends the credential on a call leaves it unset; a call answered 2xx is not taken as proof.
+- `IAutomationService` gains three optional members, `listConnectorCredentialRefs`, `setConnectorCredential` and `clearConnectorCredential`, with their types `ConnectorCredentialRef`, `ConnectorCredentialSetRequest` and `ConnectorCredentialClearRequest`. A listing entry never carries a value.
+
+`PLATFORM_OBJECT_NAMES['service-automation']` lists the new `sys_connector_credential` object.
+
+The descriptions now match the new behaviour. The stack's `connectors` field says a `credentialRef` nothing resolves, or one the upstream refuses, degrades that instance until the credential is set, where it said such a reference failed the boot. The auth and connector schemas say a credential is resolved when the instance materializes, not at boot.

.changeset/27237-connector-credential-store.md

@@ -0,0 +1,21 @@
+---
+'@objectstack/service-automation': patch
+---
+
+feat(service-automation): an administrator sets a declarative connector's credential in the environment's credential store, and a missing one degrades its instance instead of failing the boot
+
+Clause-②: yes (widening)
+Metadata-protocol: none
+
+A provider-bound `connectors:` entry names its secret with `auth.credentialRef`. Before, the open tier resolved that name from the server's environment variables only, and a name with no value failed the boot. On a hosted environment, where the administrator who installs an app has no server environment, a connector-bearing app could never be credentialed.
+
+The reference now resolves through three sources, in order: an environment variable of the same name (it always wins), then the new `sys_connector_credential` store, then the host's `credentialResolver` if one is injected. The store keeps one row per reference with its value in a `Field.secret` column: encrypted at rest by the host crypto provider and masked on every read. It is written through `setConnectorCredential` / `clearConnectorCredential` and listed by `listConnectorCredentialRefs` on the `automation` service, which the REST routes `GET`/`PUT`/`DELETE /api/v1/automation/connectors/credential-refs` serve.
+
+- A reference no source answers for degrades the instance naming it, with a reason that names the reference and how to set it. Before, it failed the boot. A `credentialResolver` that throws is still fatal at boot.
+- Setting or clearing a credential reconciles at once: every instance naming the reference re-materializes, or is torn down and degraded, with no restart.
+- A credential the upstream refuses (HTTP 401 / 403) degrades the instance with the upstream's message, never the value. An `mcp` instance is refused when it connects; a `rest` or `openapi` instance on the first call that answers 401 / 403, which still returns its answer. A stored credential is marked `rejected`, and its instance is not retried until the credential is set again, the declaration changes, or the server restarts; the mark itself stays until the value is set again or an `mcp` handshake proves it. A call answered 2xx does not clear it, since the endpoint may not check the credential; it only stamps the last-verified time of a credential still `set`. A credential from an environment variable or the host's resolver is retried on the usual backoff instead, since it can change where the service cannot see it, and the listing reports it `rejected` until a provider proves it or the server restarts.
+- Under the `group` and `isolated` tenancy postures the store is not read, and writes to it are refused with a 403 naming the posture.
+- A value starting with `secret:` is refused with a 400: the data engine would store it as a handle to another stored secret. Writes run one at a time with their reconcile, so the instances end on the value the store ends on; a first write another replica makes at the same moment answers 409.
+- The value never appears in a log record, the listing, `GET /connectors`, an error body or a read of the object. A provider message that echoes it carries `[credential withheld]` instead, and so does one that echoes it percent-encoded or base64-encoded, including the base64 `user:value` pair of a basic credential.
+
+A deployment that relied on the boot failing for an unset reference now boots, with that instance degraded and named on `GET /api/v1/automation/connectors`.

.changeset/27237-install-local-credential-refs.md

@@ -0,0 +1,10 @@
+---
+'@objectstack/cloud-connection': patch
+---
+
+feat(cloud-connection): the install-local answer lists the connector credentials the installed package names, and which are not set yet
+
+Clause-②: yes (widening)
+Metadata-protocol: none
+
+`POST /api/v1/marketplace/install-local` answers with `credentialRefs` when the package's connectors name any `auth.credentialRef`. Each entry has the reference, the instances naming it, its `status` from the automation service's credential listing (`set` / `missing` / `rejected`, or `unknown` when that listing cannot be read) and its `source`, never a value. When any of them is not set, a `credentialNote` names them and says how to set them. An unset reference never refuses the install: the instance naming it installs degraded and materializes once the credential is set (#27237). Under the `group` and `isolated` tenancy postures, where the credential routes refuse writes, the note names the environment variable or the host's resolver instead of the `PUT` route. The answer's keys are unchanged for a package whose connectors name no credential.

.changeset/27237-mcp-credential-rejected.md

@@ -0,0 +1,12 @@
+---
+'@objectstack/connector-mcp': patch
+---
+
+fix(connector-mcp): a credential the MCP server rejects is thrown as `ConnectorCredentialRejectedError`, so the instance is degraded instead of failing the boot
+
+Clause-②: yes (widening)
+Metadata-protocol: none
+
+An HTTP 401 / 403 on connect, handshake or `tools/list` used to throw a plain `Error`, which failed the boot. The provider now throws `ConnectorCredentialRejectedError` from `@objectstack/spec/integration`, with the status and the SDK error as `cause`; its message no longer says the fault "is not retried", since whether it is now depends on where the credential came from. The automation service degrades the instance naming the refusal and marks a stored credential `rejected` (#27237). Every other connect or `tools/list` failure is unchanged.
+
+An http instance whose handshake carried its credential (any auth but `none`, and not an api-key sent as a query parameter) now answers `credentialVerified: true`, so a stored credential the server accepted is recorded as verified.

content/docs/automation/connectors.mdx

@@ -76,7 +76,10 @@ At boot, the automation service resolves each provider-bound entry:
    three in `plugins:` by default, paired with the `automation` capability
    that performs the materialization.
 2. **The factory validates `providerConfig`** and the automation service
-   resolves `auth.credentialRef` to its secret.
+   resolves `auth.credentialRef` to its secret (see
+   [What `credentialRef` resolves through](#what-credentialref-resolves-through)).
+   A reference that is not set yet does not stop the boot: that one instance
+   registers degraded until an administrator sets it.
 3. **The result is registered** on the connector registry — indistinguishable
    from a hand-written connector. Its **actions are derived from the
    upstream** (the OpenAPI document's operations, the MCP server's
@@ -224,8 +227,9 @@ paginated tool list is followed through every page (`nextCursor`), up to 100
 pages; a server still paging after that fails discovery instead of yielding a
 truncated tool list. A server that is unreachable or failing degrades the
 instance, which is retried. A credential the server rejects (HTTP 401 / 403 on
-connect, handshake or `tools/list`) is a configuration fault instead: it fails
-the boot and is not retried — fix the credential and reload.
+connect, handshake or `tools/list`) is a configuration fault instead: the
+instance is degraded naming the refusal, as described under *A credential the
+upstream refuses* below.
 
 <Callout type="warn">
 **Declarative stdio transports are denied by default.** Materializing one
@@ -275,18 +279,94 @@ Variant notes:
 - **`basic`** — the `username` stays in metadata (it is not a secret); only
   the password resolves through `credentialRef`.
 
-**What `credentialRef` resolves through.** In the open tier the ref is the
-name of an **environment variable** (`BILLING_API_TOKEN` above means
-`process.env.BILLING_API_TOKEN` on the server). A ref that resolves to nothing
-is a **hard boot error** — an app must not silently run with a dead connector.
-The enterprise tier swaps in a vault/KMS-backed resolver
-(`AutomationServicePluginOptions.credentialResolver`) without changing what
-you author: the entry still just names a ref.
+### What `credentialRef` resolves through
+
+The metadata author names a credential; the administrator who installs the app
+holds it. The automation service asks three sources, in order, and the first
+that answers wins:
+
+1. **An environment variable of the same name** on the server
+   (`BILLING_API_TOKEN` above means `process.env.BILLING_API_TOKEN`). A
+   self-hosted operator keeps this path, and it always wins.
+2. **The connector credential store**, `sys_connector_credential`: one row per
+   reference, its value encrypted at rest by the host's crypto provider and
+   masked on every read. An administrator fills it through the API below, with
+   no access to the server environment, which is the hosted case.
+3. **The host's resolver**, when it injects one
+   (`AutomationServicePluginOptions.credentialResolver`, such as a vault or KMS
+   client). You author the same entry either way: it still just names a ref.
+
+A reference no source answers for **does not fail the boot**. An app installs
+first and gets its credentials after, so that one instance registers
+**degraded**: it is listed on `GET /api/v1/automation/connectors` with a reason
+that names the reference, and a dispatch to it fails with the same reason.
+
+#### Setting a credential
+
+Three routes, each requiring the `manage_metadata` capability:
+
+```http
+GET    /api/v1/automation/connectors/credential-refs
+PUT    /api/v1/automation/connectors/credential-refs/BILLING_API_TOKEN   { "value": "…" }
+DELETE /api/v1/automation/connectors/credential-refs/BILLING_API_TOKEN
+```
+
+- **`GET`** lists every reference an installed connector names, plus every
+  stored one that nothing names any more. Each entry has a `status` (`set`,
+  `missing` or `rejected`), a `source` (`env`, `object`, `vault` or `null`),
+  whether this door can write it, and the instances naming it with their state.
+  A stored one also says when and by whom it was last set (`updatedAt`,
+  `updatedBy`) and when it was last seen working (`lastVerifiedAt`). It never
+  includes a value.
+- **`PUT`** stores the value, then reconciles. Every instance naming the
+  reference re-materializes with it **without a restart**, and the answer is
+  the reference's entry after that. The route refuses, before writing anything:
+  a reference no installed connector declares (`404`), one an environment
+  variable already supplies (`409`, because the variable wins and a stored value
+  would never be read), an empty value, and a value starting with `secret:`
+  (`400`: the store would read it as a handle to another stored secret). With no
+  crypto provider it answers `503`, since a credential is only ever stored
+  encrypted. Writes on one server run one at a time, each with its reconcile, so
+  the instances always end on the value the store ends on.
+- **`DELETE`** removes the stored value. The instances naming the reference
+  are torn down and registered degraded again.
+
+Installing a package reports the same facts for it: the install-local answer
+(`POST /api/v1/marketplace/install-local`, which `os package install` calls)
+lists the `credentialRefs` the package's connectors name, each with its status,
+plus a `credentialNote` naming the ones still to set.
+
+**A credential the upstream refuses** (HTTP `401` / `403`) degrades the
+instance, with a reason quoting the upstream's message, never the value. An
+`mcp` instance finds this out when it connects. A `rest` or `openapi` instance
+finds out on the first call answering `401` / `403`; that call returns its
+answer as usual, and the instance is degraded after it. What happens next
+depends on where the value came from:
+
+- **A value stored here** is marked `rejected`, and its instance is not
+  retried until the credential is set again, its declaration changes, or the
+  server restarts. The mark outlives a restart: it clears only when the value
+  is set again or an `mcp` handshake that carries it succeeds. A `rest` /
+  `openapi` call answered `2xx` does not clear it, because the endpoint may not
+  check the credential at all.
+- **A value from an environment variable or the host's resolver** can change
+  where this door cannot see it (a rotation, or a `403` rate limit that lifts),
+  so its instance is retried on the same 5s→5min backoff as an unreachable
+  upstream. The listing reports it `rejected` until an `mcp` handshake proves
+  it or the server restarts, even while the instance is serving again.
+
+Credentials are **environment-level**: one value per reference for the whole
+environment, read by whichever instance names it. On a multi-organization
+environment (the `group` or `isolated` tenancy posture) the store is not used
+at all. The three routes refuse with a message naming the posture, and a
+reference resolves from the environment variable or the host's resolver only,
+so a row stored before the posture changed is not read.
 
 The same resolver serves an `api` flow's start-node `config.secretRef`, the
-name of its inbound hook secret. That one is read at every post rather than at
-boot, so a name with no value refuses each post with `503` instead of failing
-the boot ([Flows](/docs/automation/flows)).
+name of its inbound hook secret. That one goes through the injected resolver
+alone (environment variables by default) and is read at every post rather than
+at boot, so a name with no value refuses each post with `503`
+([Flows](/docs/automation/flows)).
 
 Inlining a secret is rejected when the stack is validated. Writing the runtime
 shape on a provider-bound instance —
@@ -359,7 +439,8 @@ retry):
 |:---|:---|
 | `provider` names no installed factory | **Hard boot error** naming the entry, the provider, and the plugin that supplies it — add the executor to `plugins:` |
 | Invalid `providerConfig` (missing `baseUrl` / `spec` / `transport`, wrong shape) | **Hard boot error** from the provider factory |
-| `credentialRef` resolves to nothing | **Hard boot error** — set the env var (open tier) or fix the vault ref |
+| `credentialRef` resolves to nothing | **Not fatal**: the instance registers **degraded**, naming the reference — set it (`PUT …/credential-refs/:ref`), and it materializes without a restart |
+| The upstream refuses the credential (`401` / `403`) | **Not fatal**: the instance registers **degraded**. A stored credential is marked `rejected` and waits until it is set again (or a restart); an environment or vault credential is retried on the backoff |
 | Instance `name` collides with a plugin-registered connector | **Hard boot error** — there is no silent precedence between the two worlds |
 | stdio transport without host opt-in | **Hard boot error** with the `declarativeStdio` fix inline — a security rejection is never retried |
 | MCP server unreachable / `tools/list` fails; remote spec URL unreachable or transiently failing (408/429/5xx) | **Not fatal**: the instance registers **degraded** (visible on `GET /api/v1/automation/connectors` with the reason), dispatches fail with that reason, and the platform retries on a 5s→5min backoff — recovery is automatic |