Skip to content

Commit a2f434d

Browse files
fix(service-automation): a connector credential written through the door reaches every replica (#27941)
<!-- claude-projects-attribution: {"github_login":"os-elon-musk"} --> _Requested by **Elon** · [project thread](https://claude.ai/code/project/chan_01WedDzV26nniwxv7jC88B9D?thread=cmsg_01WedDzV26nniwxv7jC88B9DEvHfdgE5C54xKaAXm6nH7d)_ Fixes #27428 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:多副本部署里,任一副本通过凭据门写入或清除连接器凭据后,发一条仅含地址的集群广播(`connector-credential.mutated`,载荷 `{ originNode, ref }`);其它副本收到后跑自己的连接器对账,重读共享凭据行,被挂起的连接器随即恢复。 - 为什么改:此前只有接请求的那个副本恢复,其它副本上被上游拒过的连接器一直挂着,直到元数据重载或重启(裁决 6513279565,字母 A 挂 A1)。 - 风险与回滚:只加一条内部通道,单副本或内存集群驱动下不挂载、行为不变;至多一次投递,丢失由下次重载或重启兜底。回滚即撤本 PR。 - 先看:`packages/services/service-automation/src/plugin.ts`(门的发布与收方对账)、`packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts`(第四条通道)、`packages/services/service-automation/src/connector-credential-cluster.test.ts`(双内核 pin)。 - 两分钟验证:`pnpm --filter @objectstack/service-automation exec vitest run src/connector-credential-cluster.test.ts`,预期 5 个用例全绿。 - 需要你做的:无,席位落地。 ## What changes - **The publish.** The connector credential door (`PUT|DELETE /api/v1/automation/connectors/credential-refs/:ref`) publishes `{ originNode, ref }` on `connector-credential.mutated` right after `store.set` succeeds and after `store.clear` clears a row, then reconciles its own replica as before. Address only: no value, fingerprint or revision rides the channel. A refused write (400 / 403 / 404 / 409 / 503) publishes nothing. Best-effort and fire-and-forget, like lanes 2 and 3: a publish failure is logged at `warn` and never fails the write. - **The receiver.** A peer drops its own `originNode`, then runs `reconcileDeclaredConnectors` on its reconcile queue (`materializeDeclaredConnectors(ctx, { fatal: false })`), which re-reads the shared `sys_connector_credential` rows. The row's revision joins each instance's signature, so exactly the instances naming the changed reference re-materialize, held ones included; a cleared reference degrades them, naming it. It logs one `info` line per peer write it follows. - **The seam.** `AutomationEngine.attachConnectorCredentialPubSub(pubsub, nodeId)` delegates to the plugin's credential door (`ConnectorCredentialDoor.attachClusterPubSub`), idempotent on `(pubsub, nodeId)` and detached at `destroy()`. The channel name and payload type live in `connector-credentials.ts`; nothing new is exported from the package root. - **Lane 4 in `MetadataClusterBridgePlugin`.** Duck-typed against the `automation` service's `attachConnectorCredentialPubSub()` beside lanes 1 to 3, skipped on the in-process `memory` driver, its own attach error and detach on `kernel:shutdown`. - **Docs.** `content/docs/kernel/cluster.mdx` §6.2 gains the lane 4 row; the family's prose now says four lanes and names the credential rows as lane 4's source of truth. - Changeset: `patch` for `@objectstack/service-automation` and `@objectstack/service-cluster`. ## Why Ruling 6513279565 on #27428 (letter A, attached at A1): one new address-only lane of the §6.2 family, published by the credential door, received as a reconcile that re-reads the shared row, enumerated in the bridge with the other three lanes. Not a synthetic `metadata.mutated`, no polling, no self-attach inside service-automation. ## Tests - `packages/services/service-automation/src/connector-credential-cluster.test.ts` (new): two `AutomationServicePlugin` kernels over ONE SQLite database and one shared bus that delivers to every subscriber, publisher included. Each replica is attached through `attachConnectorCredentialPubSub()`, the seam the bridge calls. - a PUT on A re-materializes the instance B holds (its stored value was refused upstream, so no retry timer) with the new value, with no reload on B; - a DELETE on A degrades B's live instance, naming the reference; - control: B with no transport stays held after A's PUT, the pre-fix shape; - the payload is exactly `{ originNode, ref }` with partition key `connector-credential:REF`, after a set and after a clear, and a refused write publishes nothing; - A's own publish is dropped on A and followed on B, and a write on B reaches A. - `packages/services/service-cluster/src/metadata-cluster-bridge-plugin.test.ts`: six lane-4 cases (attach on a cross-process driver with the cluster's pubsub and node id, all four lanes together, skip on `memory`, quiet skip with no service or no seam, a throwing attach that leaves lanes 1 to 3 up, detach on shutdown when the lane-3 detach throws). ## Acceptance notes - The dev's earlier observation on this card (a connector changed by a publish may materialise only on the publishing replica) was not measured here, as the ruling directs; it is the sibling question for its own card. - `AutomationEngine` gains one public method, the duck-typed seam. The claim's `Clause-②: no` is copied as written; the report asks the seat whether that method counts as a widened public surface. - Draft PR #27819 also edits `service-automation/src/plugin.ts`; this diff keeps to the door's two publish lines, the receiver and the destroy line, so the later lander merges `main`. ## Verification Head `f78a7f75`. <details><summary>Red on the base, ablations, restore</summary> - The new test file against `origin/main` `8c7021df` (the three source files restored from the base): 5/5 red, `TypeError: engine.attachConnectorCredentialPubSub is not a function`. - Through `scripts/ablation-replace.mjs` (anchor hit 1 → 0, restored blob == HEAD, `git diff HEAD` empty after each): - the set door's publish removed: 4 red (the PUT, DELETE, payload and loopback cases), the control green; - the clear door's publish removed: 2 red (the DELETE and payload cases), 3 green; - the loopback guard removed: 1 red (the loopback case), 4 green. </details> <details><summary>Package suites and gates</summary> - `pnpm --filter @objectstack/service-automation exec vitest run --maxWorkers=2`: 219 files, 2808/2808 passed. - `pnpm --filter @objectstack/service-cluster test`: 7 files, 111/111 passed; `typecheck` exit 0 for both packages. - `node scripts/pm/dispatch-gates.mjs --commands`: 92 derived families, all run, `--ran` reconciliation "92 derived, 92 run, 0 NOT-MEASURED, 0 UNRUN". 89 exited 0 on the first pass; three first answered exit 3 (prerequisite) and exited 0 once it was met: `check-plugin-teardown-shape --self-test` (pinned commit fetched into the shallow clone), `spec check:skill-examples` and `check:dual-build-cjs-loads` (after a whole-tree `turbo run build`). - `pnpm lint`: exit 0. </details> --- _Generated by [Claude Code](https://claude.ai/code/session_016WV4d1Ds3FbhoRdoBLGUvZ)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 421bb6b commit a2f434d

8 files changed

Lines changed: 672 additions & 17 deletions

File tree

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
'@objectstack/service-automation': patch
3+
'@objectstack/service-cluster': patch
4+
---
5+
6+
fix(service-automation): a connector credential written or cleared through the door reaches every replica, not only the one that took the request
7+
8+
Clause-②: no
9+
Metadata-protocol: none
10+
11+
`PUT` and `DELETE /api/v1/automation/connectors/credential-refs/:ref` reconciled the declarative connector instances of the replica that served the request. On a multi-replica deployment every other replica kept what it had: an instance whose stored value its upstream had refused is held with no retry timer, so a corrected credential took effect there only after a metadata reload or a restart, and requests landing on those replicas kept failing against a degraded connector.
12+
13+
- **A fourth cluster lane.** After the row is written or cleared, the door publishes `{ originNode, ref }` on the `connector-credential.mutated` cluster channel. The payload is an address only: never a value, a fingerprint or a revision.
14+
- **Peers converge from their own read.** A peer drops its own `originNode`, then runs its connector reconcile on its reconcile queue, which re-reads the shared `sys_connector_credential` rows. Exactly the instances naming a changed reference re-materialize, held ones included; a cleared reference degrades them, naming it.
15+
- **Attached where the other lanes are.** `MetadataClusterBridgePlugin` (`@objectstack/service-cluster`) attaches the lane at `kernel:ready` beside `metadata.changed`, `metadata.mutated` and `datasource.mutated`, duck-typed against the automation service's `attachConnectorCredentialPubSub()`. As with lanes 2 and 3, it is skipped on the in-process `memory` cluster driver, so a single replica behaves as before.
16+
- **What it does not promise.** Delivery is at-most-once, as for every lane of the family: a lost message is not retried, and the next metadata reload or restart still reconciles that replica.

‎content/docs/kernel/cluster.mdx‎

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -298,7 +298,7 @@ they declare scope and strategy, and the runtime does the rest.
298298
Metadata is the most cache-hot data in the system. Every read path —
299299
ObjectQL planning, REST routing, UI rendering, permission checks —
300300
touches it. Making it correct across a cluster needs a well-defined
301-
invalidation signal per state owner — the three lanes of §6.2 — and, in this
301+
invalidation signal per state owner — the four lanes of §6.2 — and, in this
302302
protocol, deliberately **no** per-item version on the wire (§6.1).
303303

304304
### 6.1 Monotonic version
@@ -326,39 +326,41 @@ cache compare it on receipt.
326326
### 6.2 The cluster-invalidation family
327327

328328
Cross-node metadata invalidation is not one channel — it is a **family** of
329-
three independent PubSub lanes, late-bound by `MetadataClusterBridgePlugin`
329+
four independent PubSub lanes, late-bound by `MetadataClusterBridgePlugin`
330330
(`@objectstack/service-cluster`) at `kernel:ready`, one per state owner that
331-
goes stale when metadata changes on a different node:
331+
goes stale when a shared record changes on a different node:
332332

333333
| Lane | Channel | Payload type | Package | Converges |
334334
|---|---|---|---|---|
335335
| 1 — metadata service cache | `metadata.changed` | `ClusterMetadataChangedPayload` | `@objectstack/metadata` | `MetadataManager`'s registry/list caches |
336336
| 2 — ObjectQL object registry | `metadata.mutated` (`METADATA_MUTATION_CLUSTER_CHANNEL`, #13331) | `ClusterMetadataMutationPayload` | `@objectstack/metadata-protocol` | ObjectQL's OBJECT registry, from a `sys_metadata` re-read |
337337
| 3 — ObjectQL driver registry | `datasource.mutated` (`DATASOURCE_MUTATION_CLUSTER_CHANNEL`, #13805) | `ClusterDatasourceMutationPayload` | `@objectstack/service-datasource` | ObjectQL's DRIVER registry, from a shared datasource-record re-read |
338+
| 4 — declarative connector instances | `connector-credential.mutated` (`CONNECTOR_CREDENTIAL_MUTATION_CLUSTER_CHANNEL`, #27428) | `ClusterConnectorCredentialMutationPayload` (`{ originNode, ref }`) | `@objectstack/service-automation` | the automation service's materialized connectors, from a reconcile that re-reads the shared `sys_connector_credential` rows; published by the credential door (`PUT` / `DELETE /api/v1/automation/connectors/credential-refs/:ref`) after the row is written or cleared |
338339

339-
Lanes 2 and 3 share one contract, stated once here rather than in each
340+
Lanes 2 to 4 share one contract, stated once here rather than in each
340341
channel's own doc comment:
341342

342343
- **Address-only signal.** No message body ever rides these channels — the
343-
payload names only what changed (a metadata address, or a datasource
344-
name), never the changed value.
344+
payload names only what changed (a metadata address, a datasource name,
345+
or a connector credential reference), never the changed value.
345346
- **Receipt runs convergence, not trust.** A peer that receives a signal does
346347
not apply the payload as content; it re-reads its OWN copy of the source of
347-
truth (`sys_metadata` for lane 2, the shared datasource record for lane 3)
348-
and converges its local registry from that read. A duplicate or re-ordered
348+
truth (`sys_metadata` for lane 2, the shared datasource record for lane 3,
349+
the shared credential rows for lane 4) and converges its local registry
350+
from that read. A duplicate or re-ordered
349351
delivery converges to the same state as a single one.
350352
- **`originNode` loopback suppression.** Every payload carries the publishing
351353
node's id; a peer that receives its own `originNode` drops the message —
352-
the write-through or pool rebuild already ran, synchronously, at the door
353-
that persisted the change.
354+
the write-through, pool rebuild or connector reconcile already ran,
355+
synchronously, at the door that persisted the change.
354356
- **At-most-once, bounded by boot rehydration.** No shipped `IPubSub` driver
355357
exceeds at-most-once delivery (§4.2), so a lost message is not retried.
356358
Each lane narrows the staleness window this always had — "until the next
357359
full reload on restart" — to "one network hop"; it does not promise more.
358360

359-
All three lanes skip their fan-out on the same condition — the in-process
361+
All four lanes skip their fan-out on the same condition — the in-process
360362
`memory` cluster driver, described for lane 1 immediately below, applies
361-
identically to lanes 2 and 3 (#14021).
363+
identically to lanes 2 to 4 (#14021).
362364

363365
#### Lane 1 — `metadata.changed` (the metadata service cache)
364366

0 commit comments

Comments
 (0)