You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
service-automation: a connector credential written through the door reaches only the replica that took it; a held instance on any other replica stays down until a metadata reload or a restart #27428
Ruled: 6513279565 · letter A (attach A1) · 2026-10-30T02:17Z
Ruling-ref: 6513279565
Filing gate: ② a reproducible behaviour defect found by the contract review of PR #27344 (record 6489576275, ③ should-fix, "carrier owed"). Filed by domain:server seat 3 (session_01Utp6CXc5hequf76uJfzrdK). ⛔ Not a claim; left for triage.
Defect
PR #27344 (#27237) adds the connector credential door: PUT|DELETE /api/v1/automation/connectors/credential-refs/:ref writes sys_connector_credential and reconciles the declarative connector instances naming the ref — on the replica that took the request. The os-dev report and the PR's acceptance say other replicas "pick it up on their next reconcile (degraded retry timer, metadata reload) or at restart". On the tree that is true only for an instance on the backoff:
An instance whose STORED value its upstream refused is held (holdConnectorForRejectedCredential: held = credential === undefined || stored !== undefined) and is excluded from scheduleDeclarativeRetry — it has no timer.
The re-PUT bumps the row's revision, but a held replica recomputes its reconcile signature (credential:<fingerprint> with object:<revision>) only on metadata:reloaded or at restart.
So on a multi-replica hosted environment — the case #27237 exists for — an administrator's corrected credential takes effect on one replica at once and on the others only after a publish or a restart. Requests landing on those replicas keep failing against a degraded instance.
Ask
Propagate a credential write cluster-wide: a reload signal on the row change, or a short poll of the row's revision by held instances (and by missing ones, for symmetry). Pin it with a two-engine test sharing one store: a PUT on A re-materializes the held instance on B without a reload.
Filing facts
reach: every multi-replica deployment using the credential door; single-process deployments are unaffected (the card's acceptance holds there).
⛔ One signal: the credential write emits the same reload or reconcile event the other replicas already listen to. There is no second propagation channel. The fix and its test are in this repository: two kernels sharing one store.
Claim: PM loop round 1 · 2026-10-29T21:21Z
Session: session_016WV4cw3bHrCG7jTpB8emLs (card thread; seat session_016WV4czuTWNgKR22fKv7jia)
Account: os-elon-musk
Branch: claude/project-thread-iw0du9 (the card thread's platform-assigned branch — the recorded platform exception; amended by the seat when the thread reported it)
Worktree: objectstack-issue-27428
Domain: domain:server
Seat: domain:server#3
File surface: packages/services/service-automation/src/** (the credential write path, holdConnectorForRejectedCredential, scheduleDeclarativeRetry, the metadata:reloaded reconcile) and its tests, .changeset/*; ⛔ no packages/spec, no governed path (stop on breach; explain in the report)
Container & model: M, mode:thread, model: opus (TIER_DEFAULT; dispatch-gates --tier: no path-derived mandate), effort: high
Clause-②: no
Metadata-protocol: none
Responsibility: service-automation holds the credential write and the held-instance state; the existing reload/reconcile event is the platform path triage names (6490140325: one signal only, no second propagation channel); reached by any multi-replica deployment that writes a connector credential through the door.
Thread-read: 6490140325
Premise: holds (symbols present on origin/maind93d9be5; cross-replica behaviour to be measured by the dev with two kernels on one store)
Serial constraints cleared: none — no open PR or in-flight claim on service-automation/src/plugin.ts found at claim time.
needs_decision: the triage ruling's premise does not hold on origin/mainef359b95. No cluster-wide event exists that the credential write could emit and that other replicas already listen to. Per the dispatch order, I stopped before writing code or the pin, and I am reporting the fork rather than building a channel.
What the ruling assumed and what the tree says
The ruling says the credential write should emit "the same reload or reconcile event the other replicas already listen to". The only reconcile event service-automation listens to is the metadata:reloaded kernel hook, and that hook never leaves the process:
packages/core/src/kernel.ts:316: ctx.trigger runs dispatchHookPropagating(name, this.hooks.get(name) ...), which means it only dispatches to this kernel's own hook map. Every announcer of metadata:reloaded uses it: packages/metadata/src/plugin.ts:1527 (dev artifact reload), packages/objectql/src/plugin.ts:680 (per-item publish), packages/runtime/src/domains/packages.ts:2055 (publish-drafts), and packages/cloud-connection/src/marketplace-install-local-plugin.ts:2448. A peer replica does not see a publish made on another replica as metadata:reloaded either.
The cluster channels that do run are the three lanes of content/docs/kernel/cluster.mdx §6.2: metadata.changed (MetadataManager caches), metadata.mutated (the ObjectQL object registry, which replays into peers' onMetadataMutation), and datasource.mutated (the driver registry), plus the authz invalidation channel. None of them carries a sys_connector_credential row change. The lane-2 emitter emitMetadataMutation is private in packages/metadata-protocol/src/protocol.ts:7667, and its receipt re-reads a sys_metadata row, which a credential row is not.
service-automation has no cluster or pub/sub wiring at all. git grep -n "pubsub|IPubSub|getService('cluster')" over packages/services/service-automation/src (non-test) returns zero hits, and multi-replica-resume-staleness.test.ts says the same in its header. Its onMetadataMutation listener (plugin.tssubscribeFlowMutations) reacts to type === 'flow' only.
So any fix adds a propagation path. The dispatch order says that choice goes to a human.
Options
Option
Work
Surface
A
A new address-only lane (for example connector-credential.mutated, payload { originNode, ref }), published by the credential door after store.set / store.clear. Peers drop their own originNode, then run reconcileDeclaredConnectors on their reconcile queue, which re-reads the shared row (revision then fingerprint). This is lane 3's shape exactly: the admin service publishes on the write doors it owns.
M: about 150 to 250 lines plus a two-kernel pin over one store and one shared pub/sub fake, and a §6.2 table row
service-automation, and either a fourth duck-typed lane in MetadataClusterBridgePlugin (packages/services/service-cluster, outside the claimed surface) or a self-attach at kernel:ready inside service-automation
B
Ride lane 2: open a public emit on the metadata protocol and announce a synthetic connector/NAME mutation.
S to M
packages/metadata-protocol (outside the surface). It breaks the lane's contract that the payload is the address of a sys_metadata row, and makes peers re-run a registry write-through for nothing.
C
Held and missing instances poll their row's revision on a short timer (the card body's second suggestion).
S to M
service-automation only. This is a second mechanism, which the ruling forbids ("no second propagation channel"). It costs a periodic read per held instance per replica, but it also covers the lost messages that at-most-once lanes accept.
D
No runtime change. Correct the multi-replica claim in the door's documentation: other replicas converge on their next reload or restart.
Long-term soundness (≥50%): A follows the §6.2 family contract word for word (address-only signal, receipt runs convergence, originNode loopback, at-most-once bounded by boot rehydration). B bends lane 2's contract, so it would be a workaround. C adds a polling idiom that no other state owner uses.
AI-error resistance: A and C are both structural and need no new authorable key. B leaves future readers with a "metadata mutation" that matches no metadata row.
No scope spread: A adds one channel, with no spec or metadata surface. C adds no channel, but it adds a timer. D adds nothing.
Recommendation: A. One signal, emitted by the state owner on the write door, following the existing family; it's the closest available reading of the ruling's intent. Two sub-choices the ruling should settle at the same time: (1) where the lane attaches (the bridge plugin in service-cluster, consistent with lanes 1 to 3, versus a self-attach inside service-automation, which stays inside the claimed surface), and (2) whether the same lane should also cover DELETE (it should; it's the same address).
Observation (not measured, not filed)
A peer replica also never receives metadata:reloaded for a publish made on another replica, and service-automation's onMetadataMutation listener ignores connector items. If that is right, a declarative connector added or changed by a publish materializes only on the publishing replica. This was read from the code only, so it is recorded here and not filed. If A is chosen, it is the obvious sibling question for the same lane.
{
"issue": 27428,
"status": "needs_decision",
"branch": "claude/project-thread-iw0du9",
"pr": null,
"session": "session_016WV4cw3bHrCG7jTpB8emLs",
"premise_still_valid": false,
"summary": "The defect premise stands (held instances have no timer, and the signature moves only on metadata:reloaded or restart), but the ruling's mechanism premise is false: metadata:reloaded is a process-local kernel hook (core kernel.ts ctx.trigger dispatches to this kernel's own hook map), the three cluster lanes in cluster.mdx section 6.2 carry no credential row, and service-automation has zero cluster or pub/sub wiring. Any fix adds a propagation path, so I stopped before code, per the dispatch order. No branch push and no PR.",
"tests": "none run: stopped at the premise check, before the pin (the order says to stop and report if a channel would have to be added)",
"gates": "none run: no diff",
"dogfood_battery": "n/a: no PR",
"mcp_calls": "0",
"api_writes": "1: this comment, via the fleet-write relay comment op",
"open_questions": [
{
"question": "No existing cluster-wide event reaches other replicas' connector reconcile. Which propagation path should a credential write use?",
"options": [
"A: a new address-only cluster lane published by the credential door (set and clear), received by peers as a reconcileDeclaredConnectors run that re-reads the shared row; it follows the section 6.2 family. Sub-choice: attach in MetadataClusterBridgePlugin (service-cluster) or self-attach in service-automation.",
"B: a public emit on the metadata protocol and a synthetic connector mutation on metadata.mutated; it bends lane 2's sys_metadata-row contract.",
"C: held and missing instances poll their row revision on a short timer; no channel, but this is the second mechanism the ruling forbids.",
"D: documentation only; the defect stays."
],
"recommendation": "A, because it is one signal from the state owner on its own write door, in exactly the shape lanes 2 and 3 already use, with no spec or metadata surface; it needs the ruling to name the attach point and to widen the file surface if that is service-cluster."
}
],
"out_of_scope_findings": ["carrier: none · noted, not filed: a peer replica receives no metadata:reloaded for a publish on another replica, and onMetadataMutation in service-automation ignores connector items, so a published connector change may materialize only on the publishing replica (read from code only, unmeasured)"]
}
Release: claim 6508617582 (session_016WV4cw3bHrCG7jTpB8emLs, branch claude/project-thread-iw0du9, nothing pushed) is released by the seat; the card moves pm:dispatched → needs-user-decision and leaves the active queue. Resume-from: none (no commit). Seat domain:server#3 re-claims it once ruled.
Governing text: content/docs/kernel/cluster.mdx §6.2(集群通道族:仅地址信号、收方自行收敛、originNode 回环过滤、至多一次由启动重建兜底);分诊裁决 6490140325「One signal … no second propagation channel」。
Director seat, summon #54 continuation 1 (ledger objectstack#12708, opening mark 6493929869). Presented to the maintainer in batch #429 as item 2 of 3 — server seat 3's decision request 6508800226 on dev report 6508765478 (needs_decision, no code).
Seat increment (re-read at origin/mainded67bfb, 2026-10-30T01:53Z):
Premises re-run, all hold: content/docs/kernel/cluster.mdx §6.2 (L326) lists exactly three lanes — metadata.changed, metadata.mutated, datasource.mutated (L335–337) — none carrying a sys_connector_credential row; MetadataClusterBridgePlugin lives in packages/services/service-cluster; service-automation/src (non-test) has 0 hits for pubsub|IPubSub|getService('cluster'); the kernel's ctx.trigger dispatches metadata:reloaded to its own hook map only. So triage's mechanism premise (6490140325: "the write emits the event the other replicas already listen to") has no carrier on main; its principle (one signal, no second propagation channel) stands and is what option A implements.
Prior rulings: triage 6490140325 (principle); ADR-0143 D3 (secrets are references; tokens live in a connection record — the row a peer re-reads). check-prior-rulings.mjs --card 27428 names ADR-0094 D2 / D3, ADR-0097 §4 / §6, ADR-0023 §3 — term matches, none decides a cluster lane for credential rows.
Facets: agree with the seat on all four. ① A is the §6.2 family contract applied to one more state owner (address-only payload, receiver converges by re-reading the shared row, originNode loopback filter, at-most-once bounded by boot rehydration); B bends lane 2's "payload is a sys_metadata row address"; C adds a polling idiom no other state owner uses; D leaves a hosted multi-replica deployment's corrected credential effective on one replica. ② the door (connectors: sys_connector_credential + open-tier resolver chain (env → credential object → vault), reconcile on credential change, install-local prompt for missing credentialRefs (ruling A on #27230) #27237) exists for hosted multi-replica environments; every such deployment writes credentials. ③ A and C are structural; B leaves a "metadata mutation" that matches no metadata row. ④ A adds one internal lane, no spec or metadata surface.
Seat's letter: A, attach A1 — a fourth duck-typed lane in MetadataClusterBridgePlugin (service-cluster), beside lanes 1–3, published by the credential door after store.set and store.clear (DELETE included: same address), received as a reconcileDeclaredConnectors run that re-reads the shared row; the claim's file surface widens to packages/services/service-cluster/src/** and the §6.2 table row in content/docs/kernel/cluster.mdx. A1 over A2 because the one place that enumerates cluster lanes stays one place; a self-attach inside service-automation would be the first lane that does not appear there. Not B, not C, not D.
Workload of A1: domain:server · 1 PR · M (about 150–250 lines + a two-kernel pin over one store with one shared pub/sub fake + one docs row) · Clause-②: no, Metadata-protocol: none · precedent: lane 3 (datasource.mutated, the admin service publishing on its own write door). The dev's unmeasured observation (a published connector change may materialise only on the publishing replica) is the sibling question for the same lane, measured after A lands — not in this card.
After the ruling: server seat 3 (os-elon-musk) re-claims from Resume-from: none with Ruling-ref: and the widened surface; needs-user-decision → pm:dispatched is that seat's stroke.
Ruling: batch #429 item 2 · letter A (attach A1) · maintainer 「其他同意」 2026-10-30T02:13Z
Director seat, summon #54 continuation 1 (ledger objectstack#12708, opening mark 6493929869, brief 6513006937). The maintainer answered batch #429 in the summon chat, verbatim: 「27681 多维度综合分析,给我建议。其他同意」 — 同意 = the seat's letter on this card: A, attached at A1.
What the ruling binds (letter A, A1): a credential write through PUT|DELETE /api/v1/automation/connectors/credential-refs/:ref propagates cluster-wide through one new address-only lane of the §6.2 cluster-invalidation family (content/docs/kernel/cluster.mdx), published by the credential door after store.set and after store.clear (DELETE is the same address), payload { originNode, ref }; peers drop their own originNode and run reconcileDeclaredConnectors on their reconcile queue, which re-reads the shared row (revision, then fingerprint). The lane is the fourth duck-typed lane in MetadataClusterBridgePlugin (packages/services/service-cluster), beside metadata.changed, metadata.mutated and datasource.mutated, so the one place that enumerates cluster lanes stays one place; the §6.2 table gains its row. At-most-once, bounded by boot rehydration, as the family states. Not taken: A2 (a self-attach inside service-automation — the first lane absent from the bridge), B (a synthetic metadata.mutated that names no sys_metadata row), C (polling — a second mechanism), D (docs only). Triage's principle in 6490140325 (one signal, no second propagation channel) stands; its mechanism premise (an existing event) was false on main and is replaced by this lane.
Governing text: content/docs/kernel/cluster.mdx §6.2; ADR-0143 D3 (tokens live in a connection record; the row a peer re-reads); triage 6490140325. Premises re-read at origin/mainded67bfb (brief 6513006937).
Workload of the letter:domain:server · 1 PR · M (about 150–250 lines; a two-kernel pin over one store with one shared pub/sub fake; one docs row) · Clause-②: no, Metadata-protocol: none · file surface = claim 6508617582's plus packages/services/service-cluster/src/** and the §6.2 row in content/docs/kernel/cluster.mdx · precedent: lane 3.
Execution:needs-user-decision → pm:queue in this act (no assignee: seat 3 released claim 6508617582); server seat 3 re-claims with Ruling-ref: naming this comment and the widened surface, and opens the card thread from zero (Resume-from: none). The dev's unmeasured observation (a published connector change may materialise only on the publishing replica) is measured after this lane lands, as a sibling question on the same lane, on its own card.
os-decision-facets
① 长远:沿用 §6.2 通道族契约,枚举通道的地方仍只有一处;B 弯曲通道语义,C 引入轮询惯用法
② 拉动:凭据门为托管多副本而建,每个这样的部署写凭据都撞上
③ 防错:仅地址信号、收方重读共享行,结构性、无新可写键
④ 不扩散:一条内部通道,无 spec / 元数据面
Claim: PM loop round 2 · 2026-10-30T03:34Z
Ruling-ref: 6513279565
Resume-from: none
Session: session_016WV4d1Ds3FbhoRdoBLGUvZ (card thread; seat session_016WV4czuTWNgKR22fKv7jia)
Account: os-elon-musk
Branch: claude/project-thread-m511py (the card thread's platform-assigned branch — the recorded platform exception; amended by the seat when the thread reported it)
Worktree: objectstack-issue-27428
Domain: domain:server
Seat: domain:server#3
File surface: packages/services/service-automation/src/** (the credential door's publish after store.set / store.clear, the receiver's reconcileDeclaredConnectors run) and its tests; packages/services/service-cluster/src/** (the fourth duck-typed lane in MetadataClusterBridgePlugin) and its tests; the §6.2 lane row in content/docs/kernel/cluster.mdx; .changeset/* — the ruling's widened surface; ⛔ no packages/spec, no governed path (stop on breach; explain in the report)
Container & model: M, mode:thread, model: opus (TIER_DEFAULT; dispatch-gates --tier: no path-derived mandate), effort: high
Clause-②: no
Metadata-protocol: none
Responsibility: credential writes through PUT|DELETE /api/v1/automation/connectors/credential-refs/:ref on a multi-replica deployment; ruled letter A attached at A1 (one new address-only lane of the §6.2 cluster-invalidation family, payload { originNode, ref }, receiver re-reads the shared row).
Thread-read: 6513279565
Premise: holds (re-read by the director at origin/mainded67bfb, 6513006937)
Serial constraints cleared: draft PR #27819 (#27746) edits service-automation/src/plugin.ts and the credential source; the dev keeps to the door's publish and the receiver, and the later lander merges main. No open PR touches service-cluster or cluster.mdx at claim time.
PR #27941 (draft) at head f78a7f75: ruling 6513279565 (letter A, attached at A1) implemented. CI is running; the seat lands after its review.
{
"issue": 27428,
"status": "done",
"branch": "claude/project-thread-m511py",
"pr": "https://github.com/objectstack-ai/objectstack/pull/27941",
"session": "session_016WV4d1Ds3FbhoRdoBLGUvZ",
"premise_still_valid": true,
"summary": "Ruling 6513279565 (letter A at A1) implemented at head f78a7f75. The credential door publishes { originNode, ref } on connector-credential.mutated after store.set and after a successful store.clear (refused writes publish nothing); peers drop their own originNode and run reconcileDeclaredConnectors on their reconcile queue, re-reading the shared sys_connector_credential rows, so a held instance re-materializes and a cleared ref degrades. The seam is AutomationEngine.attachConnectorCredentialPubSub (delegating to the plugin's credential door), attached as the fourth duck-typed lane in MetadataClusterBridgePlugin, skipped on the in-process memory driver; cluster.mdx section 6.2 gains the lane 4 row and its prose says four lanes. Changeset patch for service-automation and service-cluster.",
"tests": "New connector-credential-cluster.test.ts: two AutomationServicePlugin kernels over one SQLite database and one shared bus (publisher included): PUT on A re-materializes B's held instance with the new value, no reload on B; DELETE on A degrades B naming the ref; control (B unattached) stays held; payload exactly { originNode, ref } with partitionKey connector-credential:REF, none on refused writes; A's own publish dropped on A, B's write reaches A. Red on base 8c7021df: 5/5 (TypeError attachConnectorCredentialPubSub is not a function). Ablations via scripts/ablation-replace.mjs (anchor 1->0, restored blob == HEAD, git diff HEAD empty each time; tests import src, no dist leg): set-door publish removed -> 4 red, control green; clear-door publish removed -> 2 red (DELETE, payload); loopback guard removed -> 1 red (loopback). Six lane-4 cases added to metadata-cluster-bridge-plugin.test.ts. service-automation full suite 219 files 2808/2808; service-cluster 7 files 111/111; typecheck exit 0 for both (all at f78a7f75).",
"gates": "node scripts/pm/dispatch-gates.mjs --commands at f78a7f75: 92 derived families, all run with exit codes captured before any pipe; --ran: '92 derived, 92 run, 0 NOT-MEASURED, 0 UNRUN' (derived zero). 89 exit 0 first pass; 3 exit 3 then 0 after the prerequisite: check-plugin-teardown-shape --self-test (pinned commit fetched into shallow clone), spec check:skill-examples and check:dual-build-cjs-loads (after whole-tree turbo build, 116/116). pnpm lint exit 0. CI check-runs on PR #27941: in_progress at report time.",
"dogfood_battery": "n/a — no ! in the PR title and no (narrowing) arm",
"mcp_calls": "0 — GitHub reads via gh api (session token, read-only); no MCP GitHub tool used",
"api_writes": "2 — relay pr_create (POST pulls, PR #27941, read-back identical 7139 bytes); relay comment (this report on #27428). Plus git push (not REST).",
"open_questions": [
{
"question": "Clause-②: the claim says no and the PR copies it, but AutomationEngine (an exported class) gains one public method, attachConnectorCredentialPubSub, the duck-typed seam the bridge needs; the channel constant and payload type are not exported from the package root. Does a new public method on an exported service class count as an enlarged public surface?",
"options": [
"A keep Clause-②: no (an internal cluster seam, the same shape lane 3 added to DatasourceAdminService)",
"B amend the claim to Clause-②: yes (widening); the seat rewrites the PR body and changeset lines and runs its own review of record"
],
"recommendation": "A, because the method is a cluster wiring seam called only by the bridge, matching lane 3's precedent; the seat decides."
}
],
"out_of_scope_findings": [
"carrier: own card per ruling 6513279565 · the earlier unmeasured observation (a connector changed by a publish may materialise only on the publishing replica) was not measured in this card, as the ruling directs; nothing new seen · noted, not filed"
]
}
Activity
objectstack-fleet commentedon Oct 29, 2026
triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U
Path: 审批与自动化 | none | P⑤
Triage: p2 ·
domain:server·area:automation· 2026-10-29T06:57Z⛔ Not a claim.
sys_connector_credential+ open-tier resolver chain (env → credential object → vault), reconcile on credential change, install-local prompt for missingcredentialRefs (ruling A on #27230) #27237 exists for, so p2.bug,pm:queue.Generated by Claude Code
objectstack-fleet commentedon Oct 29, 2026
domain:server#3 · os-elon-musk · session_016WV4czuTWNgKR22fKv7jia
Claim: PM loop round 1 · 2026-10-29T21:21Z
Session:
session_016WV4cw3bHrCG7jTpB8emLs(card thread; seatsession_016WV4czuTWNgKR22fKv7jia)Account:
os-elon-muskBranch:
claude/project-thread-iw0du9(the card thread's platform-assigned branch — the recorded platform exception; amended by the seat when the thread reported it)Worktree:
objectstack-issue-27428Domain:
domain:serverSeat:
domain:server#3File surface:
packages/services/service-automation/src/**(the credential write path,holdConnectorForRejectedCredential,scheduleDeclarativeRetry, themetadata:reloadedreconcile) and its tests,.changeset/*; ⛔ nopackages/spec, no governed path (stop on breach; explain in the report)Container & model:
M,mode:thread,model: opus (TIER_DEFAULT; dispatch-gates --tier: no path-derived mandate),effort: highClause-②: no
Metadata-protocol: none
Responsibility:
service-automationholds the credential write and the held-instance state; the existing reload/reconcile event is the platform path triage names (6490140325: one signal only, no second propagation channel); reached by any multi-replica deployment that writes a connector credential through the door.Thread-read: 6490140325
Premise: holds (symbols present on
origin/maind93d9be5; cross-replica behaviour to be measured by the dev with two kernels on one store)Serial constraints cleared: none — no open PR or in-flight claim on
service-automation/src/plugin.tsfound at claim time.objectstack-fleet commentedon Oct 29, 2026
domain:server#3 · os-elon-musk · session_016WV4cw3bHrCG7jTpB8emLs
os-dev-report
needs_decision: the triage ruling's premise does not hold on
origin/mainef359b95. No cluster-wide event exists that the credential write could emit and that other replicas already listen to. Per the dispatch order, I stopped before writing code or the pin, and I am reporting the fork rather than building a channel.What the ruling assumed and what the tree says
The ruling says the credential write should emit "the same reload or reconcile event the other replicas already listen to". The only reconcile event service-automation listens to is the
metadata:reloadedkernel hook, and that hook never leaves the process:packages/core/src/kernel.ts:316:ctx.triggerrunsdispatchHookPropagating(name, this.hooks.get(name) ...), which means it only dispatches to this kernel's own hook map. Every announcer ofmetadata:reloadeduses it:packages/metadata/src/plugin.ts:1527(dev artifact reload),packages/objectql/src/plugin.ts:680(per-item publish),packages/runtime/src/domains/packages.ts:2055(publish-drafts), andpackages/cloud-connection/src/marketplace-install-local-plugin.ts:2448. A peer replica does not see a publish made on another replica asmetadata:reloadedeither.content/docs/kernel/cluster.mdx§6.2:metadata.changed(MetadataManager caches),metadata.mutated(the ObjectQL object registry, which replays into peers'onMetadataMutation), anddatasource.mutated(the driver registry), plus the authz invalidation channel. None of them carries asys_connector_credentialrow change. The lane-2 emitteremitMetadataMutationisprivateinpackages/metadata-protocol/src/protocol.ts:7667, and its receipt re-reads asys_metadatarow, which a credential row is not.git grep -n "pubsub|IPubSub|getService('cluster')"overpackages/services/service-automation/src(non-test) returns zero hits, andmulti-replica-resume-staleness.test.tssays the same in its header. ItsonMetadataMutationlistener (plugin.tssubscribeFlowMutations) reacts totype === 'flow'only.So any fix adds a propagation path. The dispatch order says that choice goes to a human.
Options
connector-credential.mutated, payload{ originNode, ref }), published by the credential door afterstore.set/store.clear. Peers drop their ownoriginNode, then runreconcileDeclaredConnectorson their reconcile queue, which re-reads the shared row (revision then fingerprint). This is lane 3's shape exactly: the admin service publishes on the write doors it owns.MetadataClusterBridgePlugin(packages/services/service-cluster, outside the claimed surface) or a self-attach atkernel:readyinside service-automationconnector/NAMEmutation.packages/metadata-protocol(outside the surface). It breaks the lane's contract that the payload is the address of asys_metadatarow, and makes peers re-run a registry write-through for nothing.missinginstances poll their row'srevisionon a short timer (the card body's second suggestion).Four axes
sys_connector_credential+ open-tier resolver chain (env → credential object → vault), reconcile on credential change, install-local prompt for missingcredentialRefs (ruling A on #27230) #27237 card) exists for hosted multi-replica environments, and the card's reach is every such deployment that writes a credential. A, B and C each close the defect. D does not.originNodeloopback, at-most-once bounded by boot rehydration). B bends lane 2's contract, so it would be a workaround. C adds a polling idiom that no other state owner uses.Recommendation: A. One signal, emitted by the state owner on the write door, following the existing family; it's the closest available reading of the ruling's intent. Two sub-choices the ruling should settle at the same time: (1) where the lane attaches (the bridge plugin in service-cluster, consistent with lanes 1 to 3, versus a self-attach inside service-automation, which stays inside the claimed surface), and (2) whether the same lane should also cover
DELETE(it should; it's the same address).Observation (not measured, not filed)
A peer replica also never receives
metadata:reloadedfor a publish made on another replica, and service-automation'sonMetadataMutationlistener ignoresconnectoritems. If that is right, a declarative connector added or changed by a publish materializes only on the publishing replica. This was read from the code only, so it is recorded here and not filed. If A is chosen, it is the obvious sibling question for the same lane.{ "issue": 27428, "status": "needs_decision", "branch": "claude/project-thread-iw0du9", "pr": null, "session": "session_016WV4cw3bHrCG7jTpB8emLs", "premise_still_valid": false, "summary": "The defect premise stands (held instances have no timer, and the signature moves only on metadata:reloaded or restart), but the ruling's mechanism premise is false: metadata:reloaded is a process-local kernel hook (core kernel.ts ctx.trigger dispatches to this kernel's own hook map), the three cluster lanes in cluster.mdx section 6.2 carry no credential row, and service-automation has zero cluster or pub/sub wiring. Any fix adds a propagation path, so I stopped before code, per the dispatch order. No branch push and no PR.", "tests": "none run: stopped at the premise check, before the pin (the order says to stop and report if a channel would have to be added)", "gates": "none run: no diff", "dogfood_battery": "n/a: no PR", "mcp_calls": "0", "api_writes": "1: this comment, via the fleet-write relay comment op", "open_questions": [ { "question": "No existing cluster-wide event reaches other replicas' connector reconcile. Which propagation path should a credential write use?", "options": [ "A: a new address-only cluster lane published by the credential door (set and clear), received by peers as a reconcileDeclaredConnectors run that re-reads the shared row; it follows the section 6.2 family. Sub-choice: attach in MetadataClusterBridgePlugin (service-cluster) or self-attach in service-automation.", "B: a public emit on the metadata protocol and a synthetic connector mutation on metadata.mutated; it bends lane 2's sys_metadata-row contract.", "C: held and missing instances poll their row revision on a short timer; no channel, but this is the second mechanism the ruling forbids.", "D: documentation only; the defect stays." ], "recommendation": "A, because it is one signal from the state owner on its own write door, in exactly the shape lanes 2 and 3 already use, with no spec or metadata surface; it needs the ruling to name the attach point and to widen the file surface if that is service-cluster." } ], "out_of_scope_findings": ["carrier: none · noted, not filed: a peer replica receives no metadata:reloaded for a publish on another replica, and onMetadataMutation in service-automation ignores connector items, so a published connector change may materialize only on the publishing replica (read from code only, unmeasured)"] }Generated by Claude Code
objectstack-fleet commentedon Oct 29, 2026
domain:server#3 · os-elon-musk · session_016WV4czuTWNgKR22fKv7jia
待裁:多副本下凭据写入如何通知其它副本 · 2026-10-29T21:30Z
Release: claim 6508617582 (
session_016WV4cw3bHrCG7jTpB8emLs, branchclaude/project-thread-iw0du9, nothing pushed) is released by the seat; the card movespm:dispatched→needs-user-decisionand leaves the active queue. Resume-from: none (no commit). Seatdomain:server#3re-claims it once ruled.一句话问题: 多副本部署里,管理员在某一副本上写入连接器凭据,只有那一个副本恢复;其它副本上被挂起的连接器一直不可用,直到下次元数据重载或重启。
前提已被证伪(dev 报告 6508765478,
origin/mainef359b95): 分诊裁决假定「写入发出一个其它副本已在监听的重载/对账事件」。实测没有这样的事件:metadata:reloaded是进程内 kernel hook(packages/core/src/kernel.tsctx.trigger只派发本 kernel 的 hook);content/docs/kernel/cluster.mdx§6.2 的三条集群通道都不承载凭据行;service-automation零 pub/sub 接线。任何修法都要新增传播路径,按规则交人工裁。Governing text:
content/docs/kernel/cluster.mdx§6.2(集群通道族:仅地址信号、收方自行收敛、originNode回环过滤、至多一次由启动重建兜底);分诊裁决 6490140325「One signal … no second propagation channel」。connector-credential.mutated),凭据门在 set/clear 后发布;其它副本收到后跑既有的reconcileDeclaredConnectors重读共享行metadata.mutated,公开 metadata-protocol 的 emit,发一条伪造的 connector 变更packages/metadata-protocolrevision业务含义直译:A = 给集群加一根「凭据已改」的广播线,做法与现有三根一致;B = 借用「元数据已改」的广播线冒充;C = 让每个副本隔一会自己去查;D = 告诉客户这是已知限制。
os-decision-facets
sys_connector_credential+ open-tier resolver chain (env → credential object → vault), reconcile on credential change, install-local prompt for missingcredentialRefs (ruling A on #27230) #27237 的凭据门就是为托管多副本部署做的,今天每个这样的部署写凭据都会撞上;A/B/C 都能修,D 不修。推荐 A。 只看①选 A;②③④ 是否翻转:否。置信缺口:通道挂载点二选一需同时裁定 —— (A1) 挂在
packages/services/service-cluster的MetadataClusterBridgePlugin作第四条鸭子类型通道(与前三条一致,文件面扩到 service-cluster);(A2) 在 service-automation 内kernel:ready自挂(不出认领文件面)。席位倾向 A1(与现有三条同址)。同一通道一并覆盖 clear/DELETE。裁后执行: 本席重新认领,按所选字母与挂载点开卡线程施工(M,默认档,effort high);
Clause-②: no、Metadata-protocol: none,预计不欠契约复审。dev 另记一条未测观察(发布导致的 connector 变更可能只在发布副本生效),待 A 落地后作为同通道的兄弟问题再测,不在本卡。objectstack-fleet commentedon Oct 30, 2026
director#1 · os-bill · session_01Utp6CbQLcy8yg6vhJVXXi8
Brief-claimed-by: session_01Utp6CbQLcy8yg6vhJVXXi8 · batch #429 · 2026-10-30T01:53Z
Director seat, summon #54 continuation 1 (ledger objectstack#12708, opening mark 6493929869). Presented to the maintainer in batch #429 as item 2 of 3 — server seat 3's decision request 6508800226 on dev report 6508765478 (
needs_decision, no code).Seat increment (re-read at
origin/mainded67bfb, 2026-10-30T01:53Z):content/docs/kernel/cluster.mdx§6.2 (L326) lists exactly three lanes —metadata.changed,metadata.mutated,datasource.mutated(L335–337) — none carrying asys_connector_credentialrow;MetadataClusterBridgePluginlives inpackages/services/service-cluster;service-automation/src(non-test) has 0 hits forpubsub|IPubSub|getService('cluster'); the kernel'sctx.triggerdispatchesmetadata:reloadedto its own hook map only. So triage's mechanism premise (6490140325: "the write emits the event the other replicas already listen to") has no carrier onmain; its principle (one signal, no second propagation channel) stands and is what option A implements.check-prior-rulings.mjs --card 27428names ADR-0094 D2 / D3, ADR-0097 §4 / §6, ADR-0023 §3 — term matches, none decides a cluster lane for credential rows.originNodeloopback filter, at-most-once bounded by boot rehydration); B bends lane 2's "payload is asys_metadatarow address"; C adds a polling idiom no other state owner uses; D leaves a hosted multi-replica deployment's corrected credential effective on one replica. ② the door (connectors:sys_connector_credential+ open-tier resolver chain (env → credential object → vault), reconcile on credential change, install-local prompt for missingcredentialRefs (ruling A on #27230) #27237) exists for hosted multi-replica environments; every such deployment writes credentials. ③ A and C are structural; B leaves a "metadata mutation" that matches no metadata row. ④ A adds one internal lane, no spec or metadata surface.MetadataClusterBridgePlugin(service-cluster), beside lanes 1–3, published by the credential door afterstore.setandstore.clear(DELETE included: same address), received as areconcileDeclaredConnectorsrun that re-reads the shared row; the claim's file surface widens topackages/services/service-cluster/src/**and the §6.2 table row incontent/docs/kernel/cluster.mdx. A1 over A2 because the one place that enumerates cluster lanes stays one place; a self-attach inside service-automation would be the first lane that does not appear there. Not B, not C, not D.domain:server· 1 PR · M (about 150–250 lines + a two-kernel pin over one store with one shared pub/sub fake + one docs row) ·Clause-②: no,Metadata-protocol: none· precedent: lane 3 (datasource.mutated, the admin service publishing on its own write door). The dev's unmeasured observation (a published connector change may materialise only on the publishing replica) is the sibling question for the same lane, measured after A lands — not in this card.os-elon-musk) re-claims fromResume-from: nonewithRuling-ref:and the widened surface;needs-user-decision→pm:dispatchedis that seat's stroke.Prior rulings read: cluster lane, metadata.mutated, connector credential, replica, reconcile → triage 6490140325, ADR-0143 D3; thread: 6508765478, 6508800226; repo: objectstack-ai/objectstack
objectstack-fleet commentedon Oct 30, 2026
director#1 · os-bill · session_01Utp6CbQLcy8yg6vhJVXXi8
Ruling: batch #429 item 2 · letter A (attach A1) · maintainer 「其他同意」 2026-10-30T02:13Z
Director seat, summon #54 continuation 1 (ledger objectstack#12708, opening mark 6493929869, brief 6513006937). The maintainer answered batch #429 in the summon chat, verbatim: 「27681 多维度综合分析,给我建议。其他同意」 — 同意 = the seat's letter on this card: A, attached at A1.
What the ruling binds (letter A, A1): a credential write through
PUT|DELETE /api/v1/automation/connectors/credential-refs/:refpropagates cluster-wide through one new address-only lane of the §6.2 cluster-invalidation family (content/docs/kernel/cluster.mdx), published by the credential door afterstore.setand afterstore.clear(DELETE is the same address), payload{ originNode, ref }; peers drop their ownoriginNodeand runreconcileDeclaredConnectorson their reconcile queue, which re-reads the shared row (revision, then fingerprint). The lane is the fourth duck-typed lane inMetadataClusterBridgePlugin(packages/services/service-cluster), besidemetadata.changed,metadata.mutatedanddatasource.mutated, so the one place that enumerates cluster lanes stays one place; the §6.2 table gains its row. At-most-once, bounded by boot rehydration, as the family states. Not taken: A2 (a self-attach inside service-automation — the first lane absent from the bridge), B (a syntheticmetadata.mutatedthat names nosys_metadatarow), C (polling — a second mechanism), D (docs only). Triage's principle in 6490140325 (one signal, no second propagation channel) stands; its mechanism premise (an existing event) was false onmainand is replaced by this lane.Governing text:
content/docs/kernel/cluster.mdx§6.2; ADR-0143 D3 (tokens live in a connection record; the row a peer re-reads); triage 6490140325. Premises re-read atorigin/mainded67bfb(brief 6513006937).Workload of the letter:
domain:server· 1 PR · M (about 150–250 lines; a two-kernel pin over one store with one shared pub/sub fake; one docs row) ·Clause-②: no,Metadata-protocol: none· file surface = claim 6508617582's pluspackages/services/service-cluster/src/**and the §6.2 row incontent/docs/kernel/cluster.mdx· precedent: lane 3.Execution:
needs-user-decision→pm:queuein this act (no assignee: seat 3 released claim 6508617582); server seat 3 re-claims withRuling-ref:naming this comment and the widened surface, and opens the card thread from zero (Resume-from: none). The dev's unmeasured observation (a published connector change may materialise only on the publishing replica) is measured after this lane lands, as a sibling question on the same lane, on its own card.os-decision-facets
① 长远:沿用 §6.2 通道族契约,枚举通道的地方仍只有一处;B 弯曲通道语义,C 引入轮询惯用法
② 拉动:凭据门为托管多副本而建,每个这样的部署写凭据都撞上
③ 防错:仅地址信号、收方重读共享行,结构性、无新可写键
④ 不扩散:一条内部通道,无 spec / 元数据面
objectstack-fleet commentedon Oct 30, 2026
domain:server#3 · os-elon-musk · session_016WV4czuTWNgKR22fKv7jia
Claim: PM loop round 2 · 2026-10-30T03:34Z
Ruling-ref: 6513279565
Resume-from: none
Session:
session_016WV4d1Ds3FbhoRdoBLGUvZ(card thread; seatsession_016WV4czuTWNgKR22fKv7jia)Account:
os-elon-muskBranch:
claude/project-thread-m511py(the card thread's platform-assigned branch — the recorded platform exception; amended by the seat when the thread reported it)Worktree:
objectstack-issue-27428Domain:
domain:serverSeat:
domain:server#3File surface:
packages/services/service-automation/src/**(the credential door's publish afterstore.set/store.clear, the receiver'sreconcileDeclaredConnectorsrun) and its tests;packages/services/service-cluster/src/**(the fourth duck-typed lane inMetadataClusterBridgePlugin) and its tests; the §6.2 lane row incontent/docs/kernel/cluster.mdx;.changeset/*— the ruling's widened surface; ⛔ nopackages/spec, no governed path (stop on breach; explain in the report)Container & model:
M,mode:thread,model: opus (TIER_DEFAULT; dispatch-gates --tier: no path-derived mandate),effort: highClause-②: no
Metadata-protocol: none
Responsibility: credential writes through
PUT|DELETE /api/v1/automation/connectors/credential-refs/:refon a multi-replica deployment; ruled letter A attached at A1 (one new address-only lane of the §6.2 cluster-invalidation family, payload{ originNode, ref }, receiver re-reads the shared row).Thread-read: 6513279565
Premise: holds (re-read by the director at
origin/mainded67bfb, 6513006937)Serial constraints cleared: draft PR #27819 (#27746) edits
service-automation/src/plugin.tsand the credential source; the dev keeps to the door's publish and the receiver, and the later lander mergesmain. No open PR touchesservice-clusterorcluster.mdxat claim time.objectstack-fleet commentedon Oct 30, 2026
domain:server#3 · os-elon-musk · session_016WV4d1Ds3FbhoRdoBLGUvZ
os-dev-report
PR #27941 (draft) at head
f78a7f75: ruling 6513279565 (letter A, attached at A1) implemented. CI is running; the seat lands after its review.{ "issue": 27428, "status": "done", "branch": "claude/project-thread-m511py", "pr": "https://github.com/objectstack-ai/objectstack/pull/27941", "session": "session_016WV4d1Ds3FbhoRdoBLGUvZ", "premise_still_valid": true, "summary": "Ruling 6513279565 (letter A at A1) implemented at head f78a7f75. The credential door publishes { originNode, ref } on connector-credential.mutated after store.set and after a successful store.clear (refused writes publish nothing); peers drop their own originNode and run reconcileDeclaredConnectors on their reconcile queue, re-reading the shared sys_connector_credential rows, so a held instance re-materializes and a cleared ref degrades. The seam is AutomationEngine.attachConnectorCredentialPubSub (delegating to the plugin's credential door), attached as the fourth duck-typed lane in MetadataClusterBridgePlugin, skipped on the in-process memory driver; cluster.mdx section 6.2 gains the lane 4 row and its prose says four lanes. Changeset patch for service-automation and service-cluster.", "tests": "New connector-credential-cluster.test.ts: two AutomationServicePlugin kernels over one SQLite database and one shared bus (publisher included): PUT on A re-materializes B's held instance with the new value, no reload on B; DELETE on A degrades B naming the ref; control (B unattached) stays held; payload exactly { originNode, ref } with partitionKey connector-credential:REF, none on refused writes; A's own publish dropped on A, B's write reaches A. Red on base 8c7021df: 5/5 (TypeError attachConnectorCredentialPubSub is not a function). Ablations via scripts/ablation-replace.mjs (anchor 1->0, restored blob == HEAD, git diff HEAD empty each time; tests import src, no dist leg): set-door publish removed -> 4 red, control green; clear-door publish removed -> 2 red (DELETE, payload); loopback guard removed -> 1 red (loopback). Six lane-4 cases added to metadata-cluster-bridge-plugin.test.ts. service-automation full suite 219 files 2808/2808; service-cluster 7 files 111/111; typecheck exit 0 for both (all at f78a7f75).", "gates": "node scripts/pm/dispatch-gates.mjs --commands at f78a7f75: 92 derived families, all run with exit codes captured before any pipe; --ran: '92 derived, 92 run, 0 NOT-MEASURED, 0 UNRUN' (derived zero). 89 exit 0 first pass; 3 exit 3 then 0 after the prerequisite: check-plugin-teardown-shape --self-test (pinned commit fetched into shallow clone), spec check:skill-examples and check:dual-build-cjs-loads (after whole-tree turbo build, 116/116). pnpm lint exit 0. CI check-runs on PR #27941: in_progress at report time.", "dogfood_battery": "n/a — no ! in the PR title and no (narrowing) arm", "mcp_calls": "0 — GitHub reads via gh api (session token, read-only); no MCP GitHub tool used", "api_writes": "2 — relay pr_create (POST pulls, PR #27941, read-back identical 7139 bytes); relay comment (this report on #27428). Plus git push (not REST).", "open_questions": [ { "question": "Clause-②: the claim says no and the PR copies it, but AutomationEngine (an exported class) gains one public method, attachConnectorCredentialPubSub, the duck-typed seam the bridge needs; the channel constant and payload type are not exported from the package root. Does a new public method on an exported service class count as an enlarged public surface?", "options": [ "A keep Clause-②: no (an internal cluster seam, the same shape lane 3 added to DatasourceAdminService)", "B amend the claim to Clause-②: yes (widening); the seat rewrites the PR body and changeset lines and runs its own review of record" ], "recommendation": "A, because the method is a cluster wiring seam called only by the bridge, matching lane 3's precedent; the seat decides." } ], "out_of_scope_findings": [ "carrier: own card per ruling 6513279565 · the earlier unmeasured observation (a connector changed by a publish may materialise only on the publishing replica) was not measured in this card, as the ruling directs; nothing new seen · noted, not filed" ] }objectstack-fleet commentedon Oct 30, 2026
domain:server#3 · os-elon-musk · session_016WV4czuTWNgKR22fKv7jia
ACCEPT → PR #27941 comment 6515438761 · 2026-10-30T05:06Z · ready + auto-merge armed through the relay.