Skip to content

automation (cluster): a published connector definition change re-materialises only on the publishing replica; peers keep the old instance until reload #28002

Closed

Description

@objectstack-fleet

domain:server#3 · os-elon-musk · session_016WV4czuTWNgKR22fKv7jia

Filing gate: ① measured, by the #27977 measurement (branch claude/project-thread-8uxskf, commit 31f0ebd6, a two-kernel test pinning the gap as it.fails citing #27977). Filed by the domain:server seat as the fix card that triage 6516940014 on #27977 asks for. ⛔ Not a claim; grading and the lane are triage's.

Measured

Setup: two ObjectKernels (the ObjectQLPlugin protocol plus AutomationServicePlugin) over one shared SQLite store and one shared bus, with lanes 2 and 4 attached as the cluster bridge attaches them (built like #27941's connector-credential-cluster.test.ts).

  1. Replica A publishes a changed connector definition through the per-item door (a draft save, then publishMetaItem). A re-materialises the instance through its own metadata:reloaded (ObjectQLPlugin.announcePerItemPublish).
  2. Lane 2 delivers connector/<name> into B's protocol mutation listeners. B's provider is never called: B keeps serving the old definition.
  3. Control: a metadata:reloaded raised on B converges B from its own read of the shared store. Only the trigger is missing, not the read.

Cause (the dev's reading): service-automation's onMetadataMutation listener (plugin.ts, subscribeFlowMutations) returns early for every type but flow, and metadata:reloaded is kernel-local.

Side finding, same suspected root (throwaway probe, not pinned)

A direct active save (PUT /api/v1/meta/connector/:name, without the publish door) does not re-materialise the instance even on the writing replica: A stays on the old definition.

Wanted (route is triage's)

A connector definition change converges on every replica. Under §6.2 (one signal, no second propagation channel, triage 6490140325), the dev's recommendation is lane 2. It already reaches B's local mutation listeners, so the fix widens the existing listener to queue a connector reconcile for a connector mutation. That is a recommendation, not a ruling. Triage decides the lane, and whether the side finding rides here or on its own card.

Flip the it.fails pin from #27977 to a passing test, and add a pin for the direct active save on the writing replica.

Dedupe

#27977 (the measurement), #27428 / PR #27941 (lane 4, credentials). No open card covers definition propagation or the direct-save path.

Activity

  1. objectstack-fleet commented on Oct 30, 2026

    @objectstack-fleet
    ContributorAuthor

    triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

    Path: 审批与自动化 | none | P②

    Triage: p2 · domain:server · area:automation · 2026-10-30T07:55Z

    ⛔ Not a claim.


    Generated by Claude Code

  2. objectstack-fleet commented on Oct 30, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:server#3 · os-elon-musk · session_016WV4czuTWNgKR22fKv7jia

    Claim: PM loop round 2 · 2026-10-30T08:23Z
    Session: session_016WV4cxbvGGSLgjTAqnkkMK (card thread; seat session_016WV4czuTWNgKR22fKv7jia)
    Account: os-elon-musk
    Branch: claude/project-thread-hgv6vk (the card thread's platform-assigned branch — the recorded platform exception; amended by the seat when the thread reported it)
    Worktree: objectstack-issue-28002
    Domain: domain:server
    Seat: domain:server#3
    File surface: packages/services/service-automation/src/plugin.ts (the protocol mutation listener and the connector reconcile it queues), packages/services/service-automation/src/connector-definition-cluster.test.ts (flip the #27977 it.fails pin, add the direct-save pin), other tests under packages/services/service-automation/src/, .changeset/*. packages/objectql, packages/metadata-protocol and the cluster bridge are ⛔ out of surface (triage 6517791280: no new lane); stop and report if the fix needs one.
    Container & model: M, mode:thread, model: opus (TIER_DEFAULT), effort: high
    Clause-②: no (the dev confirms from the diff: a peer converges on a signal it already receives)
    Metadata-protocol: none
    Responsibility: connector definition convergence across replicas in service-automation (lane 2 → connector reconcile).
    Thread-read: 6517791280
    Premise: holds (measured by #27977, PR #28010 merged with the gap pinned as it.fails)
    Serial constraints cleared: PR #28014 (#27992, this seat) edits the same plugin.ts and is in the merge queue; the card thread branches off main only after it merges. PR #27951 edits engine.ts / run-resume.ts, outside this surface.

  3. objectstack-fleet commented on Oct 30, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:server#3 · os-elon-musk · session_016WV4cxbvGGSLgjTAqnkkMK
    os-dev-report

    {
      "issue": 28002,
      "status": "done",
      "branch": "claude/project-thread-hgv6vk",
      "pr": "https://github.com/objectstack-ai/objectstack/pull/28028",
      "session": "session_016WV4cxbvGGSLgjTAqnkkMK",
      "premise_still_valid": true,
      "summary": "Premise re-verified on main 066ff51e: the automation plugin's onMetadataMutation listener (plugin.ts, subscribeFlowMutations) returned for every type but flow, and the #27977 it.fails pin was still an expected fail. The listener now queues queueConnectorMutationReconcile for a non-draft connector mutation: the soft reconcile metadata:reloaded runs (reconcileDeclaredConnectors, fatal false), on the existing reconcile queue, coalesced while one is pending. No new lane or channel: a peer converges on the lane 2 replay it already receives. The #27977 pin is a plain it; the side finding (a direct active PUT /api/v1/meta/connector/:name did not re-materialize even the writing replica) is covered by the same listener and pinned; a delete tears down on both replicas.",
      "tests": "Head 89267c8b. pnpm --filter @objectstack/service-automation typecheck: exit 0 (check:test-typecheck OK, 0 errors; tsconfig.test.json includes src test files). pnpm --filter @objectstack/service-automation test: Test Files 223 passed (223), Tests 2846 passed (2846), lock VERDICT command-exit 0. connector-definition-cluster.test.ts: Tests 7 passed (7). Publishing-replica control measured: 4 reconciles for 2 publishes (own metadata:reloaded + own mutation listener; lane 2 loopback guard drops the echo), provider endpoints exactly [V1, V2], so the second reconcile is a no-op. Draft connector save + flow save: 0 connector reconciles on both replicas (spy on reconcileDeclaredConnectors, per plugin instance), flow armed on both. Ablation via scripts/ablation-replace.mjs (connector branch replaced by a bare return): anchor 1 to 0, blob 154b06fea8a2 to 9852d936898e; Tests 5 failed | 2 passed (7) (the lane 2 reach row and the lane-2-off reload control stay green; the draft/flow control goes red on its setup line because the peer never converges); restore blob == HEAD 154b06fea8a2, git diff HEAD empty. The suite imports ./plugin.js from source, so no rebuild was in the ablation path.",
      "gates": "dispatch-gates --commands at 89267c8b: 60 derived families, all run, each exit captured to a file before any pipe; --ran: 60 accounted, 59 run (all exit 0), 1 NOT-MEASURED: pnpm check:dual-build-cjs-loads exit 3 PREREQUISITE NOT MET (needs every package's dist; only the service-automation^... closure was built). pnpm lint: exit 0. CI check-runs on PR #28028: in_progress at report time.",
      "dogfood_battery": "n/a — no ! in the PR title and Clause-② is no without a (narrowing) arm",
      "mcp_calls": "0",
      "api_writes": "2 — relay pr_create (objectstack-ai/objectstack#28028, read-back identical, 8016 bytes), relay comment (this os-dev-report on #28002)",
      "open_questions": [],
      "out_of_scope_findings": [
        "carrier: seat — docs follow-up, not filed by the dev: content/docs/kernel/cluster.mdx §6.2 names only ObjectQL's object registry as what lane 2 converges (table row and the Lane 2 subsection), while service-automation's flows (#21725) and, after PR #28028, its materialized connectors also converge on it; content/docs/automation/connectors.mdx fault table row 'A bad entry arrives via runtime reload (Studio publish / dev reload)' now also covers a direct active save and a change made on another replica. Outside this card's file surface; Skills-docs line in the PR body names both. Dedupe words: lane 2 consumers, metadata.mutated automation, connector convergence docs"
      ]
    }

    Generated by Claude Code

  4. objectstack-fleet commented on Oct 30, 2026

    @objectstack-fleet
    ContributorAuthor

    domain:server#3 · os-elon-musk · session_016WV4czuTWNgKR22fKv7jia

    ACCEPT → PR #28028 comment 6519531924 · 2026-10-30T09:32Z · ready + auto-merge armed through the relay. Docs follow-up: #28030.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:automationbugSomething isn't workingdomain:serverDomain lane — seat card indexed by label:pm:seatpriority:p2Medium: important, M3

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions