Skip to content

fix(#1464): WM→SWM promote e2e flake — dedicated-CG isolation + un-masking#1483

Merged
Jurij89 merged 2 commits into
mainfrom
fix/1464-wm-swm-promote-migration
Jul 7, 2026
Merged

fix(#1464): WM→SWM promote e2e flake — dedicated-CG isolation + un-masking#1483
Jurij89 merged 2 commits into
mainfrom
fix/1464-wm-swm-promote-migration

Conversation

@Jurij89

@Jurij89 Jurij89 commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

What & why

Fixes #1464 — the node-UI devnet e2e (wm-swm-vm-ui-cycle) intermittently failed at the WM→SWM promote step: the UI reported a promote error and the root "never landed in Shared Working Memory." It read like a product migration gap, so the issue guessed at a storage/timeout flake.

It wasn't a timeout. The real cause is cross-test contamination on a shared devnet context graph:

  • The spec drives the UI's promote-all action, which iterates every WM assertion currently on the CG.
  • Sibling specs share the same context graph (cgs[0]) and leave behind assertions that carry RDF named-graph quads.
  • assertionPromote correctly rejects those (KA_NAMED_GRAPH_SHARE_UNSUPPORTED) — so whenever a foreign assertion was present, promote-all threw.
  • Whether the foreign assertion was there depended on test ordering → intermittent red.

The failure was hard to diagnose because it was masked: success and error both rendered the same layer-action-result testid, so a thrown promote could read as success/ambiguous rather than a named throw. We shipped the un-masking + diagnostics first; on its first CI run it named the real error (KA_NAMED_GRAPH_SHARE_UNSUPPORTED, not a timeout), which killed the storage-timeout hypothesis outright.

This is not a test-only PR. Alongside the spec fix it hardens two real product behaviours (UI result/error masking, and operator-facing promote diagnostics). Details below.

Changes

The fix (root cause) — test isolation

  • wm-swm-vm-ui-cycle.devnet.spec.ts: provision a dedicated, unique-per-run context graph in beforeAll, so promote-all only ever sees this fixture's own clean assertion. This structurally removes the contamination (a fresh unique CG cannot hold a foreign named-graph assertion). Also tightened the spec — positive-success assertion, count-0 disambiguation, pruned dead selectors.

Product hardening #1 — un-mask the UI result (this is what let the failure hide)

  • layer-widgets.tsx: split the result testid so a failed layer action renders under layer-action-error, never layer-action-result. A thrown promote can no longer masquerade as success.
  • layer-widgets-publish.test.ts: regression test asserting a promote failure surfaces under layer-action-error.

Product hardening #2 — name the failing promote step for operators

  • promote-step-tag.ts (new) + dkg-publisher.ts: each pre-insert promote read is wrapped so a rejection is re-surfaced as [promote:<step>] <original message> — the operator banner now says which step failed. Identity-preserving: original stack / name / code survive, so nothing downstream that keys on those is affected.
  • async-promote-worker.ts: strips the [promote:<step>] tag before retry-classification, so the diagnostic label can never change a retry decision. Regression tests on both sides.

Verification

  • Re-ran the node-UI devnet e2e lane 5× on CI → 5/5 green. Because the fix removes the cause structurally, this is deterministic, not luck.
  • Full CI: 36 checks pass (remaining are skips).
  • The devnet e2e lane cannot run on Windows, so CI is the runtime test surface here.

Not in this PR (reverted)

An earlier revision (PR2) added a per-call SPARQL read timeout to "widen" promote reads, on the assumption a 30s abort was the cause. The diagnostic disproved that, so it was fully reverted (preserved as tag backup/1464-pr2-timeout). The underlying O(store-size) assertionScopedQuads enumeration it targeted is a genuine latent perf concern and is tracked as a separate follow-up — it is not the cause of this flake.

… behavior change)

The devnet e2e `wm-swm-vm-ui-cycle … triples actually migrate` fails
intermittently reading `countRootInSharedMemory > 0` as 0. Root cause (per a
multi-agent analysis + review): the promote is fully synchronous, so a real
`200 {promotedCount>0}` durably commits the root to the count-visible SWM graph
— every count-0 run is a run where a pre-insert op THREW (a SPARQL read / the
finalize seal crossing the 30s timeout under CI-devnet load). But the failure is
MASKED: success and error both rendered into `data-testid="layer-action-result"`,
and the spec's only guard rejected the literal "No triples were promoted" string
— so an "✕ Promote failed" banner read as success and the redness surfaced later
as a bare count-0, misdiagnosed as a silent migration gap. A longer timeout
never helped because it's a throw, not slowness.

This is a DIAGNOSTIC PR — it makes the failure fail LOUDLY with a NAMED op; it
does not add retry/timeout (that's the follow-up, targeting whatever op CI
names). Changes:

- node-ui: errors render under a DISTINCT `data-testid="layer-action-error"` at
  both collision sites (LayerActionShell + LayerWidgetStrip); fixed the one unit
  test that asserted an error under the success testid; added a masking-regression
  test (proven red before the split).
- publisher: wrap each pre-insert promote op in `tagPromoteStep(step, …)` so a
  throw surfaces `[promote:<step>] <original>` (identity-preserving: keeps
  stack/name/code/instanceof; idempotent; no retry/swallow/reorder; writes +
  store.insert untagged) + `promote-step-tag.test.ts`. Verified `resolveKaNumber`
  is non-null for a finalized fixture ⇒ the bare-bucket path is not this flake.
- cli: `classifyPromoteError` now strips a leading `[promote:<step>]` tag before
  substring-classifying, so a step LABEL can never inject a classifier trigger
  token (the "encodeWorkspaceGossipPayload" label would otherwise flip a
  transient error to non-retryable `cap_exceeded`) + regression test.
- e2e spec: pruned dead selectors; replaced the weak "not-no-op" guard with a
  POSITIVE success assertion that surfaces the named error on failure; added
  mandatory count-0 disambiguation (WM-drain + marker → classifies pre-insert
  throw vs committed-but-unenumerated).

Diagnostic-only; no promote/seal semantics changed. Adversarial gate GO (one
blocker — the classifier label collision — caught + fixed). Plan +
analysis: agent-docs/1464-wm-swm-promote-plan.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M3swmeQGT8WGttaZsp1NQJ
Comment thread packages/publisher/src/promote-step-tag.ts
Comment thread packages/publisher/src/dkg-publisher.ts
Comment thread packages/publisher/src/dkg-publisher.ts
Comment thread packages/storage/src/adapters/sparql-http.ts Outdated
Comment thread packages/publisher/test/promote-read-timeout.test.ts Outdated
@Jurij89 Jurij89 force-pushed the fix/1464-wm-swm-promote-migration branch from f1ea2a6 to c659fd7 Compare July 6, 2026 16:29
…aph (the real fix)

PR1's diagnostic named the real cause on the first CI run — NOT a timeout: the
promote threw KA_NAMED_GRAPH_SHARE_UNSUPPORTED. Root cause (verified): the spec
ran on the SHARED seeded CG (`cgs[0]`), and the UI's bulk promote-all iterates
EVERY WM assertion in the CG and aborts the whole batch on any throw. Sibling
specs (cg-variants, the generic quads-create route) leave foreign `_named_graph`
WM assertions in that shared CG, which the promote guard correctly rejects — so
the fixture's own clean root failed to migrate INTERMITTENTLY, depending on test
ordering. The fixture's own Markdown import is default-graph-only (clean).

Fix (spec-only; no product change): provision a DEDICATED, unique-per-run context
graph in beforeAll (`POST /api/context-graph/create`, local/no on-chain register)
and run the import + promote-all there, so promote-all sees ONLY this spec's own
assertion. This is exactly what the spec's own fixme note already called for. A
bounded poll on `listContextGraphs` fails fast (clear message) if the new CG
isn't visible, instead of a mystery openProjectTab timeout. accessPolicy:0 mirrors
the seeded CGs so the caller-scoped listing can't filter it out. No cleanup route
exists for CG definitions; the unique-per-run CG can't contaminate the shared CG
and CI tears the devnet down after the run.

The timeout hypothesis (reverted PR2, tag backup/1464-pr2-timeout) was a red
herring the mask had hidden; PR1's un-mask + this isolation are the fix.
Verified: playwright --list loads the 3 tests. Devnet-lane green pending.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M3swmeQGT8WGttaZsp1NQJ
Comment thread packages/node-ui/e2e/specs/devnet/wm-swm-vm-ui-cycle.devnet.spec.ts
@Jurij89 Jurij89 changed the title fix(#1464): un-mask + name the intermittent WM→SWM promote failure (diagnostic) fix(#1464): WM→SWM promote e2e flake — dedicated-CG isolation + un-masking Jul 6, 2026
@Jurij89 Jurij89 merged commit f1cc2ae into main Jul 7, 2026
165 checks passed
Bojan131 added a commit that referenced this pull request Jul 7, 2026
… lane (#1483)

The UI-driven WM→SWM→VM cycle spec was trimmed to devnet-local/ because of
its shared-CG promote flake — #1483 (merged) fixed that flake at the root
(dedicated-CG isolation + un-masking) and shipped the fixed spec for the CI
lane on main. Honor that: the spec (with #1483's fix, folded in verbatim by
the merge) moves back to specs/devnet/ so main's coverage is not discarded.
The other three heavy specs (on-chain KA mint/update attestation, clearAfter
wipes, 30s soak) stay local-only in devnet-local/ as before.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Intermittent WM→SWM promote migration fails to land the root in shared memory (blocks node-ui e2e on main)

2 participants