Skip to content

fix(backend): close SSE reconcile race that could permanently drop a user's event triggers#32

Open
LukasHirt wants to merge 1 commit into
mainfrom
fix/sse-reconcile-race
Open

fix(backend): close SSE reconcile race that could permanently drop a user's event triggers#32
LukasHirt wants to merge 1 commit into
mainfrom
fix/sse-reconcile-race

Conversation

@LukasHirt

Copy link
Copy Markdown
Collaborator

Summary

The per-user SSE consumer manager (backend/pkg/sse/manager.go) had two related timing/race bugs that could cause event-triggered workflows to silently never fire uploads:

  1. Dead consumer never retried. consumeForUser inserted its m.active[userID] entry before calling GetAutomation, but never removed it if GetAutomation failed (e.g. the user has an event trigger but hasn't connected automation yet, or their app-password was revoked). Since the entry never left m.active, every later reconcile() pass saw the user as "already active" and never retried the consumer, even after automation was later connected — the consumer was dead forever.

    Fix: consumeForUser now removes its own m.active entry when it exits due to a GetAutomation failure. All the other return paths happen because ctx was cancelled by reconcile's removal logic or by shutdown — those already remove the entry themselves, so consumeForUser must not also delete it there (that would race with a fresher entry a later reconcile pass may have already installed for the same userID). This is guarded with a small per-entry generation id (activeConsumer.id) so self-cleanup only ever deletes the entry it actually owns.

  2. Up-to-30s blind window. reconcile() only ran on a fixed 30s ticker, so a workflow's event trigger being created/enabled, or a user's automation being connected, could miss an upload that happened before the next tick opened that user's SSE stream.

    Fix: added Manager.Kick() — a non-blocking, drop-if-pending signal — that Start's select loop also listens on alongside the ticker. Wired it into the two write paths that can affect the wanted-consumer set: WorkflowsHandler.syncTriggerIndex (workflow create/update, in pkg/service/workflows.go) and automation.Service.Connect (in pkg/automation/automation.go). Both take an optional Kick()-satisfying interface so they stay decoupled from the concrete *sse.Manager type; backend/pkg/command/server.go was reordered to construct the sse.Manager first and wire it into both.

Test plan

TDD: wrote the following tests against the old code first and watched them fail for the right reason, then implemented the fix and watched them pass.

  • TestReconcileRetriesConsumerAfterTransientGetAutomationFailure (new, pkg/sse/manager_test.go) — reproduces bug chore(deps): Bump actions/setup-node from 6.3.0 to 7.0.0 #1: automation missing on first reconcile, connected later, asserts the consumer is retried.
  • TestKickTriggersImmediateReconcile (new, pkg/sse/manager_test.go) — reproduces bug chore(deps): Bump actions/checkout from 6.0.2 to 7.0.0 #2: uses a "warm-up" trigger to prove Start's select loop is actually running (avoiding a scheduling race), then asserts Kick() picks up a newly-added trigger well within the (1-hour, in the test) ticker interval.
  • All existing pkg/sse tests continue to pass, including TestReconcileStartsAndStopsConsumersAsTriggersChange (removal-path coverage).

Ran locally, matching CI's backend job exactly:

  • go build ./...
  • go vet ./...
  • go test ./...
  • go test ./... -race (extra, not in CI but run to validate the new mutex/generation-id logic)
  • go build -tags=e2e ./... / go vet -tags=e2e ./... (constructor signature changes didn't break e2e compilation)

No unrelated code touched — the share-file action node commits on this branch's ancestry were left untouched; this PR only modifies pkg/sse/manager.go (+tests), pkg/service/workflows.go, pkg/automation/automation.go, and pkg/command/server.go wiring.

@LukasHirt
LukasHirt requested a review from a team as a code owner July 24, 2026 17:19
…user's event triggers

Two related bugs in the per-user SSE consumer manager (pkg/sse/manager.go)
combined to cause an up-to-30s or, in the worst case, permanent gap between
a workflow's event trigger becoming active and its uploads actually firing:

1. consumeForUser inserted its m.active[userID] entry *before* calling
   GetAutomation, but never removed it if GetAutomation failed (e.g. the
   user has an event trigger but hasn't connected automation yet, or their
   app-password was revoked). Because the entry never left m.active, every
   later reconcile() pass treated that user as "already active" and never
   retried the consumer — even once automation was connected. Fixed by
   having consumeForUser remove its own entry on that one failure path.
   The other return paths (ctx cancelled by reconcile's removal logic or by
   shutdown) already have their entries removed by the canceller, so
   self-removal there is skipped to avoid racing with a fresher entry a
   later reconcile pass may have already installed for the same userID —
   guarded with a per-entry generation id (activeConsumer.id) so the
   cleanup only ever deletes the entry it actually owns.

2. reconcile() only ran on a fixed 30s ticker, so a workflow's event
   trigger being added/enabled, or a user's automation being connected,
   could miss an upload that happened in the blind window before the next
   tick opened that user's SSE stream. Added Manager.Kick(), a
   non-blocking, drop-if-pending signal that Start's select loop also
   listens on, and wired it into the two write paths that can affect the
   wanted-consumer set: WorkflowsHandler.syncTriggerIndex (workflow
   create/update) and automation.Service.Connect.

Tests added to pkg/sse/manager_test.go under TDD, watched failing against
the old code before the fix:
 - TestReconcileRetriesConsumerAfterTransientGetAutomationFailure
 - TestKickTriggersImmediateReconcile

Full backend suite (go build, go vet, go test ./... -race) passes.

Signed-off-by: Lukas Hirt <info@hirt.cz>
@LukasHirt
LukasHirt force-pushed the fix/sse-reconcile-race branch from ac26ece to f714fe8 Compare July 24, 2026 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant