feat: migrate push to SSE with WebSocket rollout compatibility#2265
feat: migrate push to SSE with WebSocket rollout compatibility#2265niemyjski wants to merge 7 commits into
Conversation
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
This PR migrates real-time push from WebSockets to SSE while keeping a temporary WebSocket compatibility path for rolling deploys, updates both frontends to consume SSE, and adds rollout/shutdown configuration for long-lived push connections.
Changes:
- Adds SSE connection, middleware, broker fanout, diagnostics, and
EnablePushconfiguration. - Replaces Svelte/Angular WebSocket clients with fetch/stream-based SSE clients and updates tests.
- Adds Kubernetes drain settings and documents push being disabled until ingress supports SSE.
Reviewed changes
Copilot reviewed 30 out of 30 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/Exceptionless.Web/Hubs/SseConnection.cs |
Adds bounded SSE write queue, deduplication, keep-alives, and disposal. |
src/Exceptionless.Web/Hubs/SseConnectionManager.cs |
Manages SSE connections, keep-alives, cleanup, and metrics. |
src/Exceptionless.Web/Hubs/SseMiddleware.cs |
Adds authenticated /api/v2/push SSE endpoint. |
src/Exceptionless.Web/Hubs/WebSocketPushMiddleware.cs |
Adds temporary WebSocket compatibility middleware. |
src/Exceptionless.Web/Hubs/WebSocketConnectionManager.cs |
Reworks WebSocket manager for compatibility fanout and metrics. |
src/Exceptionless.Web/Hubs/MessageBusBroker.cs |
Fans out push messages to SSE and WebSocket managers. |
src/Exceptionless.Web/Hubs/MessageBusBrokerMiddleware.cs |
Removes old WebSocket-only middleware. |
src/Exceptionless.Web/Startup.cs |
Wires SSE and WebSocket push middleware behind EnablePush. |
src/Exceptionless.Web/Program.cs |
Adds host shutdown timeout for drain alignment. |
src/Exceptionless.Web/Bootstrapper.cs |
Registers the SSE connection manager. |
src/Exceptionless.Web/Utility/Handlers/ThrottlingMiddleware.cs |
Renames push throttle exemption to SSE-oriented naming. |
src/Exceptionless.Core/Configuration/AppOptions.cs |
Replaces EnableWebSockets with EnablePush and legacy fallback. |
src/Exceptionless.Core/Bootstrapper.cs |
Updates push-disabled startup warning. |
src/Exceptionless.Core/Utility/AppDiagnostics.cs |
Adds SSE/WebSocket push connection counters. |
src/Exceptionless.Web/ClientApp/src/lib/features/websockets/sse-client.svelte.ts |
Adds Svelte SSE client with reconnect and stream parsing. |
src/Exceptionless.Web/ClientApp/src/lib/features/websockets/sse-client.test.ts |
Adds Svelte SSE client unit tests. |
src/Exceptionless.Web/ClientApp/src/lib/features/websockets/web-socket-client.svelte.ts |
Removes Svelte WebSocket client. |
src/Exceptionless.Web/ClientApp/src/lib/features/websockets/web-socket-client.test.ts |
Removes WebSocket client tests. |
src/Exceptionless.Web/ClientApp/src/routes/(app)/+layout.svelte |
Switches app layout push integration to SSE. |
src/Exceptionless.Web/ClientApp.angular/components/websocket/websocket-service.js |
Rewrites Angular legacy push service to SSE. |
tests/Exceptionless.Tests/Hubs/SseTests.cs |
Adds SSE manager, broker, and deduplication tests. |
tests/Exceptionless.Tests/Hubs/SseIntegrationTests.cs |
Adds HTTP pipeline tests for SSE endpoint behavior. |
tests/Exceptionless.Tests/Hubs/FakeHttpResponse.cs |
Adds fake response helper for SSE tests. |
tests/Exceptionless.Tests/Hubs/WebSocketCompatibilityTests.cs |
Adds WebSocket compatibility and dual-fanout tests. |
tests/Exceptionless.Tests/Hubs/WebSocketTests.cs |
Removes old WebSocket broker tests. |
tests/Exceptionless.Tests/Hubs/WebSocketConnectionManagerTests.cs |
Removes old WebSocket manager tests. |
tests/Exceptionless.Tests/Hubs/TestWebSocket.cs |
Updates test WebSocket encoding and close count layout. |
tests/Exceptionless.Tests/appsettings.yml |
Enables push for test host configuration. |
tests/http/push.http |
Adds manual SSE smoke requests for localhost. |
k8s/exceptionless/templates/api.yaml |
Adds drain lifecycle settings and disables push pending ingress migration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Replace the WebSocket-based real-time push infrastructure with SSE:
Backend:
- Add SseConnection (Channel<T>-based, bounded queue, write loop)
- Add SseConnectionManager (keep-alive timer, connection limits)
- Add SseMiddleware (handles GET /api/v2/push with auth)
- Update MessageBusBroker to use SSE connections
- Remove WebSocketConnectionManager and MessageBusBrokerMiddleware
- Remove UseWebSockets() from Startup
- Maintain throttle exemption for /api/v2/push
Frontend (Svelte 5):
- Add sse-client.svelte.ts using fetch+ReadableStream
- Use Authorization header (no token-in-URL)
- Exponential backoff, auth failure detection, visibility integration
- Remove old web-socket-client.svelte.ts
Frontend (Angular legacy):
- Rewrite websocket-service.js to use fetch+ReadableStream SSE
- Same event dispatch pattern ($rootScope.$emit)
- Authorization header auth, exponential backoff
Tests:
- 11 backend unit tests (SseConnectionManager + SseBroker)
- 6 integration tests (require Elasticsearch)
- 15 frontend unit tests (connection lifecycle, reconnection, auth)
- 5 throttling tests pass (no regression)
Security audit passed: proper auth pipeline, bounded channels,
per-user connection limits (10), no SSE injection, token revocation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fix: Address SSE review blockers - dead conn cleanup, config rename, UserMembershipChanged
Blocker fixes from adversarial review:
1. Config flag renamed: EnableWebSockets → EnablePush (backward compat reads both)
- k8s api.yaml now sets EnablePush=true (SSE was accidentally disabled)
- Test appsettings also enables push so integration tests can exercise SSE
- Default remains true for environments without explicit config
2. Dead connection cleanup: WriteLoopAsync now cancels CTS in finally block
- Ensures ConnectionAborted fires even when write loop exits due to
IOException/ObjectDisposedException, so middleware cleanup always runs
- Keep-alive sweep now proactively prunes connections that fail writes
(previously only pruned already-canceled connections)
- Prevents stale connection IDs from accumulating in Redis/IConnectionMapping
and poisoning the per-user connection limit
3. Svelte UserMembershipChanged parity with Angular
- Org/project queries are now invalidated when membership changes
- Matches the Angular client's fan-out behavior
4. Documented soft connection limit (race is acceptable - distributed lock
would add latency without security benefit for a cap of 10)
5. Added DroppedMessages counter to SseConnection for observability
of backpressure events under burst load
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…st config - SseConnectionManager: implement IAsyncDisposable so DI container on .NET 8+ calls DisposeAsync() at shutdown, eliminating the blocking GetAwaiter().GetResult() that could deadlock under async test hosts; sync Dispose() delegates to DisposeAsync() - Program.cs: set ShutdownTimeout = 45s to match k8s drain window (terminationGracePeriodSeconds 60s - preStop sleep 15s = 45s for ASP.NET Core drain) - k8s api.yaml: add terminationGracePeriodSeconds: 60 and preStop: sleep 15 lifecycle hook so the ALB/ingress deregisters the pod before SIGTERM cancels active SSE connections - tests/appsettings.yml: EnableWebSockets → EnablePush (canonical new key) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- restore the legacy Angular push service to prefer WebSocket so cached and in-flight Angular clients keep their current release-notification refresh behavior during rollout - add a thin SSE fallback for Angular if WebSocket cannot be established at startup, while leaving the Svelte app on SSE only - preserve the legacy WebSocket manager/test surface with temporary compatibility wrappers and explicit deprecation comments so the backend can dual-write with minimal churn - harden AppWebHostFactory Elasticsearch slice reuse so rebased full-suite runs clean up stale test indices before startup and do not return a slice to the reuse pool until the previous host is fully disposed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
niemyjski
force-pushed
the
niemyjski/websocket-to-sse-migration
branch
from
09d643e to
7398308
Compare
niemyjski
changed the title
- restore the legacy Angular websocket client and leave only a deprecation note - tighten SSE and websocket lifecycle cleanup plus bounded queue behavior - fix client lint and typecheck failures and harden test-host Elasticsearch isolation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- skip keepalive enqueue when the bounded SSE queue is full so critical notifications are never evicted - pause hidden Svelte tabs without scheduling reconnect churn - add queue regression coverage for critical-message preservation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- treat a missing /api/v2/push endpoint as unavailable instead of reconnectable - reset the unavailable state when the auth token changes - cover the rollout-off path with a no-reconnect client test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
|
||
| Assert.Equal(SseConnection.EnqueueResult.Skipped, result); | ||
| Assert.Equal("critical-1", item!.Value.Data); | ||
| Assert.False(item.Value.IsKeepAlive); |
| { | ||
| await _writeLoop.ConfigureAwait(false); | ||
| } | ||
| catch (OperationCanceledException) { } |
| await _response.Body.FlushAsync(ct); | ||
| } | ||
| } | ||
| catch (OperationCanceledException) { } |
| } | ||
| } | ||
| catch (OperationCanceledException) { } | ||
| catch (ObjectDisposedException) { } |
| } | ||
| catch (OperationCanceledException) { } | ||
| catch (ObjectDisposedException) { } | ||
| catch (IOException) { } |
| { | ||
| await disposeTask.ConfigureAwait(false); | ||
| } | ||
| catch (OperationCanceledException) { } |
| await disposeTask.ConfigureAwait(false); | ||
| } | ||
| catch (OperationCanceledException) { } | ||
| catch (ObjectDisposedException) { } |
| // Hold the response open until the client disconnects or the connection is aborted | ||
| await Task.Delay(Timeout.Infinite, connection.ConnectionAborted).ConfigureAwait(false); | ||
| } | ||
| catch (OperationCanceledException) { } |
| await socket.CloseAsync(WebSocketCloseStatus.NormalClosure, "Closed by manager", CancellationToken.None).ConfigureAwait(false); | ||
| } | ||
| catch (WebSocketException ex) when (ex.WebSocketErrorCode == WebSocketError.ConnectionClosedPrematurely) | ||
| catch (WebSocketException ex) when (ex.WebSocketErrorCode == WebSocketError.ConnectionClosedPrematurely) { } |
| } | ||
| catch (WebSocketException ex) when (ex.WebSocketErrorCode == WebSocketError.ConnectionClosedPrematurely) | ||
| catch (WebSocketException ex) when (ex.WebSocketErrorCode == WebSocketError.ConnectionClosedPrematurely) { } | ||
| catch (ObjectDisposedException) { } |
|
|
Addressed the outstanding review feedback in the latest pushes: restored the Angular legacy client to the existing WebSocket path with only a deprecation note, tightened SSE/WebSocket lifecycle cleanup, fixed the SSE queue and keepalive backpressure edge case, and stopped the Svelte client from retrying when push is disabled.\n\nLocal validation is green (, , , , and Using launch settings from /Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/Properties/launchSettings.json... [+94/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(5s) [+239/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(8s) [+338/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(11s) [+483/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(14s) [+506/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(17s) [+569/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(20s) [+684/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(23s) [+743/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(26s) [+835/x0/?0] Exceptionless.Tests.dll (net10.0|arm64)(29s) skipped Exceptionless.Tests.Pipeline.EventPipelineTests.GeneratePerformanceDataAsync (0ms) [+1042/x0/?1] Exceptionless.Tests.dll (net10.0|arm64)(35s) [+1151/x0/?1] Exceptionless.Tests.dll (net10.0|arm64)(38s) skipped Exceptionless.Tests.Repositories.EventRepositoryTests.GetAsyncPerformanceAsync (0ms) [+1277/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(44s) [+1424/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(47s) [+1567/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(50s) [+1650/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(53s) [+1792/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(56s) [+1864/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(59s) [+1873/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 02s) [+1873/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 05s) [+1874/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 08s) [+1875/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 11s) [+1876/x0/?3] Exceptionless.Tests.dll (net10.0|arm64)(1m 14s) /Users/blake/Projects/Exceptionless/copilot-worktrees/Exceptionless/niemyjski-stunning-robot/tests/Exceptionless.Tests/bin/Debug/net10.0/Exceptionless.Tests.dll (net10.0|arm64) passed (1m 14s 985ms) Test run summary: Passed! |
Why
Exceptionless push is fundamentally server-to-client fanout, so SSE is the right long-term transport for the new UI and the backend. But we cannot cut WebSockets out of this release entirely because the legacy Angular site still needs to keep its current release-notification refresh behavior while cached and in-flight clients roll forward. This PR moves the platform to SSE as the primary transport while keeping a tightly scoped WebSocket compatibility path during rollout.
What changed
Backend
SseConnection,SseConnectionManager, andSseMiddlewarefor/api/v2/push.MessageBusBrokernow dual-writes push messages to both SSE and WebSocket connection managers.PlanOverage,ReleaseNotification,SystemNotification) are preserved under pressure.WebSocketPushMiddlewareandWebSocketConnectionManagercompatibility layer for rollout safety. Both files are explicitly temporary./api/v2/pushexempt from throttling.EnableWebSocketswithEnablePush, while still honoring the legacy setting as a backward-compatible alias.Frontends
sse-client.svelte.ts./api/v2/pushendpoint as rollout-disabled and does not spin in a reconnect loop whenEnablePushis off.Shutdown and rollout safety
SseConnectionManagerimplementsIAsyncDisposableso host shutdown does not block on sync disposal.terminationGracePeriodSeconds: 60,preStop: sleep 15, andShutdownTimeout = 45sso active push connections drain cleanly during pod shutdown.AppWebHostFactoryElasticsearch slice lifecycle so full-suite test runs clean stale indices before startup and do not return a slice to the reuse pool until the previous host is fully disposed.Rollout prerequisites for production AKS
Azure Application Gateway for Containers Ingress API still does not support this SSE route safely without Gateway API plus
RoutePolicy routeTimeout: 0s. The local k8s chart intentionally leaves push disabled until that migration is complete.networking.k8s.io/v1 Ingressto Gateway APIHTTPRouteRoutePolicywithrouteTimeout: 0sto/api/v2/pushEnablePush: truein Helm valuesex.push.connections.sse.activeandex.push.connections.websocket.activeduring rolloutWebSocketConnectionManagerandWebSocketPushMiddlewareonce WebSocket active connections stay at 0Tests
npm run lintnpm run checknpm run buildnpm run test:unitdotnet test --project tests/Exceptionless.Tests/Exceptionless.Tests.csprojNon-obvious trade-offs