Skip to content

feat: replace TCP sockets with WebSocket, MPLogger with OTel, StorageController as thread#1144

Draft
vringar wants to merge 1 commit intomasterfrom
refactor/websocket-otel-storage
Draft

feat: replace TCP sockets with WebSocket, MPLogger with OTel, StorageController as thread#1144
vringar wants to merge 1 commit intomasterfrom
refactor/websocket-otel-storage

Conversation

@vringar
Copy link
Copy Markdown
Contributor

@vringar vringar commented Feb 24, 2026

Summary

Major architectural refactoring of OpenWPM's internal communication and infrastructure:

  • WebSocket: Replace raw TCP socket IPC between Extension and Python with a single bidirectional WebSocket connection per browser (removes dependency on privileged browser.sockets experiment API)
  • StorageController: Convert from a separate process (with async TCP server) to a thread using multiprocess.Queue for data flow
  • Telemetry: Replace MPLogger with OpenTelemetry tracing + standard Python logging (structured traces, per-browser context, optional OTLP export)
  • Command API: Introduce ExtensionSocket Protocol type for backward-compatible BaseCommand.execute() signature
  • Unicode fix: Remove UTF-8 byte encoding in escapeString (was needed for old binary socket protocol, breaks JSON/WebSocket)

New files

  • openwpm/telemetry.py — OTel TracerProvider + Python logging setup
  • openwpm/websocket_bridge.py — WebSocketBridge (asyncio WS server in thread) + ExtensionSocketAdapter
  • Extension/src/websocket-client.ts — Browser-side WebSocket client

Key design decisions

  • BrowserManager stays as a separate process (crash isolation for robustness at scale)
  • Data flows: Extension → WebSocket → BrowserManager process → multiprocess.Queue → StorageController thread
  • Public config API (BrowserParams, ManagerParams, CommandSequence, BaseCommand) unchanged

Test plan

  • All 110 tests passing (16 simple_commands, 17 HTTP, 6 JS instrument, 21 profile, 5 task_manager, 10 storage, + others)
  • mypy clean (44 files, 0 errors)
  • Extension lint clean (0 errors)
  • POST data Unicode encoding verified (was broken, fixed via escapeString change)

@codecov
Copy link
Copy Markdown

codecov bot commented Feb 24, 2026

Codecov Report

❌ Patch coverage is 76.92308% with 66 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.64%. Comparing base (58218c7) to head (f3d75f8).

Files with missing lines Patch % Lines
openwpm/telemetry.py 61.53% 35 Missing ⚠️
openwpm/websocket_bridge.py 85.41% 14 Missing ⚠️
openwpm/storage/storage_controller.py 85.00% 9 Missing ⚠️
test/test_mp_logger.py 0.00% 3 Missing ⚠️
openwpm/browser_manager.py 89.47% 2 Missing ⚠️
test/test_custom_function_command.py 50.00% 2 Missing ⚠️
test/test_xvfb_browser.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1144      +/-   ##
==========================================
- Coverage   62.16%   57.64%   -4.52%     
==========================================
  Files          40       42       +2     
  Lines        3898     3978      +80     
==========================================
- Hits         2423     2293     -130     
- Misses       1475     1685     +210     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vringar vringar added the backlog Not a priorty, but nice to have label Feb 27, 2026
…Controller process with thread

Major architectural refactoring of OpenWPM's internal communication and infrastructure:

- Replace raw TCP socket IPC between Extension and Python with a single
  bidirectional WebSocket connection per browser
- Convert StorageController from a separate process (with TCP server) to a
  thread using multiprocess.Queue for data flow
- Replace MPLogger with OpenTelemetry tracing + standard Python logging
- Introduce ExtensionSocket Protocol type for backward-compatible command API
- Fix Unicode double-encoding in escapeString (no longer needed with JSON/WebSocket)

New files:
- openwpm/telemetry.py: OTel TracerProvider + Python logging setup
- openwpm/websocket_bridge.py: WebSocketBridge + ExtensionSocketAdapter
- Extension/src/websocket-client.ts: Browser-side WebSocket client
@vringar vringar force-pushed the refactor/websocket-otel-storage branch from c4b29e7 to f3d75f8 Compare March 28, 2026 19:05
vringar added a commit that referenced this pull request Mar 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backlog Not a priorty, but nice to have

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant