OTLP browser telemetry export: follow-up improvements (KERNEL-1596) by archandatta · Pull Request #317 · kernel/kernel-images

archandatta · 2026-07-16T13:07:06Z

tldr

Follow-up improvements to the in-VM OTLP browser-telemetry exporter (KERNEL-1596), post #308. Independent, reviewable commits. Off-by-default export is preserved; nothing changes for existing sessions unless the new toggle/knobs are set.

What changed

Externalize batch knobs — BTEL_OTLP_MAX_QUEUE_SIZE (2048), BTEL_OTLP_EXPORT_INTERVAL (1s), BTEL_OTLP_EXPORT_TIMEOUT (30s). Defaults match the OTel SDK's, so behavior is unchanged unless set; validated as > 0.
Per-session export toggle — new export.otlp.enabled on the telemetry API. Export is off by default and gated on the toggle, independent of whether an export destination is configured, so an always-injected relay does not generate work that gets dropped. The writer moves under a runtime-toggleable controller; toggling is best-effort (never fails a telemetry apply).
Drop / failure / exported metrics — queue-overflow drops (previously log-only) plus export failures and exported counts as Prometheus counters (kernel_otlp_records_dropped_total, kernel_otlp_export_failures_total, kernel_otlp_records_exported_total), owned above the writer so they stay monotonic across export toggles.
Crash/OOM severity — service_crashed and system_oom_kill now map to ERROR in the OTLP converter (were INFO), so consumers can alert on them.
Renderer-crash event — detect Inspector.targetCrashed (an "Aw, Snap!" renderer crash that leaves the browser process alive, so no service_crashed fires) and emit a page_crashed event, mapped to ERROR. Categorized as page (produced by the CDP monitor, captured only while the page category is enabled).

Live test results

Verified full-stack against the headless image + a real dockerized OpenTelemetry collector, driving the real telemetry API and a real Chromium over CDP:

Batch knobs: custom BTEL_OTLP_* values surfaced in the boot config (otlp_max_queue_size=512 otlp_export_interval=500ms otlp_export_timeout=30s); export shows available, not started, at boot.
Export toggle: GET /telemetry → 404 at boot; PUT with export.otlp.enabled:true → 201 echoing it. With export off but page/network capture on, exported_total froze (10 → 10) and the collector received nothing new; re-enabling resumed export (10 → 28), confirming the controller rebuilds the one-shot writer.
Renderer crash: a real chrome://crash produced a page_crashed record at the collector with category=page, url=chrome://crash/, and SeverityText: ERROR.
Converter fidelity: network_response carried promoted attributes (http.request.method, url.full, http.response.status_code=200); screenshot/monitor categories never reached the collector.
Metrics: /metrics served kernel_otlp_records_dropped_total 0, kernel_otlp_export_failures_total 0, kernel_otlp_records_exported_total 28.

🤖 Generated with Claude Code

Note

Medium Risk
Changes default OTLP behavior (no export until API toggle) and extends the telemetry API contract; export reconciliation is best-effort and should not break capture, but operators relying on always-on export must enable the new flag.

Overview
Follow-up on in-VM OTLP browser telemetry: export is gated per session via export.otlp.enabled on PUT/PATCH /telemetry (off by default even when a relay endpoint is provisioned). The API wires an OTLPExportController through reconcileExport / stopTelemetryState; boot only constructs the sink and stops it after HTTP drain, mirroring S2.

Operational visibility: BTEL_OTLP_MAX_QUEUE_SIZE, BTEL_OTLP_EXPORT_INTERVAL, and BTEL_OTLP_EXPORT_TIMEOUT tune the OTel batch processor; Prometheus counters (kernel_otlp_records_*) track drops, export failures, and successes via a drop-counting slog handler on the OTel global logger.

Events: CDP monitor enables Inspector and emits page_crashed on renderer crashes; OTLP severity maps page_crashed, service_crashed, and system_oom_kill to ERROR. Re-enabled export reads from the event stream tail so the ring is not replayed.

^{Reviewed by Cursor Bugbot for commit be7dc1d. Bugbot is set up for automated code reviews on this repo. Configure here.}

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Export is off by default and gated on the toggle, independent of whether an endpoint is provisioned, so an always-injected relay does not export by default. Writer lifecycle moves under a runtime-toggleable controller. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Surfaces queue-overflow drops (previously log-only) plus export failures and exported counts as Prometheus counters, owned above the writer so they stay monotonic across export toggles. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Detects Inspector.targetCrashed (an 'Aw, Snap!' renderer crash that leaves the browser process alive, so no service_crashed fires) and emits a page_crashed event, mapped to ERROR severity in the OTLP converter. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Exercises the export path end-to-end against a real dockerized collector: severity mapping, category exclusion, promoted attributes, export counters, and a forced queue overflow proving the drop metric. Run with -tags livetest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…, crash guard - Drop 'on the VM' from the public export spec descriptions. - Thread request ctx into export reconcile; log via the context logger, and drain with context.WithoutCancel so a toggle-off outlives the request. - Still emit page_crashed (and warn) if a crash fires on an untracked session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… test Forces a real batch-queue overflow against a local hung listener so CI exercises the SDK->global-logger->counter chain every run; an SDK message rename now fails here instead of silently zeroing the metric. Removes the build-tagged docker live test (its coverage is now unit-side). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… dash - page_crashed on an untracked session now falls back to target_type "other" instead of an empty string, so the emitted event stays a valid BrowserTargetType. Adds a regression test for the untracked path. - Restore the PATCH /telemetry no-op short-circuit when the body carries neither a category block nor an export toggle. - Drop an em dash from a merge comment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…eout - Stop OTLP export after the HTTP servers drain (in main, mirroring s2Writer) instead of concurrently inside apiService.Shutdown, so events emitted during the shutdown window are still exported. - OTLPStorageWriter.Stop now shuts the SDK provider down even when the read-loop wait exhausts ctx, using a short detached close budget, so a slow flush can't orphan the batch-export goroutines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The runtime export toggle rebuilds the OTLP writer on each enable, and the writer read from seq 0, so a disable/re-enable re-exported the whole retained ring (up to the buffer capacity), duplicating records at the collector and inflating kernel_otlp_records_exported_total. Start the OTLP writer's reader from the current stream tail instead, so only events published while export is on are forwarded. Adds NewStorageWriterAfter (S2 keeps starting from 0) and a regression test that toggles off/on and asserts no replay and no off-window export. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes using high effort and found 4 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit be7dc1d. Configure here.}

cursor · 2026-07-21T19:18:32Z

+	// the writer is rebuilt on each enable, so reading from the oldest buffered
+	// event would re-export the retained ring every time it is turned back on.
+	// Only events published while export is on are forwarded.
+	w.writer = NewStorageWriterAfter(w.es, storage, w.log, w.es.Seq())


Export enable can skip in-flight events

Medium Severity

Start snapshots es.Seq() then builds a tail reader, but Publish increments seq before ring.publish (and may spend time in truncateIfNeeded). An event whose seq is observed during that window is skipped forever by the new reader, even if it lands in the ring after export is enabled. Under concurrent capture this can drop the first events after an export toggle-on.

^{Reviewed by Cursor Bugbot for commit be7dc1d. Configure here.}

cursor · 2026-07-21T19:18:32Z

+		})
+	}
+	return h.Handler.Handle(ctx, r)
+}


Drop counter wrapper loses attrs

Medium Severity

dropCountingHandler embeds slog.Handler and only overrides Handle, so promoted WithAttrs / WithGroup return the unwrapped base handler. Any logr/OTel path that names or attaches values before logging dropped log records bypasses the counter, leaving kernel_otlp_records_dropped_total stuck at 0 despite real queue overflow.

^{Reviewed by Cursor Bugbot for commit be7dc1d. Configure here.}

cursor · 2026-07-21T19:18:32Z

+		TargetId:   info.targetID,
+		TargetType: targetType,
+		Url:        info.url,
+	})


Stale URL on page crash

Medium Severity

page_crashed fills url from m.sessions attach-time target info, which is never updated on navigation. Other page events use the computed nav context’s current URL. After the first navigation, crash events report the wrong page URL even though the OpenAPI text describes the URL at crash time.

Additional Locations (1)

server/lib/cdpmonitor/handlers.go#L730-L735

^{Reviewed by Cursor Bugbot for commit be7dc1d. Configure here.}

cursor · 2026-07-21T19:18:32Z

+		stopCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), otlpStopTimeout)
+		defer cancel()
+		_ = s.otlpExport.Stop(stopCtx)
+	}


Mutex held during export drain

Medium Severity

Disabling OTLP export calls Stop while monitorMu is held, and OTLPExportController.Stop also holds its own mutex for the full drain (up to otlpStopTimeout, 5s). That stalls all telemetry GET/PUT/PATCH handlers and ApiService.Shutdown for the drain window.

Additional Locations (1)

server/lib/events/otlpstorage.go#L343-L354

^{Reviewed by Cursor Bugbot for commit be7dc1d. Configure here.}

archandatta force-pushed the archand/kernel-1596/otlp-export-followups branch from 643f503 to 5e56d6a Compare July 17, 2026 13:11

archandatta requested a review from Sayan- July 21, 2026 18:24

archandatta marked this pull request as ready for review July 21, 2026 18:24

archandatta and others added 10 commits July 21, 2026 18:25

Externalize OTLP batch knobs (queue size, export interval, timeout)

b8ea737

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Map service-crash and OOM events to ERROR severity in OTLP converter

4180439

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Document that page_crashed requires the page category; drop em dashes

d845b30

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

archandatta force-pushed the archand/kernel-1596/otlp-export-followups branch from 3286139 to 666c148 Compare July 21, 2026 18:26

cursor Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread server/cmd/api/main.go

Comment thread server/cmd/api/main.go

Comment thread server/lib/events/otlpstorage.go

cursor Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread server/lib/events/otlpstorage.go

cursor Bot reviewed Jul 21, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

OTLP browser telemetry export: follow-up improvements (KERNEL-1596)#317

OTLP browser telemetry export: follow-up improvements (KERNEL-1596)#317
archandatta wants to merge 12 commits into
mainfrom
archand/kernel-1596/otlp-export-followups

archandatta commented Jul 16, 2026 •

edited by cursor Bot

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Jul 21, 2026

Uh oh!

cursor Bot Jul 21, 2026

Uh oh!

cursor Bot Jul 21, 2026

Uh oh!

cursor Bot Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

archandatta commented Jul 16, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tldr

What changed

Live test results

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jul 21, 2026

Choose a reason for hiding this comment

Export enable can skip in-flight events

Uh oh!

cursor Bot Jul 21, 2026

Choose a reason for hiding this comment

Drop counter wrapper loses attrs

Uh oh!

cursor Bot Jul 21, 2026

Choose a reason for hiding this comment

Stale URL on page crash

Uh oh!

cursor Bot Jul 21, 2026

Choose a reason for hiding this comment

Mutex held during export drain

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

archandatta commented Jul 16, 2026 •

edited by cursor Bot

Loading