Skip to content

obs(orchestrator): add teamID to valuable sandbox failure logs (revives #3674) - #3727

Open
AdaAibaby wants to merge 2 commits into
e2b-dev:mainfrom
AdaAibaby:obs/orchestrator-log-teamid
Open

AdaAibaby wants to merge 2 commits into
e2b-dev:mainfrom
AdaAibaby:obs/orchestrator-log-teamid

Conversation

@AdaAibaby

Copy link
Copy Markdown
Contributor

Revives the intent of #3674 (add teamID to orchestrator failure logs) against current main, and extends it beyond the two create-path logs to the failure logs across the whole sandbox lifecycle.

Why

With 100K+ teams and a goal of 1M+, orchestrator failure logs that carry only sandboxID can't be filtered or aggregated by team at scale. Adding teamID to the error/warn logs where the team is already in scope makes per-team failure debugging and alerting possible with no runtime cost.

What

logger.WithTeamID(...) added to the following error/warn logs — no new DB lookup, no signature change, teamID taken from the runtime/metadata already held at each site:

file log teamID source
routing/publisher.go (×2) failed to publish / delete sandbox routing record sbx.Runtime.TeamID
sandbox/hoststats_collector.go (×3) push baseline / collect sample / final sample h.metadata.TeamID
sandbox/oom.go failed to seed the OOM kill watermark c.sandbox.Runtime.TeamID
sandbox/reboot.go ignoring invalid reboot CPU template override runtime.TeamID
server/prefetch_harvest.go (×3) resume prefetch harvest failed; stop/close throwaway sbx.Runtime.TeamID / runtime.TeamID
server/checkpoint_detached.go left the live map before the kill live.Runtime.TeamID

Intentionally left unchanged

Logs whose scope has no team handle are not touched, to keep this to the zero-cost shape of #3674 (no struct/signature changes): fc process teardown (SandboxFiles has no team field), network slot release (Slot holds only sandboxID), cgroup FD release (takes only sandboxID), and template build. Wiring team into those would be a separate, larger change.

Verification

go build ./... and go vet pass. Net change: 6 files, 11 WithTeamID additions.

With 100K+ teams and growing, orchestrator failure logs that only carry the
sandboxID cannot be filtered or aggregated by team at scale. This adds
logger.WithTeamID to the error/warn logs where the team is already in scope
(no new DB lookup, no signature change), following the same shape as e2b-dev#3674:

- routing/publisher.go: failed to publish / delete sandbox routing record
- sandbox/hoststats_collector.go: push baseline / collect sample / final sample
- sandbox/oom.go: failed to seed the OOM kill watermark
- sandbox/reboot.go: ignoring invalid reboot CPU template override
- server/prefetch_harvest.go: resume prefetch harvest failed; stop/close throwaway
- server/checkpoint_detached.go: left the live map before the kill

teamID is taken from the runtime/metadata already held at each site
(sbx.Runtime.TeamID / runtime.TeamID / h.metadata.TeamID). Logs whose scope
has no team handle (fc process teardown, network slot release, cgroup FD
release, template build) are intentionally left unchanged.

Revives the intent of e2b-dev#3674 (closed) against current main, extended beyond the
two create-path logs to the failure logs across the sandbox lifecycle.
… logs

Follow-on within the same PR: the three failure logs that previously could not
carry teamID without a struct/signature change now do.

- sandbox.go releaseCgroupFD: add a teamID parameter (both call sites already
  hold runtime.TeamID) so the cgroup FD release warning is team-labelled.
- fc/process.go teardown warnings (remove metrics FIFO, SIGTERM, SIGKILL): use
  the teamID the Process already holds (NewProcess receives it for metrics),
  so no new plumbing is needed.
- network/slot.go: Slot gains a teamID field set by AssignSandbox(sandboxID,
  teamID) and cleared on release; getNetworkSlot threads runtime.TeamID through
  so the non-DNS-UDP slot log is team-labelled. Test call sites updated for the
  new signatures.

Note on envd: the two envd-init error logs from the original e2b-dev#3674 are NOT
touched here — on current main they log through s.log() (= Runtime.Logger()),
whose LogFields() already emits WithTeamID on every line, so adding it
explicitly would duplicate the field. The create-path log e2b-dev#3674 targeted is
likewise already covered by the sites in the previous commit. Intentionally
left out: fc vcpu telemetry spans (telemetry.ReportError, trace-level, already
correlatable) and the sandboxtools command path (bare sandboxID threaded
through ~10 call sites for low-value Debug/span lines).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants