perf(moq-pub-mmtp): parallelize publisher pipeline + gated pprof profiling endpoint

[kkroo]

What

Two changes to moq-pub-mmtp, one combined PR:

1. Parallelize the publisher pipeline (the headline). The publisher ran its three long-lived halves in a single tokio::select!:

tokio::select! {
    res = session.run() => ...,
    res = publisher.publish_namespace(tracks_reader) => ...,   // egress/encode
    res = run_publisher(...) => ...,                            // multicast ingest
}

That is one future = one task, and tokio never parallelizes a single task across workers — so recv_from -> route -> per-shred open_uni -> encode -> quinn-write all serialize on one core (~0.97 core, ~90% userspace) no matter how many workers the runtime has. (This is why bumping the pod CPU limit 1→2 core only removed CFS throttling; it did not lift the ceiling.)

This PR spawns the three halves as separate tasks and races their JoinHandles, letting the multi-thread runtime place ingest and egress on different workers.

2. Optional on-demand CPU profiler behind the profiling cargo feature (off by default) and the MOQ_PUB_PROFILE_ADDR env var (unset by default) — the instrument used to confirm which userspace frames dominate the pub's single-thread core.

Wire-safety (relay unaffected)

The producer (SubgroupsWriter) and consumer (SubgroupsReader) already communicate through an Arc-backed watch::State, so the split is a scheduling change only — the wire output (objects, groups, subgroups, framing, ordering) is byte-identical. run_publisher stays a single task, so the monotonic group_id assignment (datagram.rs) remains a single ordered point. Teardown is preserved: the first task to finish (Ok or Err) is propagated exactly as the old select! did, so the external watchdog still respawns on error.

Profiling: default build is unchanged

With the feature off, the default cargo build --release never compiles pprof/tiny_http, so the shipped binary is byte-for-byte unchanged. pprof-rs samples via SIGPROF/ITIMER_PROF (userspace, no CAP_SYS_ADMIN), so it runs under baseline PodSecurity. tiny_http serves on its own std thread (cannot perturb the tokio hot path).

# build a profiling image
docker build -f Dockerfile --build-arg PROFILING=1 -t <img>:profiling .
# run with the endpoint on loopback, then capture
MOQ_PUB_PROFILE_ADDR=127.0.0.1:6060 ...
kubectl port-forward pod/<pub> 6060:6060
curl 'http://127.0.0.1:6060/debug/pprof/flamegraph?seconds=30' -o pub-flame.svg

Deploy note (do NOT auto-roll from this merge)

This PR lands the code only. The prod shred pub is digest-pinned/operator-managed (not Argo-auto-rolled), so merging here does not deploy. The parallelization roll is intentionally gated on: (a) a flamegraph confirming the userspace core is the produce/egress split rather than per-shred open_uni (which is wire-locked and out of scope), and (b) a >1h no-crash soak + disconnect→watchdog-respawn validation, off a digest-pinned image with clean-slate recovery staged.

Testing (local, on this branch)

• cargo check -p moq-pub-mmtp (default) ✅ — proves the three futures are Send + 'static on moq-transport 0.15
• cargo check -p moq-pub-mmtp --features profiling ✅
• cargo clippy -p moq-pub-mmtp --no-deps (both feature states) ✅ no warnings
• cargo test -p moq-pub-mmtp ✅ 64 tests pass
• cargo fmt --check ✅ ; REUSE covered by the moq-pub-mmtp/** glob

🤖 Generated with Claude Code

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.

reviewed head: 9900c59

Critical Issues (0)

None.

Important Issues (0)

None.

Suggestions (3)

• [gstack/review] moq-pub-mmtp/src/profiling.rs:190-208 — spawn_if_enabled()/serve() bind whatever address MOQ_PUB_PROFILE_ADDR names with no code-level check that it's loopback, and the endpoint is unauthenticated (each request can hold the thread for up to 120s of sampling). The double gate (feature off by default, env var unset by default) makes this low-risk today, but nothing stops a future operator from setting MOQ_PUB_PROFILE_ADDR=0.0.0.0:6060 on a profiling-enabled image and exposing an unauthenticated CPU-sampling DoS surface pod-wide. Consider a defensive check that refuses/warns loudly on a non-loopback bind address.
• [native-codex] moq-pub-mmtp/src/main.rs:142-150 — .abort() on the two losing JoinHandles is best-effort: since the three halves now run on genuinely separate OS threads (not cooperative branches of one select!), a losing task can still be mid-execution of synchronous, side-effecting code (e.g. a quinn write) at the moment abort() is called, and only stops at its next .await point. The old single-task select! guaranteed losing branches were always suspended at a poll boundary. In practice this window is harmless here because main returns and the whole process/runtime exits immediately after, but it's a real semantic shift worth a one-line comment if this pattern gets reused somewhere teardown timing matters more.
• [pr-review-toolkit:tests] No test exercises the new spawn/select/abort teardown path itself (e.g. that a JoinError from a panicking task correctly bail!s, or that the first task to error wins the race). The existing 64 tests are unchanged coverage over main.rs's previous logic. Given the PR already gates the actual production rollout on a manual soak + watchdog-respawn validation, this is acceptable to defer, but worth a lightweight unit test around the Ok(inner) => inner?, Err(join_err) => bail!(...) match arm if convenient.

Strengths

• Clear, honest PR description: explains the actual mechanism (one task ≠ parallel across workers), backs the claim with the CFS-throttling observation, and is explicit about what's still gated behind manual validation before prod rollout.
• Wire-safety argument is sound — run_publisher stays a single task, so the monotonic group_id ordering point is preserved; the split is genuinely scheduling-only.
• Profiling feature is well isolated: default cargo build --release never touches pprof/tiny_http, verified via feature-gated mod profiling, cfg on the spawn call, and a separate opt-in Dockerfile stage that only runs if [ -n "$PROFILING" ].
• Cargo.lock additions are exactly pprof/tiny_http's expected transitive deps (inferno, symbolic-demangle, chunked_transfer, etc.) — no unexpected or suspicious packages.
• Docker layering keeps the profiling rebuild after the default build/copy and before the final COPY --from=builder, so the opt-in binary swap can't accidentally leak into the default image or vice versa.

Recommended Action

1. No blocking issues — safe to merge on review grounds.
2. Suggestions are optional hardening; address opportunistically.
3. Remember the PR's own stated gate: don't auto-roll the parallelized pub to prod off this merge without the flamegraph + soak validation it describes.