Multi-agent gateway: QUIC / multi-TCP / mux transports, Ed25519 identity + enrollment#9
Merged
Conversation
Checkpoint of in-progress work predating the honesty-pass commits: the glass design system (styles/*, DESIGN.md, component/screen restyle), the RoleSwitcher, the scroll rubber band, the ui-design-reviewer agent, doc updates, CI/version bumps, and dependency refreshes. Committed as a baseline so the honesty-pass features land as clean, separate commits on top. Reword or split as you like.
Design of record for closing four Reality-check lies (each deletes its CLAUDE.md row when it lands): stop advertising CapTunnelUDP, correct/hide stub UI copy, implement Prometheus /metrics, and wire the offline MOTD responder into the gateway. Bandwidth cap and multi-agent namespacing are carved out into their own specs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CapTunnelUDP was in SupportedCapabilities but no UDP relay exists and validateSpec rejects type:"udp" — the gateway would echo the capability back in hello_ok and the agent could act on a udp spec that then dies. A live protocol lie against the "never advertise what isn't implemented end-to-end" invariant. Remove CapTunnelUDP from the shared SupportedCapabilities slice (drops it from both the agent offer and the gateway's accepted intersection in one edit), delete the const, and correct the now-stale comments in control.go and config.go. TestSupportedCapabilities gains a positive guard that "tunnel-udp" is never advertised. No golden-frame impact (capabilities is a live-negotiated omitempty field, absent from the byte-identical compat assertions) and no ProtocolVersion bump (dropping a capability is backward-compatible). Protocol-only change; UDP support is a later lane. Deletes the UDP Reality-check row's oversell framing and the invariant bullet's live-violation note; retires the tunnel-udp counterexample in the wire-protocol skill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three copy/visibility fixes so the UI stops promising what the backend doesn't do: - Minecraft-aware hint: drop the false "poll for MOTD/player count/version" claim (no status poller exists); keep the true username-sniffing half. - Remove the "Per-connection" transport option the agent never honors, and correct the Transport field hint. - Remove the inert "Minimize to tray" / "Start on login" toggles and their now-empty Behavior section, plus its SectionRail entry (a dangling rail entry would mis-measure the scrollspy). The Offline-MOTD, Bandwidth-cap, and Prometheus controls stay — they become real in the other honesty-pass commits. Per-conn / tray / autostart remain in the Reality-check table (hidden, not implemented). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MetricsConfig was stored and the Settings toggle round-tripped it, but no server existed. Add a hand-rolled text-format handler (no client_golang dependency — the exposition format is trivial and the repo keeps deps minimal) that renders a single Engine.Status() snapshot per scrape, so a scrape adds zero work to the data path. The server spawns under runCtx alongside the sampler/resolver with its own metricsDone drain, and shuts its listener via http.Server.Shutdown before Run returns — mandatory so an in-process RestartEngine can re-bind the same port. Unlike the IPC pipe it is non-fatal: a bind failure logs a warning and disables the endpoint rather than killing the engine. Stays loopback by default and warns on a non-loopback bind; emits no player names or peer IPs as labels (privacy charter); omits -1 "unknown" gauges rather than exporting a fake zero. Tests cover the formatter (incl. sentinel omission and label escaping), a live scrape, restart port-reuse, and the disabled path. Deletes the Prometheus Reality-check row. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mc.ServeOffline was built and fuzzed but never called: a player hitting a tunnel whose local server was down got a dead socket. Wire it into handleClient via one serveOffline helper, gated strictly on a non-empty OfflineMOTD (empty keeps today's clean-close behavior — the guard is load-bearing, or every backend-down tunnel would start emitting a MOTD to opted-out operators). The primary trigger is the agent-reported health map (backendDown reads sess.health for the tunnel), covering the common "agent up, server crashed" case; the rare no-session race and an OpenStream failure route through the same helper. The offline path takes no data stream, opens no conntrack entry, and sets a serve deadline (the public conn has none otherwise). It covers all three exchanges ServeOffline handles: server-list status ping, login-attempt disconnect, and the legacy 0xFE ping. e2e: a status query returns the MOTD, a login attempt is disconnected with it, and an unset MOTD still cleanly closes — each with the agent's health probe driving a genuinely dead backend. Deletes the Offline-MOTD Reality-check row. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Thread agentID from the owning session through the data model:
- conntrack Entry/Open/Snapshot carry AgentID (gateway passes sess.agentID,
agent passes its configured id).
- analytics schemaV3 (full-wipe rebuild) adds agent_id to sessions, events,
and rrd, and to the rollup_hourly/rollup_daily primary keys; agent_id ''
is the gateway-wide series.
- recorder/event journal thread the id; rollups write both the global ('')
and per-agent series; summary/retention read the global series.
Groundwork for the multi-agent gateway; single-agent is still enforced.
Extract the tier-ring ladder into a reusable history type; the Store now owns one global history (the gateway-wide series, agent_id '' on disk) plus a map of per-agent histories, LRU-capped at maxAgentHistories. Add SampleAgent and AgentHistory alongside the unchanged global Sample/History, and thread agent_id through the rrd persistence (read, incremental write, expire, and delete-on-evict). Global-path semantics are unchanged (existing tests pass); per-agent sampling is wired to the engine next.
conntrack accumulates closed-connection bytes per agent so AgentTotals gives monotonic per-agent byte totals (plus live conn/player counts); the gateway exposes per-agent link quality via AgentQuality. The 10 Hz sampler now feeds each connected agent's series through Stats.SampleAgent alongside the global Sample. Tests: single-agent equivalence, LRU cap, and a two-agent SQLite round-trip proving the series stay distinct.
…ries TunnelSnapshot and ipc.TunnelStatus carry the owning AgentID (filled from the listener's session). PlayersQuery/SessionsQuery gain an AgentID scope: setting it isolates one agent's players/sessions even when two agents share a tunnelId; a bare tunnelId aggregates across agents. T4 locks the isolation. The Wails app mirror (UIStatus/ConnUI agentId) lands in Phase 3 with the GUI that consumes it.
Drive the real recorder via conntrack: two agents on the same tunnel_id land distinctly-attributed session rows and isolate through the scoped query. Proves the A-then-B commingling bug is fixed on the actual write path.
Replace the single-agent actor with a concurrent one: - a.current -> agents map[agentID]; listeners nested (agentID -> tunnelID), so agents sharing a tunnelID bind independently and one can never steal another's listener. - admit supersedes a matching agentID and admits distinct ones alongside; a genuine ID collision is anti-flap dampened (transient reject, not fatal). - evictLocked -> per-agent evict: closes only that agent's listeners (keeping the ghost-listener <-done wait) and its mux (the connection-drain boundary), never the global clientWG. Currency is map identity, never a generation compare, so one agent's churn can't invalidate another. - Port policy is global FCFS; the loser gets port_in_use. Coarse single-agent status accessors read 'unknown' with N>1 agents (honest) until Phase 3's Status.Agents. Relaxes the CLAUDE.md liveness invariant and documents the shared-token residual risk. e2e: TestSecondAgentRejected -> TestSecondAgentAdmitted, plus same-tunnelID (T1), concurrent port race (T2), cross-agent churn (T3), eviction isolation+drain (T5).
Introduce ipc.AgentStatus + Status.Agents (sorted, clamped to MaxStatusAgents) carrying each connected agent's identity, link quality, and tunnel/player counts; Tunnels and Connections stay flat and gain agentId so the GUI groups them per agent without O(agents*tunnels) frame growth. Gateway.Agents() feeds it; the engine composes health + counts. Add the agent_history analytics op (reads the per-agent RRD store, DB-independent) and its App.AgentBandwidthHistory binding for the drill-in chart. app UIStatus mirrors agents + agentId on tunnels/conns; wailsjs bindings regenerated.
New gateway-role Agents screen: a fleet roster of frost health cards —
sort by health/traffic/players/name, grid/list density, hostname filter
— with a self-contained per-agent drill-in: identity, link health, the
same bare bandwidth hero as Traffic scoped to the agent's own RRD, and
its tunnels + live sessions.
Nav is now role-derived (mainNav/settingsNav/navFor/labelOf) so the
Agents rail entry and its Ctrl-digit appear only for the gateway;
ConnectionPill rolls the fleet up ("N agents online" + worst-of health).
BandwidthPanel gained an optional history source + vtName so the drill-in
reuses the full instrument without a fork; worstHealth is shared via
state.ts.
Four states designed: geometry-matched skeleton, data, written-empty
(zero agents), and honest-unavailable (old daemon sends no roster). Frost
cards only, never Signal Glass — N equal cards can't each answer the
pointer (DESIGN rule 2), which the charter's page list now names.
Claude-Session: https://claude.ai/code/session_01MFGVx7RF1VhTYFgiczLn7u
&fleet=multi gives the gateway a five-machine fleet with a good/fair/poor health spread — per-agent tunnels, live sessions, and share-scaled bandwidth (bytes, connections, players, and a shifted RTT baseline) — so the roster, its sort/filter, and the drill-in all have real spread. &fleet=old sends no agents array, driving the roster's honest-unavailable state. Uptime epochs are stamped once at fixture init so the readouts actually advance, and AgentBandwidthHistory is stubbed per the binding. Claude-Session: https://claude.ai/code/session_01MFGVx7RF1VhTYFgiczLn7u
The cap gains a scope (combined | per-direction | per-connection); empty normalizes to combined. Reject an unknown scope only when a cap is set — scope is meaningless without one, and the gateway normalizes the wire value. Claude-Session: https://claude.ai/code/session_01UHmfChPcPFHLqb13Pdzvgd
Add BandwidthLimitMbps/BandwidthLimitScope (omitempty, zero = legacy/uncapped peer) so the gateway can throttle its half of the splice; specFromTunnel now copies both. No capability, no ProtocolVersion bump. Enforcement lands next. hello_compat: prove an uncapped spec is byte-identical to a legacy frame and a set cap round-trips. Claude-Session: https://claude.ai/code/session_01UHmfChPcPFHLqb13Pdzvgd
Splice gains SpliceOpts{Ctx, LimitAToB, LimitBToA}; the zero value is the
uncapped fast path and touches no context. A non-nil Limiter WaitN(ctx, n)s
before each write (n <= BufSize = the limiter's burst). Only a capped splice
derives a cancellable child ctx, cancelled when a half returns an error so an
abnormal teardown unblocks the other half's parked WaitN -- never on a clean
EOF, so the opposite direction still drains and final bytes arrive intact. A
cancelled WaitN counts as a clean close. BufSize is now exported.
Burst floor best-of-3 after: 338 MiB/s (baseline 356), worst RTT 4ms -- no
regression; the uncapped call sites pass a zero SpliceOpts.
Claude-Session: https://claude.ai/code/session_01UHmfChPcPFHLqb13Pdzvgd
BuildSet/Resolve encode the scope -> sharing shape (combined: one shared bucket; per-direction: two; per-connection: fresh per connection) and the unit (decimal Mbps, Mbps*125000 bytes/sec, one token = one byte), with burst = relay.BufSize so WaitN never exceeds it. Reconcile applies a rate-only change in place via the limiters' concurrent-safe SetLimit, and rebuilds on a scope or capped/uncapped change; every field a lock-free reader touches is immutable or atomic. The agent-side Registry keys shared sets by tunnel ID (unique within one agent); the gateway will instead key by publicListener, per (agentID, tunnelID). Adds golang.org/x/time (no transitive deps beyond the toolchain); govulncheck clean. Not yet wired into either splice. Claude-Session: https://claude.ai/code/session_01UHmfChPcPFHLqb13Pdzvgd
Wire bwcap into both splices so a configured cap actually throttles. Gateway: publicListener gains an atomic bwcap.LimiterSet; the accept-loop callback is widened to pass pl to handleClient, which resolves (inbound, outbound) and parents the splice on a new agentSession ctx cancelled in evict (so a throttled WaitN unblocks per-agent). reconcile updates the limiter in place for a rate-only change (keeping live connections) via sameListener, and rebuilds on a scope/structural change. Buckets are per (agentID, tunnelID) via the nested listeners map -- two agents' same-named tunnels cap independently. Agent: a bwcap.Registry keyed by tunnel ID; handleDataStream resolves from tun.Options and parents the splice on the session ctx; ApplyTunnels releases a removed tunnel's bucket. e2e: capped-download tests (combined + per-direction ~= cap), per-connection scope (two streams each ~= cap), and two-agent same-tunnelID isolation (each ~= cap concurrently -- the keying regression guard). Uncapped burst floor best-of-3 unchanged (~350 MiB/s). Docs: drop the CLAUDE.md Reality-check bandwidth row; note enforcement in architecture.md (Mbps x 125000, burst = relay.BufSize, per-(agentID,tunnelID)). Claude-Session: https://claude.ai/code/session_01UHmfChPcPFHLqb13Pdzvgd
Carry BandwidthLimitMbps/Scope from the gateway's publicListener spec through the status pipeline (gateway.TunnelSnapshot -> ipc.TunnelStatus -> app.TunnelUI) so the GUI can show which tunnels are capped. The agent role fills them from its own config too. All additive (omitempty on the IPC seam); read-only display, no new behavior. Claude-Session: https://claude.ai/code/session_01UHmfChPcPFHLqb13Pdzvgd
Agent Tunnels editor: a "Cap scope" Select (Combined | Per-direction | Per-connection) beside the cap input, disabled while the cap is 0; the tunnel-summary chip appends the scope when it is non-default. Select gains a disabled prop in the ui kit. Protocol moves to its own full-width row. Gateway Agents drill-in: each tunnel line shows its cap (and scope when non-combined), read from the newly-plumbed TunnelUI fields. scopeLabel/BANDWIDTH_SCOPES live in state.ts so both screens format identically. devmock: a capped tunnel in the agent config and across the multi-agent gateway fleet so the selector, chip, and drill-in render in the mock. models.ts regenerated (wails) for the new TunnelOptions.BandwidthLimitScope and TunnelUI cap fields. Claude-Session: https://claude.ai/code/session_01UHmfChPcPFHLqb13Pdzvgd
Route a hello's credential through a Validator interface instead of an inline constant-time compare. The v1 sharedTokenValidator preserves the exact behavior — constant-time shared-token check, non-empty agentID, identical HelloErr codes, and auth-limiter accounting (bad token counts, missing agentID does not). A later per-agent-identity validator swaps in without touching the accept paths or the wire; the per-conn data accept path will authenticate through the same seam. No wire change.
The gateway<->agent TLS configs never pin CurvePreferences, so Go's default already negotiates the X25519MLKEM768 post-quantum hybrid key exchange. That is the harvest-now-decrypt-later half of PQ and it is free. Add a live-handshake test asserting the negotiated key exchange on both sides, a guard test that both configs leave CurvePreferences nil so a future edit can't silently drop the hybrid, do-not-pin comments, and a numbers-table row. The ECDSA authentication half stays classical (no retroactive risk); per-agent identity + PQ signatures are deferred.
Wire vocabulary only, no behavior: CapPerConn ("per-conn-data") and the
gateway->agent TypeOpenData message with an OpenData{ConnID} payload. The
capability is deliberately NOT appended to SupportedCapabilities and the
gateway still rejects KindData — advertising a capability before the data
accept path exists end-to-end is the same protocol lie the tunnel-udp cap
once was. A guard test asserts per-conn-data stays un-advertised until the
implementation commit; OpenData gets a framing round-trip. ProtocolVersion
unchanged (additive).
Implement the reserved per-conn transport: one dedicated agent-dialed
TCP+TLS connection per player instead of one shared yamux stream, so a
lost segment on the gateway<->agent hop can no longer head-of-line-block
every player at once (the one real defect of yamux-over-one-TCP).
The control plane is unchanged. When CapPerConn is negotiated the gateway
sends open_data{connId} on the control stream; the agent dials back a
KindData conn (dialBackData), which the gateway authenticates via the
Validator and matches to the waiting player through a lock-free,
exactly-once/loser-closes pending registry (perconn.go). Both modes then
converge on the identical open_conn + splice tail, so mcsniff, bwcap, and
RTT sampling ride unchanged. Data conns resume the control conn's TLS
session (shared ClientSessionCache) to avoid a full handshake per join,
count into the same link totals, and are drained on eviction alongside
the mux (agentSession.dataConns). The gateway advertises CapPerConn only
now that it serves the accept path end-to-end; a mux/legacy agent never
offers it and rides OpenStream.
Tests: per-conn round-trip + byte-counting, eviction drain, a burst-floor
twin (clears the floor with tighter cross-stream RTT than mux), the
pending-handoff race, and a structural HOL proof (per-conn opens one
agent->gateway conn per player; mux stays at one). Deletes the per-conn
Reality-check row, updates the SetNoDelay invariant, and documents the
data plane in architecture.md.
The Transport selector only offered mux, and its hint claimed all player traffic is multiplexed over one connection — no longer true now that the per-conn data plane is real. Add the per-connection option (value matches config transport "per-conn") and reword the hint to describe the trade-off honestly: one shared connection vs a dedicated connection per player that keeps one player's loss from stalling the others. Existing Select/Field kit, no new surface.
Ed25519 enrollment ticket, transport = auto|quic|per-conn|mux, and accept_shared_token for a gradual per-agent-identity migration. Claude-Session: https://claude.ai/code/session_01EvsrY4Tq6uZCkAqm9gacMe
Hello carries the agent pubkey/sig/enroll-ticket and its config hash/generation; hello_ok returns the assigned id, gateway id, and seed flag. Adds push/propose/config-ack frames + the gateway-config capability, HashTunnels, the revoked error code, and renames HelloOK.Generation to SessionGeneration (json tag unchanged, byte-identical). Claude-Session: https://claude.ai/code/session_01EvsrY4Tq6uZCkAqm9gacMe
Load-or-create the agent keypair, derive agt_/gw_/tnl_ fingerprints over a Crockford base32 alphabet, and sign/verify the TLS channel-binding proof of possession. Claude-Session: https://claude.ai/code/session_01EvsrY4Tq6uZCkAqm9gacMe
quic-go wrapped so agent and gateway stay transport-agnostic; keepalive off and idle timeout above the liveness budget so the app heartbeat declares death. Claude-Session: https://claude.ai/code/session_01EvsrY4Tq6uZCkAqm9gacMe
compositeValidator (pubkey proof-of-possession + shared-token fallback), AgentStore allowlist with single-use/reusable tickets + revocation, per-agent scope enforced at bind, and gateway-authoritative config sync: reconcile the stored set onto a fresh session, push on drift, adopt promoted proposals deterministically. Also picks the QUIC vs per-conn data plane per session. Claude-Session: https://claude.ai/code/session_01EvsrY4Tq6uZCkAqm9gacMe
Sign the hello with the Ed25519 identity, walk the transport fallback ladder (auto), apply gateway-pushed config (merging agent-local fields by id), and propose local edits instead of pushing them. Claude-Session: https://claude.ai/code/session_01EvsrY4Tq6uZCkAqm9gacMe
Enroll+revoke, QUIC/per-conn round-trips, and the gateway-config seed/push/promote/drift flows alongside the legacy sync fallback. Claude-Session: https://claude.ai/code/session_01EvsrY4Tq6uZCkAqm9gacMe
Replace pf1:// with pxf://host:port/v1/pair/<token>#sha256:<fp>. The /v1/ segment versions the code's shape independently of the wire ProtocolVersion, and /pair/ is a role marker so a wrong-kind or stale-format link fails loudly instead of half-parsing. Bound the input before parsing (a pxf:// deep link is attacker-reachable) and add a FuzzParsePairingCode target. pxf doubles as the OS deep-link scheme, wired up next. Claude-Session: https://claude.ai/code/session_01EvsrY4Tq6uZCkAqm9gacMe
Register the pxf protocol (wails.json) so a clicked pairing link launches the app; pull the pxf:// URL out of os.Args before cobra parses it (else it looks like an unknown subcommand) and route it into pairing. A cold protocol launch stashes the link for the frontend to drain on mount (TakePendingDeepLink); a warm one emits a pxf:deeplink event and surfaces the window. SingleInstanceLock forwards a second launch's link to the running window rather than duplicating it. Regen bindings. Claude-Session: https://claude.ai/code/session_01EvsrY4Tq6uZCkAqm9gacMe
A public-port clash no longer fails the tunnel: bindLocked reassigns it to a policy-valid free port (scope/allowlist honored, else an OS ephemeral) and keeps the requested spec so a later reconcile is a no-op and the port stays put. A rapid same-agentID contest from two IPs — the fingerprint of a cloned identity key — is superseded as before but also flagged. Both surface through a bounded gateway event ring (Gateway.Events) the GUI polls, and TunnelSnapshot now carries the requested port so a reassignment is visible. TestConcurrentPortRace asserts the new both-come-up contract. Claude-Session: https://claude.ai/code/session_01EvsrY4Tq6uZCkAqm9gacMe
Expose the gateway's agent allowlist to the GUI over the analytics op envelope (dual-mode, no new IPC message type): ListAgents (allowlist joined with live sessions), IssuePairingCode (a pxf:// code carrying a fresh enrollment ticket), Revoke/Rename/SetAgentScope, and GatewayEvents (the conflict/auto-fix ring). Enrollment tickets now carry a tkt_ prefix so a pasted code is self-describing: SetupAgent routes a ticket to per-identity enrollment and a bare-hex token to legacy shared-token auth. That routing is required — the composite validator treats a failed enrollment as definitive and never falls back to the token. Regen bindings. Claude-Session: https://claude.ai/code/session_01EvsrY4Tq6uZCkAqm9gacMe
docs: rewrite README networking for QUIC/multi-TCP/multi-agent Bring the README up to the multi-agent-gateway branch: the three data planes (yamux mux, per-conn multi-TCP, QUIC), the head-of-line-blocking rationale and its burst-floor gate, the auto fallback ladder, per-agent Ed25519 identity/enrollment/scope/revoke, gateway-authoritative config, and self-healing reconnect. Adds mermaid diagrams and a Contributing section covering the CI gates and house style. @
wizard issues single-use/reusable enrollment tickets (IssuePairingCode) instead of the shared PairingCode; Agents roster gains per-agent drill-in and conflict cards; devmock adds the matching fleet fixtures.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
gosec found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turns proxyforward from a single-agent, single-TCP tunnel into a multi-agent gateway with a pluggable data plane and per-agent cryptographic identity. 39 commits; protocol and implementation kept in separate commits per the repo convention.
Networking — three data planes behind one
transport.Sessionautoladder (shipped default): QUIC → per-conn → mux, with cooldown that only demotes a rung once a lower rung succeeds (the unambiguous "UDP blocked" signal).TestBurstThroughputPerConn,TestBurstThroughputQUIC) guard against reintroducing cross-flow head-of-line blocking.Multi-agent gateway + identity
agt_<fingerprint>), not a shared secret.tkt_…, single-use or reusable) replace the bare shared token; per-agent scope (ports/tunnels), revoke, and rename.push_config/propose_config, deterministic, generation-hashed).Other
internal/bwcap).pxf://pairing scheme doubling as an OS deep link (click-to-pair, single-instance forwarding).&fleet=multi|old).Testing
go test ./...(unit + loopback e2e + goleak + burst floor + doc citations)cd frontend && npm run build(tsc + vite) — clean