Phase 1.5: Stage A federated master - full stack on box #2 (leader-lease, billing idempotency, snapshots, sweep, join-as-master v0)

[ehsan6sha]

Phase 1.5 — Federated master Stage A: full stack on box #2

Stage A of the federated-master roadmap: run the complete backend stack on a second box (operator-run), with fenced operator-confirmed failover, double-run-proof billing, and the first cut of the capstone installer.

Ships

• update-scripts/join-as-master.sh v0 — capstone installer first cut: detect-installed/adopt-or-halt (reuses Phase-1 kubo+cluster writer if present), dockerized services (healthchecks + restart: unless-stopped), idempotent re-runs, .env-persisted params (phase-common pattern).
• FM-2 billing idempotency (additive, flag-gated):
  • deduction idempotency key: deterministic reference_id = (user_id, UTC hour bucket) + UNIQUE index (CREATE UNIQUE INDEX CONCURRENTLY, additive migration) — two masters running the hourly cron can never double-deduct;
  • blockScanner deposit insert→credit in ONE transaction (closes the crash window that strands a recorded-but-uncredited tx).
• Cron leader-lease (Postgres advisory lock, flag CRON_LEADER_LEASE, default OFF) — exactly one master runs deductionJob/blockScanner.
• Signed pinset snapshots cron + restore verification (restore path from day one).
• Replication-status sweep — alert + re-pin anything below replication_min (early Phase-6 slice).
• Fenced failover + resync runbook.

Verify (e2e on test infra, real daemons)

• Stop master Bump golang.org/x/crypto from 0.24.0 to 0.45.0 in /pool-server #1 services → fenced flip → upload/billing tick succeed on master Bump qs from 6.14.0 to 6.14.1 in /pinning-webui #2; bring Bump golang.org/x/crypto from 0.24.0 to 0.45.0 in /pool-server #1 back → resync converges; cron ran on exactly ONE master; zero duplicate hourly_deduction per (user, hour); kill -9 between deposit-insert and credit recovers exactly-once; snapshot restores clean to a scratch cluster; sweep detects + re-pins a forced under-replicated CID; re-run installer on half-installed box → completes, no clobber.

Data-safety

Additive only: new index + new reference_id format; flags default OFF; no schema rewrites; existing balances/pins untouched; mixed-fleet invariant holds (non-updated providers unaffected).

Part of the decentralization roadmap (federated management tier; storage tier stays fula-ota edge).

🤖 Generated with Claude Code