Phase 1: Cluster write-federation - support a 2nd trusted cluster writer

[ehsan6sha]

Goal / standalone deliverable

Today the cloud master is the only ipfs-cluster trusted writer (CRDT); the ~30 edge nodes are followers. If the master is down, no new pins can be committed to the shared pinset. This phase adds support for a second always-on cloud writer so writes survive the master going down. Existing pins, downloads, and the pinset are unaffected.

Ships (verifiable, standalone): stop the master ipfscluster.service -> a pin add via the 2nd writer still commits and propagates to followers; restart master -> CRDT reconverges, nothing lost.

Scope (this repo: fula-ota)

1. update-scripts/phase-1-setup-writer.sh (run on a NEW cloud box): install kubo + ipfs-cluster mirroring the master systemd unit (same CLUSTER_SECRET=sha256(clustername), CLUSTER_CLUSTERNAME, FOLLOWERMODE=false, allocator tag:group,pinqueue,reposize), bootstrap to the master, generate identity, print the new cluster peer ID + multiaddr, let CRDT replicate the pinset. No effect on the existing master. (Requires the master ipfs.service unit as the template + the new box address.)
2. update-scripts/phase-1-master-trust.sh (run on master; halts if NEW_WRITER_PEERID unset): back up /etc/systemd/system/ipfscluster.service + /uniondrive/ipfs-cluster/service.json; append the new peer ID to CLUSTER_CRDT_TRUSTEDPEERS (both the Environment= line and the ExecStart -e); daemon-reload + restart; verify peers + pinset count unchanged; print rollback. Idempotent (no-op if already trusted).
3. Edge init (docker/fxsupport/linux/ipfs-cluster/ipfs-cluster-container-init.d.sh): get_poolcreator_peerid() parses a new array field ipfs-cluster-trustedpeers from pools.fx.land/pools/{name} (falls back to the existing single ipfs-cluster-peerid -> fully backward-compatible); the jq block sets consensus.crdt.trusted_peers to the full array; followers keep the master tunnel plus a secondary bootstrap/tunnel to the 2nd writer so mesh connectivity survives the master being down.

Companion change (separate repo/PR)

functionland/mainnet src/server.js /pools/{name}: add an ipfs-cluster-trustedpeers array (keep ipfs-cluster-peerid for back-compat).

Data-safety (must hold)

• Additive only: trusted_peers is append; pebble/pinset/identity/secret never touched. Backups + documented rollback for every server-side edit.
• Edge change is backward-compatible: if the API returns only the legacy field, behavior is identical to today.
• Validate on ONE test edge device before fleet/OTA rollout.

Tests

• mainnet /pools returns the array and preserves the legacy field.
• Edge init parse: array -> trusted_peers=[all]; legacy single -> unchanged (shell/bats).
• phase-1-master-trust.sh dry-run: only appends, backs up, idempotent.

Open design point (will validate on a test device, not guess on prod)

Follower mesh connectivity when the master is down — confirm a 2nd bootstrap/tunnel to the new writer is required and works.

Operator inputs needed at run time

Master ipfs.service unit (template for the new writer), the new cloud box's reachable address, and the generated new-writer peer ID (fed to phase-1-master-trust.sh).

[ehsan6sha]

Companion follower-seam change opened: functionland/join-server#1 + PR functionland/join-server#2.

Note: the mainnet-pool-server services code lives in the GitHub repo **functionland/join-server** (the local dir is confusingly named mainnet, but its remote is join-server). The PR adds the ipfs-cluster-trustedpeersarray toGET /pools/{name}`, fully backward-compatible.