[AbirAbbas]

What this changes

Until now one relay process was the single writer for each identity's chat records. That process decided who holds a chat, accepted every save and could not be replaced. This PR turns the relay into a small Raft group per identity, so coordination no longer depends on one writer:

• Three voters by default. On the hosted relay these are three Durable Object anchors. codeaf relay --store runs three local anchors. A change counts only after a majority commits it, and ownership reads are fresh majority reads. Losing one voter is fine. Losing two refuses coordination and keeps saved chats; it never creates a writable minority.
• Joined computers are members. An open, paired computer keeps a durable learner copy. Learners never add a vote, so pairing more machines or adding workers does not change the quorum.
• A joined computer can lead. codeaf cluster status shows committed membership and the current leader. codeaf cluster lead swaps one anchor's vote to this computer through joint consensus, transfers leadership to it and returns an elected read receipt. The relay then forwards requests to that leader.
• Fallback. If the leading computer sleeps, dies or switches sync off, the two remaining anchors elect a leader and restore the third anchor. The target is about 65 seconds in healthy conditions. A returning computer stays a learner and never takes leadership back by itself; it leads again only on another explicit request.
• Workers. A delegated worker gets an expiring grant for selected chats, plus conditional assignment claims in the log. It cannot approve pairing, read the identity vault, rotate keys, enroll devices or lead.
• Quiet when idle. Once every voter applies the idle barrier, a caught-up open home makes no HTTP requests, peer frames or client attempts, and the anchors schedule no recurring alarms. Presence sweeps run only while a viewer has the events stream open.

This is the coordination layer for the scenario where a phone leads and serverless dev boxes do the work. Both are follow-ups and neither is built here: there is no phone app yet, and no dev-box provisioning or headless worker loop. The manual says so plainly.

How it is built

• Core: go.etcd.io/raft/v3 v3.7.0 RawNode, inside a deterministic wrapper (internal/raftcore). It is vendored under third_party/raft with a single patch: per-node election randomness, so replays are exact.
• One core, two runtimes. The same Go core runs natively in codeaf relay and as a WASI reactor inside the hosted Worker's RaftMemberDO. A small JS shim supplies storage, alarms and transport. The hosted bundle is 2.47 MiB compressed, against Cloudflare's 3 MiB limit.
• Shared rules. Directory rules and records live in one Go state machine (internal/clusterstate), and both runtimes apply the same logic.
• Older clients still work through a v1 adapter. They pair, list, continue and take back. Requests up to the old 1 MiB HTTP limit are staged and committed atomically, so nothing is truncated. One request is refused: a blind force of a live chat without its observed holder. That gets the existing sentence "another device continued this chat first".
• Upgrade. On first use, the existing directory database is frozen and imported into the three anchors, and the source is kept. The upgrade is one-way: an older relay started on an upgraded store refuses to start, with codeaf: directory upgraded; use a current relay build.

Numbers

• Election fallback in the seeded simulator: p50 25 s, max 35 s. Ranked election windows keep split votes rare.
• Idle open home over ten minutes, two real homes: 0 HTTP requests, 0 frames and 0 client attempts. This holds as a learner, as leader and after a fallback, on both relays.
• bin/codeaf grows by 3,379,200 bytes, mostly the protobuf code upstream Raft needs. The size cap moves by exactly that measured weight, and PERF.md records it.
• Self-hosted directory content now has an 8 MiB default allowance per identity. Larger imports keep their measured size.

Known limits

• Three voters tolerate one unavailable member. Local anchors share one host. Hosted anchors share one provider and relay path.
• Phone leadership needs the phone app. Dev-box workers need provisioning and a run loop. Both are next.
• On the native relay, a brand-new identity's first device registration sometimes misses the client's 20 s deadline under heavy disk load. Its first write sets up three local anchors. A retry pairs in about a second, and normal pairing takes 1–3 s. Under the same load, dev failed this in 1 of 4 attempts and this branch in 3 of 4. I will file it as a follow-up.
• An open home's watch socket keeps the relay's platform ping/pong for advisory presence, at most 122 frames per socket per 10 minutes. HTTP requests, client attempts, application frames and anchor alarms stay at zero.
• Private task copies (opt-in), frame hints through the Raft relay, and two-computer fixes #1944 adds frame hints to directory records. After this lands, those hints have to be carried through the Raft command and snapshot path: the v1 adapter decodes strictly, so an unknown field is refused, not ignored. I will do that follow-up.

How it was checked

• Deterministic simulator with a Porcupine linearizability checker, run over 2,000 seeds. It injects crash cuts, partitions, delayed or duplicated delivery and leader loss.
• Native and hosted journey tests, run under local workerd, cover pairing, continue, take back, quit, sync off, revoke, joining as learner, leading, forwarding, fallback and the one-way upgrade.
• A whole-branch adversarial review found eleven issues; all are fixed, each with a regression test.
• Final full verification on the merged-with-dev head:
  • raftsim and clusterstate at 2,000 seeds;
  • make pr-ready, and the touched legs on CI;
  • internal/syncsetup in 590 s;
  • the relay, directory, pair, cellsync, remote and cluster packages;
  • the hosted suite (237/237) and every hosted Raft test file, with 0 skips;
  • the size, notice, law and manual gates.
• Two-computer journey with the built binary, on both the native and the hosted relay under local workerd. It covers pairing, continue, take back, cluster status, cluster lead, the leader stopping (anchors coordinate after 40 s native and 46 s hosted, with no hand-back when it returns), revoke and a fresh install.
• Regression found and fixed during that check: a joined home had shown a dead computer as online until its 90 s lease lapsed, so taking a chat back after the holder died without saying goodbye took up to 80 s; on dev it takes 3 s. Home presence follows the relay watch again, as on dev, and leases decide only ownership. Take-back now takes 3–4 s on both relays, and the C31 tests pin it at under 30 s.

Owner steps after merge

The hosted relay is not deployed by this PR. Its deployment configuration needs the RAFT_MEMBERS binding, the v3-raft-members migration and the new worker entry, all declared in relay/hosted/wrangler.toml. Deploy to staging first and run the two-computer journey there before production.

🤖 Generated with Claude Code