diff --git a/CHANGELOG.md b/CHANGELOG.md index f26be7e..287366f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,193 @@ There is no released version section until then; everything below lands in it. ### Added +- **Lists and text.** A field can be a list — an array edited by position — + or a text — a string typed into: `splice(id, key, index, count, content)` + and `push(id, key, content)`, on the storage and on a live Record. Two + nodes inserting at one spot both keep theirs, each writer's run whole + (Fugue, the rule Loro and list-positions use), a deleted element keeps its + place for the anchors into it, and a reset drops what is below it. A + `splice` op carries the cut ranges and either the anchor of a new run or + the writer's own run to extend — a keystroke at the end of one's run adds + no key to the record. The index is resolved inside the mutation queue; + past the length is `OUT_OF_RANGE`. Mixing kinds is refused locally as for + sets (`FIELD_KIND` with `kind: 'list' | 'text'`); between peers the kinds' + precedence decides — register, counter, set, list, text, tree (see Fixed). + Lists and texts cross a + schema version through `migrate()`, are indexed and queried as the arrays + and strings they read as, and converge over a mesh in the seeded fuzz. + A list is movable: `setAt(id, key, index, value)` and `move(id, key, + from, to)` are a value and a position register per element, so two nodes + moving one element concurrently leave it once, at the newer move's place, + and a move concurrent with a removal brings the element back. + `updateText(id, key, text)` diffs a text field against a whole new string + inside the mutation queue (Myers, UTF-16 units) and sends one splice per + hunk — what an editor hands over becomes the edits that merge. A text + carries rich text: `mark(id, key, start, end, name, value, { expand, many })`, + `unmark(id, key, start, end, name)`, `unmarkById(id, key, markId)` and + `spans()` — a mark is two anchors, an element and a side + each, so the same tree rule that places typed text decides whether it takes + the style (Peritext, Loro); per name the newest mark wins, and a `many` + mark stacks with every other of its name (comments), removable only by id. + `cursor(id, key, index, { side })` names a place that survives edits around + it and `index(id, key, cursor)` says where it is now. `@alexify/syncom/order` + (1.0 KB, off the main entry) mints fractional indexes — strings that + sort with `<`, with room between any two — for ordering records by a + position field. Browser budget 42 → 52 KB. +- **Trees.** A field can be a tree: `treeMove(id, key, node, parent, { after, + before })` puts a node under a parent at a place among its siblings and + `treeRemove(id, key, node)` takes it out with everything under it, on the + storage and on a live Record; the field reads as `[{ id, children }]`. A + `tree` op is one move — the node, its parent (`null` a root, `'$trash'` + removed) and a fractional index between its neighbours — and the record + keeps every move of every node; the tree is their fold in stamp order with + a move that would put a node under itself skipped (Kleppmann et al. 2021, + the rule Loro's movable tree follows), so two nodes moving concurrently so + that each would hang under the other agree that the older move stands. A + parent under the node itself rejects with `CYCLE` (new code), a sibling not + under the parent with `NOT_FOUND`, both decided inside the mutation queue; + a move to where the node is makes no op. `migrate()` carries a tree across + a schema version and counts `trees`. Browser budget 52 → 55 KB — the move + log and the fold, and `@alexify/syncom/order` now in the core for the + places. +- **History and read-at-version.** `history(id, { limit, before })` lists a + record's past as this node's log holds it, newest first, a page at a time + (`before`, an event id, pages back from it) — one version per op the + node applied, with the record as it read after it through the lens and + `afterRead` — and `get(id, { at })` reads one back: by event id (what this + node held after that log entry), by stamp (every op stamped at or below + it, the same on every node) or by `{ vector }` (every op it covers); a + collection's `get` takes `at` too. Both are a fold of the retained log + (`src/history.js`), exact while nothing was dropped and, after compaction, + from a reset stamped above `droppedTs`; `history()` reports `exact` and + `complete`, `get()` rejects with `HISTORY_GONE` (`reason: 'compacted' | + 'scrubbed' | 'merge' | 'epoch'`) or `INVALID_VERSION` (new codes). A + purge leaves the ops listed with their content gone; an encrypted record + has no past. The read hooks see `at`. New page: Versions and history. + Browser budget 55 → 56.5 KB. +- **Tags.** `tag(name)` — a name of the record-id form — names the storage + as it is now and keeps it on this node: the catalog of the moment plus a + copy of each record's file + taken before its first change after the tag — nothing per record until it + changes, one file after, in the newest tag only. `at(name)` reads records + as they were (`get`, `has`), `get(id, { at: { tag } })` too; `untag()` + moves to the next older tag the copies it needs; `tags()` lists them. A + copy carries its chain block and is checked like the file; a copy that + cannot be written leaves the record held (`UNPERSISTED`) and the tag + reading the file; a purge reaches every tag. A tag is the whole storage: + `at(name).list()`, `at(name).collection('Post').select()`/`count()`/`get()` + over the tag's own catalog (no `watch()`, no `$text`), and + `at(name).snapshot()` pages the storage as it was for another storage's + `mergeSnapshot()`. Tags do not replicate. New codes `TAG_EXISTS` and + `TAG_NOT_FOUND`; `tags.json` and `tags/` on disk. Browser budget + 56.5 → 58 KB. +- **Diff, restore, patches, the commit event.** `diff(id, from, to)` is the + list of patches — plain JSON in the terms of the write API: `put`, + `write`, `inc`, `add`, `remove` (as observed, or exactly the `removals` + named), `splice`, `treeMove`, `treeRemove`, `delete`, `purge` — that + turns the record at one version into the record at another, per field by + kind, with a `write` plus the other kinds made again whole when a + register is gone or a field changed kind; `applyPatches(id, patches)` + realizes a list in one turn of the mutation queue — every patch checked as + the method of its name checks its arguments (`src/args.js`, a `TypeError` + otherwise), every op rehearsed over a scratch record before the first + append, so a list lands whole or not at all; `restore(id, at)` leads + back with new ops, a version of its own that merges like any write. The + storage raises `commit` for every op of its own once durable — the op, + its seq, the source, the record before and after, the kinds — built only + while something listens: what an undo stack keeps. A text's marks are not + diffed. Browser budget 58 → 60.5 KB. +- **Undo and redo.** `@alexify/syncom/undo` — `createUndo(storage, { + mergeInterval, limit, scope, exclude, actor })` — is an undo stack over one + node's own writes, fed by the `commit` event: every commit adds the + patches that take it back, commits close together make one step, `mark()` + closes a step, a new write clears the redo stack. `undo()`/`redo()` apply + a step through `applyPatches()` — new ops that merge like any write — and + leave a field anyone else changed since alone (`skipped`), a whole-record + step with a conflict whole, unless `force`; an undone add revokes its own + tag alone. Events `push`, `pop`, `change`; `canUndo`, `canRedo`, `size`, + `clear()`, `stop()` (on `close()` too). The manager pins the log from its + oldest step. A subpath (6.0 KB alone, budget 6; its patch vocabulary, the + argument checks, Myers, Emitter and shape helpers are the core's and dedupe + in a bundle). +- **Branches.** `@alexify/syncom/branch` — `branch(parent, { path, from?: + { tag }, ...options })` forks a storage into a new directory under a new + identity, standing at the parent's state or at a tag of it (its snapshot + pages joined, its vector adopted, `branch.json` written); `merge(source, + { into })` hands either side what the other lacks as ops through + `applyRemote` — `src/outbound.js`, the selection a push makes, now shared + with the sync manager, asked for every stream, a third node's included — + or, when one of those streams is out of the source's log, as a join of + state (`joined: true`); `branchInfo()`, `drop()`. A branch is a storage + like any other; a merge into a parent that dials peers reaches them. New + code `BRANCH_EXISTS`: a path that holds a branch, or any files, is never + taken over; `drop()` removes a branch alone (a storage without + `branch.json` is a `TypeError`), and a fork that fails leaves nothing + behind. A subpath that bundles no engine (3.2 KB, budget 3.5). +- **Faster lists and text.** The index a sequence field is read through + (the Fugue walk) is built once per change and kept on the cached entry + (`entry.seqIndex`, never persisted): the resolver that places an insert + and the fold that reads the field back share it. The build itself parses + a run's tuple once — an unchanged run keeps its tuple between blocks — + and keeps one per-slot array instead of three per run; a reset prunes + through the same parse; a local write observes the record's clock + without rebuilding its metadata. Typing into a two-thousand-character + text: at the start 174 → 1,100 ops/s, anywhere 320 → 2,000, at the end + 6,400 → 9,500 (in memory, on one machine); in the cross-library scenarios, + whose text grows past ten thousand characters, 155 → 399, 224 → 541 and + 1,260 → 1,780 — every op still works over every run of the field, and the + record file is written whole. +- **The log knows what it dropped.** `storage.log.droppedTs` is the newest + stamp among the ops compaction dropped, or an open could not read — `0` + while nothing ever was — persisted in `index.json`; a record's retained + entries are its whole past only from a reset stamped above it. + `storage.log.pin(keep)` holds the segments a reader of the past needs, + like `retain` does for the sync manager, until the returned function is + called; `stats().log.droppedTs` reports it. +- **`compact(id)` and deferred record files.** `compact(id)` rewrites a + record as the ops its reading takes today — a `write` of the registers, + then one op per field of another kind (a text or a list as one run, a tree + as one move per node) — so what lies under it goes by the reset rule here + and on every peer and the record reads the same: the answer to a tree's + move log and a text's runs growing until a reset. It resolves to how many + ops it took, `0` for a record that is not there. `open({ defer: ms })` + lets a local write's record file and chain block wait up to `ms`: the log + append is durable at once, the cache and the catalog read the held entry, + one timer lands every record due, one file each under one block, `tag()` and + `close()` flush first, a peer's page or a replay that writes the record + settles its slot, and a crash before the flush replays the batch like a + failed file write. Off by default; one keystroke into a 10 KB text on + `node:fs`, 1,431 → 2,810 ops/s with the chain (on the memory backend the + file is not the cost, and nothing changes). `stats()` gains `held` and + `deferred`; a push and a branch merge walk the retained log without + copying it (`OpLog.scan()`). +- **A keystroke costs its own run, not the field.** A splice that only + appends to the writer's run carries the held index over (`extendIndex`) + instead of walking the field again, and the reading with it — applying a + character at the end of a thousand-character text runs at 155,000 ops/s + in `applyOp`; an insert elsewhere, or under a peer's concurrent run, still + walks. `spans()` is one sweep over the walk, O(characters + marks × + active) instead of O(characters × marks). A tree is folded once per block + however often it is read (a 20,000-move tree: one fold of about 100 ms, + then about 1 ms for a move's checks and the render), a `move()` reads one + list value instead of rendering the list, and `history()` runs the lens + and `afterRead` for the versions it hands out only. `bench/tree.js`, + `bench/tags.js` and `bench/history.js`. +- **Telemetry for versions.** `tag()`, `untag()`, `history()`, + `get(id, { at })` and `applyPatches()` — so `restore()` and `compact()` — + are spans: `syncom.tag`, `syncom.untag`, `syncom.history`, + `syncom.version`, `syncom.patches` (each op a `syncom.commit` child), none + carrying a record id, an origin, a url or a scope. `stats().log.pins` and + `stats().tags`. `tests/package/telemetry-events.test.js` fails when a log + event, a span or a metric of `src/` has no row in the telemetry guide. +- **`INVARIANTS.md`.** Every guarantee — `SYNC-n`, `OPS-n`, `LIST-n`, + `SEQ-n`, `TREE-n`, `RLE-n`, `BOUND-n`, `LOG-n`, `VER-n`, `TAG-n`, + `PATCH-n`, `UNDO-n`, `BRANCH-n` — names the tests that hold it, and + `tests/package/invariants.test.js` fails when an id names no test or a + named title is gone. The sync-engine suite gains a seeded walk under + hazards: a link that drops, comes back and may deliver a page twice, the + dialer closed and reopened mid-walk; after it the two nodes converge and + every record folds back to its present. - **Full-text search.** An entity names its `search: ['title', 'body']` fields and `open({ search: createSearch() })` gives the index — `@alexify/syncom/search`, a subpath under 1 KB, so a node that never @@ -64,15 +251,21 @@ There is no released version section until then; everything below lands in it. - **Benchmarks against nine libraries and a comparison with eighteen.** `pnpm bench:compare` measures syncom against Yjs, Automerge, Loro, TinyBase, PouchDB, RxDB, Fireproof, GUN and OrbitDB (insert, update, read, a converging - counter, a batch, replicating a store; in memory and on disk; a cold start), - each pair in a fresh process and checked before it is timed; the libraries + counter, a batch, replicating a store; in memory and on disk; a cold start; + typing into a text at its end, its start and anywhere, against the three + CRDT libraries), each pair in a fresh process and checked before it is + timed; the libraries live in `bench/compare`'s own `package.json`, never in syncom's devDependencies. `pnpm bench:size` measures browser bundles, `pnpm bench --save` records syncom's own benchmarks, `pnpm bench:report` turns all three into the charts of the docs and the README. `docs/comparison.json` (rendered - by `pnpm docs:comparison`) compares forty features, each cell decided from - the published source. New pages: Why syncom?, Compared with alternatives, - Benchmarks, Architecture. + by `pnpm docs:comparison`) compares forty-eight features, each cell decided + from the published source. New pages: Why syncom?, Compared with + alternatives, Compared with RxDB (300 rows from its code and docs), + Compared with CRDT libraries (Yjs, Automerge and Loro as CRDT libraries + only: 141 rows from their repositories, `docs/comparison-crdt.json`, with + the size of each whole package from `pnpm bench:size`), Benchmarks, + Architecture. - **`storage.repair(id)`.** A record whose file fails its integrity check (`INTEGRITY`) or is torn is rebuilt from this node's log: the retained @@ -343,6 +536,27 @@ There is no released version section until then; everything below lands in it. ### Security +- **Lists, texts and trees are bounded against a peer.** A peer's op or + block could cost a node more than it says, or split the mesh: a `[3e8]` + gap in a snapshot run was expanded element by element; any origin could + extend another's run, so the order of arrival decided the text; the + snapshot path had no bound on pending work, so one hostile block spent it + for every honest op after it; a 4,000-deep tree chain threw in the render + on the apply path (`PAGE_FAILED`, then `STUCK`); a forged place among + siblings made every local `treeMove()` next to it throw. Now a run holds + at most 1,048,576 slots and a field 4,194,304, tombstones included; a run + is extended by its own origin alone; pending work per field — a run whose + anchor has not arrived, a cut or an extension of a run not here, a + register of an element not here, a mark switched off before it arrived — + is bounded by `open({ sync: { pendingMax } })` (4,096 by default, + `Infinity` to disable), all of it on the op path and in a snapshot block + alike (`OP_REFUSED`, reason `bound`, with `detail`, `count` and `max`); a + place is a fractional index of at most 1,024 characters, a tree reads 512 + levels, and `treeMove()` refuses a move past either (`OUT_OF_RANGE`, with + `depth` or `reason: 'fi'`); two moves under one stamp tie by node id. A + payload the engine does not admit is never silent: a `TypeError` on the + local path, `OP_REFUSED` with `reason: 'malformed'` from a peer — the op + logged and relayed as a no-op, counted in `syncom.ops.rejected`. - **Scopes bound what a peer writes, not only what it declares.** An `update()` took the scope of its delta, so with a field-based `sync.scope` (`data.$type`, as the docs show) it was minted global and reached every @@ -395,6 +609,142 @@ There is no released version section until then; everything below lands in it. ### Fixed +- **A refused handshake push no longer closes the inbound half, and a failed + dial is retried.** A push the host refused during the handshake (a rate + limit, a page over `opsPerPush`) stopped the handshake before the + `changes` subscription: nothing came from the host until a reconnect or + `enableNode(url, false)` and `true`, and with no pages there was no sign of + life to push the rest at. The subscription opens anyway now. And a + `connect(node)` that threw left the node dead until `enableNode()` or + `removeNode()`: it stays in `peers()` with its error, and the next + `sync()` dials it again — no timer, as before. +- **The types say what the code does.** `browser.d.ts` re-exports every type + `index.d.ts` does — the CRDT branch's (`Patch`, `Version`, `Tag`, + `TextSpan`, the payloads…) never reached the browser condition — and a + consistency test keeps it so. The storage `error` event is a union with + `UNPERSISTED` (`ids`, no `op`); `sync` events carry `outOfScope`; + `applyOp` takes its `{ pendingMax }`; `OpLog.pins`, `SyncManager.pendingOps()`, + `UndoManager.listenerCount()`, the hello's `version` and a write context's + `removals`/`own` are declared; `Patch` takes `mark`; and the doc comments of + `skipped`, `pageSize`, `branch()`, `branchInfo()` and `drop()` are right. +- **A malformed register, counter or set payload is reported.** A peer's + `write` or `set` without a plain object, `inc` of no finite number, or + `add`/`remove` without an array applied as nothing and said nothing — the + sequence and tree ops already reported theirs. They are `OP_REFUSED` with + `reason: 'malformed'` now, counted, still logged and relayed as no-ops. +- **A live `Record` reads as `get()` after a change too.** It loaded through + the lens and the read hooks, but every change it followed — a write here, + an op from a peer — handed it the record as stored: a redacted field showed + its raw value after an unrelated `update()`, and a record of an older + schema version showed its old field names after an `inc()`. The refresh + now reads through the lens and `beforeRead`/`afterRead`; with either in + play it runs once the write is done, outside the mutation queue (a read + hook may write), in the order of the writes. +- **A snapshot block is held to the node's `sync.pendingMax`.** The op + path took a node's bound, the snapshot path the default 4 096 whatever + the option said: a node that raised it still dropped blocks a peer's ops + could have built there. `normalizeMeta` and `normalizeSeq` take the bound + a `merge` op is applied under. +- **Small edges.** A listener the `maxListeners` cap refuses is no longer + left registered behind the throw. `diff()` of the CRDT deltas takes a + `null` side and a non-collection under an array instead of throwing, and + no longer reads a string as a set of its characters. `$text` at a tag says + that a tag keeps no text index, not that the entity has no search fields. + `update()` with an invalid id is `INVALID_ID` like every other writer, and + its `NOT_FOUND` (and `swap()`'s) carries the `id`. `applyPatches()` names + an unknown action as such. `adopt` walks a peer's vector by its own keys. +- **A failed undo never inverts part of a record twice.** When a hook or a + validator refused a later op of one record's inverse, the earlier ops of + it had landed but the record stayed on the undo stack as the step left + it, so the retry took them back again (`views` 2 → 1 → 0). The record now + waits on the stack from where the landed ops left it — their way back is + on the redo stack — and the retry takes back the rest. +- **A counter stays a number every node reads alike.** One origin's sum had + no bound: past `Number.MAX_VALUE` it became `Infinity`, saved as `null`, + so a node that reloaded the record counted again from 0 while one holding + it in memory kept `Infinity` — a divergence a peer could cause on purpose. + Each origin's sum now stays within ±`Number.MAX_SAFE_INTEGER`: an `inc` + past it is a bound refusal (`detail: 'counter'`; `FIELD_KIND` locally, + `OP_REFUSED` from a peer), the same on every node; a snapshot slot past it + is not taken; and the sums add up in origin order, so a counter of + fractions no longer reads differently by the order of arrival. A merged + record this node did not hold is joined like any other rather than copied, + so data its meta does not back — a slot or block refused, a value with no + stamp — does not come with it. +- **`compact()` keeps what a record reads as.** It made a text again from its + string and dropped its marks, dropped a counter at `0`, and refused a text + or a list longer than one run (`FIELD_KIND`, `detail: 'run'`). The marks + in force are laid again now, in stamp order and with the expand rule their + anchors had (`marksIn` in src/sequence.js; patches take a `mark` action), a + counter at `0` is made again with an `inc` of 0, and a long field in runs + of `MAX_RUN` — `restore()` too. And one `splice()` that a writer's own run + cannot hold whole anchors a new run instead of overflowing it, as typing + past a full run already did. +- **`spans()` reads a name that has both plain and stacking marks.** A + stacking (`many`) mark newer than a plain mark of its name over the same + characters threw `marks[key].push is not a function` from `spans()` — a + read any peer could break with two marks — and a plain mark whose value + was an array took the stack's entries into the block's own value. On one + name the two now mix in stamp order: a plain mark takes the place of what + is there, a stack included, and a newer stacking mark starts a stack over + it. +- **An array spliced into a text, or a string into a list, is refused.** + `push(id, 'body', [1])` on a text field was let through the local kind + check; the list it made outranks a text, so the text was dropped — on + every node the op reached. A splice now names its kind by its content + and meets the field's before it is stamped (`FIELD_KIND`, the field left + whole), as a string on a list already did inside the engine. +- **A kind clash converges.** Two nodes making one field as different kinds + at once — a write carrying it as a register, an `inc`, an `add` or a + `splice` to it — stayed apart for good: each refused the other's op, while + a snapshot merge of the same two states let the register win. One rule + now, on the op path as in the merge, by the kinds alone: a register + outranks a counter, a counter a set, a set a list, a list a text, a text a + tree. Where the winner's op arrives, the loser's state is dropped; where + the winner is held, the loser's op is refused for that field + (`OP_REFUSED`). Locally a write of another kind is refused before it is + stamped (`FIELD_KIND`), for `write`, `set`, `inc`, `add` and `remove` as + for the sequence and tree ops. +- **The edges of branches, patches and tags.** `drop()` wiped any storage + handed to it; `branch()` opened over an existing storage and merged the + parent into it; a fork that failed left a half-made directory for the + retry to reuse; a merge asked only after the source's own streams, so a + third node's ops the parent had compacted past were skipped as gaps while + `merge()` reported success. A patch was checked more loosely than the + method of its name, so `applyPatches()` could mint nothing and answer with + a count; a `SyncomError` from a second op left the first committed, so a + `restore()` could land its reset without the rebuild; the `removals` a + patch named were kept by reference in the log; a `tags.json` line's name + became a path under `tags/` unchecked (such a line is dropped at load, + `tags.dropped` with `reason: 'name'`); `open('path')` opened the default + directory in silence — `open()` takes an options object, and anything + else is a `TypeError`. +- **A failed undo never inverts a record twice.** A throw on the third + record of a step — a hook refusing its inverse — put the step back whole, + though the first two were already inverted and their redo was lost; the + next `undo()` inverted them again. A step keeps its seqs per record: on a + failure the records taken back leave the step with their way back on the + redo stack, and the rest wait for another try. The own seqs the manager + kept are forgotten below its pin instead of growing for its life. +- **An unreadable `tags.json` no longer fails `open()`.** The tags are + auxiliary, yet a torn `tags.json` failed the open like `node.json` does. + The storage now opens without tags, keeps the file aside as + `tags.corrupt.json` for a repair by hand, warns `tags.unreadable`, + reports `stats().tags.corrupt` and refuses `tag()` with `CORRUPT`; every + record changed meanwhile has its file kept once under `tags/.pending/`, + and the next open with a readable file hands those copies to the newest + tag, so the tags a repair brings back read as they did. +- **A snapshot that carries a purge scrubs the joiner's log.** A node that + held a record's ops and then bootstrapped from a peer where the record was + purged kept the payloads in its log and served them to the peers behind + it. The bootstrap merge now scrubs every id the page purges, as a peer's + page does. +- **An op on a sealed record is skipped, not accepted.** A peer's op on a + record this node keeps `encrypted` was logged as applied, so a replay or + a fold of the log would have applied what the node never did; it is + flagged `skipped` now. Also: `ahead: 'hold'` accepts `retention.ops: 0` + (keep everything), and `repair(id)` counts a held op this node has since + applied. - **Offline writes past retention reach the host.** A node that wrote more ops than `retention.ops` before reaching a host had the oldest compacted, and the host took none of them — nor any later one of that stream — @@ -616,6 +966,40 @@ There is no released version section until then; everything below lands in it. ### Changed +- **A tag is read at its first use.** An open read every tag's catalog and + copy listing, so its time grew with the tags times their records; it reads + `tags.json` and the newest tag alone now (the one the write path copies + into), and an older tag's catalog and copies once, at its first read. + `at(name)` stays synchronous; its `has()` and `list()` are promises, and a + cursor at a tag waits for the catalog. A tag whose catalog cannot be read + is dropped at that first use, as `untag()` drops it (`tags.dropped` with + `reason: 'catalog'`), and its reader is told `TAG_NOT_FOUND`. +- **The field kinds' API and the versions API are modules.** `storage.js` + had grown back to 1,821 lines: `src/fields.js` (`inc` … `treeRemove`, + `spans`, `cursor`, `index`) and `src/versions.js` (`get(id, { at })`, + `history`, `diff`, `applyPatches`, `restore`, `compact`, `tag`, `untag`, + `at`) run over the core Storage lends its write paths, the facade one + delegate per method; the run-length content of a run is `src/rle.js`, out + of `src/sequence.js`; the write API's argument checks are `src/args.js`, + shared with `realize()` in `src/patch.js` and so with the undo subpath. No + public name, type or message changed. +- **Browser budget 60.5 → 65 KB, under a ceiling.** The bounds above + (62 KB), the argument checks and the rehearsal of `applyPatches()` + (62.5 KB), the two API modules (64.5 KB — the core's property names where + private fields minified away) and `src/rle.js` (65 KB; 64.6 measured, 65.0 + with the tags read at their first use). 64 KB was set as the v1 ceiling + with the second review and lifted the same day for the module split; a + raise still names its feature in `scripts/size.js`, and one of more than a + kilobyte is a design decision, taken in a plan. `PROTOCOL` 1 is the wire + as of the first publication: every op type and `packOps` slot added before + it bumps nothing. +- **Comparison: Jazz at 2.0.** The Jazz column described 0.20 (Jazz Classic); + it now describes 2.0.0-alpha.59, the version jazz.tools documents — a + server-merged relational database with queries, lenses, row policies and + queryable history, and without CoValues, per-change signatures, list or + text CRDTs or a purge. The method gains the rule for a project whose npm + `latest` is legacy, and the prose counts follow. + - **Browser budget 41 → 42 KB.** The fixes above — scopes checked on the record, own ops kept until delivered, `REWOUND`/`UNDELIVERABLE`, `UNPERSISTED`, scoped collections, canonical index keys — add about 1.2 KB diff --git a/CLAUDE.md b/CLAUDE.md index 61a2d1f..f4b5b36 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,8 +21,12 @@ wrpc.js / wrpc.d.ts the @alexify/syncom/wrpc subpath (src/wrpc.js) contracts.js / .d.ts the @alexify/syncom/contracts subpath (src/contract.js) — never back in the barrel idb.js / idb.d.ts the @alexify/syncom/idb subpath (src/idb.js) — the IndexedDB backend, browser-only search.js / search.d.ts the @alexify/syncom/search subpath (src/search.js) — the trigram text index behind `$text`; the core knows only the TextIndex contract +order.js / order.d.ts the @alexify/syncom/order subpath (src/order.js) — fractional indexes as strings, for ordering records +undo.js / undo.d.ts the @alexify/syncom/undo subpath (src/undo.js) — the undo stack over the commit event; requires patch.js, emitter.js only +branch.js / branch.d.ts the @alexify/syncom/branch subpath (src/branch.js) — fork a storage, merge either way; never requires storage.js index.d.ts the Node type surface (hand-written) -browser.d.ts re-exports index.d.ts, overrides the 3 async crypto fns +browser.d.ts re-exports index.d.ts, overrides the 5 async crypto fns +THIRD_PARTY_NOTICES.md the MIT notices of the algorithms adapted (Fugue, Yjs, list-positions…) and the files that carry them src/backend.js the fsi contract (isFsi/checkFsi), createMemoryFsi, withSync — no requires src/emitter.js the Emitter both halves extend — no requires src/logger.js createLogger, the duck-typed logger writer — no requires @@ -30,6 +34,22 @@ src/keys.js loadKeyPair — the on-disk key pairs (RSA for sealing src/scope.js GLOBAL_SCOPE, inScope, opInScope — the one definition of "in scope" — no requires src/signing.js canonical form, signOp, verifyOp — what a signature covers (the header + a salted payload digest) and the verdict on a peer's op src/shape.js isPlainObject, asJson, isRecordId, isStreamName, wellFormed — what enters is checked here +src/args.js spliceArgs, treeArgs, markArgs, setElements, jsonValue, isNodeId — the write API’s argument + checks, shared by Storage and realize() in patch.js — no requires +src/rle.js run-length content: a run's items and gaps, cut, append, merge, the canonical form under MAX_RUN — + no requires; sequence.js is the caller and re-exports what the tests read +src/sequence.js the sequence engine behind the list and text kinds: Fugue runs, the walk, local placement, pending ops, + pos/vals/marks registers, cursors, prune and join — requires vector.js and rle.js only +src/tree.js the tree kind: per node its stamped moves, the fold in stamp order with the cycle check (Kleppmann + 2021, Loro), MAX_DEPTH levels read, prune and join — requires vector.js and order.js (a place is a + fractional index or the move is malformed) +src/order.js fractional indexes over a 62-letter alphabet (Loro's rule, MIT) — no requires; the core uses it for a tree's places +src/history.js the past out of the log: a version (event id / stamp / vector) as a cut, the exact start above droppedTs, the fold +src/tags.js Tags — named versions kept locally: tags.json, a catalog per tag, copy-on-write record files in the newest tag +src/patch.js patches — the one vocabulary of diff()/applyPatches()/restore(): diffViews by kind, realize() to an op — requires shape.js, myers.js only +src/undo.js UndoManager — steps of inverse patches from `commit` events, refuse-to-overwrite, pins the log; the core does not use it +src/outbound.js outbound(log, { exclude, vector, scopes, own }) — the ops a receiver lacks, the first retained seq per own stream; a push and a merge both pick with it +src/branch.js branch/merge/branchInfo/drop — a fork opened through parent.constructor, merged as ops through applyRemote src/errors.js SyncomError (an operational failure, with a code) and typed() — no requires src/symbols.js COMMIT / REFRESH / SOURCE — the module-private seams between Storage and what it hands out — no requires src/options.js normalizeStorageOptions — open()'s plain options checked and defaulted (lock, cache, chain, skew…) @@ -39,6 +59,10 @@ src/catalog.js the record catalog: live ids, collections, indexed val src/query.js compileQuery (Mango predicate), compareRows, Cursor — reads through storage[SOURCE] src/shared-log.js isSharedLog — the reserved broker-log contract (wrpc BrokerLog shape), no consumer yet src/storage.js Storage — the facade: open sequence, the mutation queue, the public API, the `core` it lends +src/fields.js the API of the field kinds beyond registers — inc/add/remove, splice…treeRemove, spans/cursor/index — + over the lent core; Storage keeps a delegate per method +src/versions.js the versions API — get(id, { at }), history, diff/applyPatches/restore/compact, tag/untag/at — over the + lent core; Storage keeps a delegate per method src/entries.js EntryStore — record files under data/, the bounded cache, the catalog row each change leaves src/record.js Reference, Record, Collection — what the application holds; public API + the symbols only src/commit.js the one local write path: prepare → validate → persist → announce (`storage[COMMIT]`) @@ -60,13 +84,18 @@ bench/compare/ the cross-library comparison: its OWN package.json (th devDependencies), adapters/ (one per library), worker.js, size.js bench/report/ model.mjs (what every chart shows) + svg.mjs (README images); bench/report.js writes both bench/results/ compare.json, size.json, engine.json — the inputs of the charts, committed -docs/comparison.json the feature matrix; scripts/comparison.js renders it into docs/guide/comparison.md +docs/comparison*.json the feature matrices; scripts/comparison.js renders comparison.json into docs/guide/comparison.md, + comparison-rxdb.json into vs-rxdb.md, comparison-crdt.json into vs-crdt.md scripts/size.js bundle-size report with browser budgets docs/ VitePress site ``` ## Rules that bite +- **An adapted algorithm brings its notice.** Code or a rule taken from an MIT project is named in + `THIRD_PARTY_NOTICES.md` (project, commit, the syncom files that carry it) with the upstream notice. + Nothing is copied from json-joy (AGPL-3.0), Collabs or `fugue/fugue` (Apache-2.0) — those are read + as specifications only; the clones live in `../../open-source/local-first-alternatives/`. - **CommonJS everywhere in `src/`.** No `import`, no `type: module`, no build. - **No runtime dependencies.** Helpers go in `src/utils.js`. `scripts/size.js` fails at resolve time on any bare import reaching a browser bundle, so a @@ -134,8 +163,10 @@ docs/ VitePress site push path. - **`ops.js` walks own keys only.** `Object.keys` + `isKey` (`__proto__` is never a field) + `own()` for every lookup into `meta.clock`/`meta.counters`; - `normalizeMeta` deep-copies a peer's meta into the trusted shape. A - `for...in` or a bare `obj[key]` over peer-shaped data is a bug. + `normalizeMeta` deep-copies a peer's meta into the trusted shape (under the + node's `pendingMax`), and `mergeEntry` joins a side alone with an empty + entry rather than copy it, so data its meta does not back never comes + along. A `for...in` or a bare `obj[key]` over peer-shaped data is a bug. - **The chain is optional, the log is not.** `open({ chain: false })` leaves `core.chain` null: `persistLocal`/`seal` in `src/commit.js` and `#persistBatch` write `block: null`, `verify` is forced off, `validate()` @@ -148,13 +179,16 @@ docs/ VitePress site - **Live Records are weak.** `#records` holds `WeakRef`s (a `FinalizationRegistry` clears the slot); go through `#live(id)`, never `#records.get`. A Record the application still holds pins its cache entry; - one it dropped is collected with its pin. + one it dropped is collected with its pin. A write path hands `core.refresh` + the entry: with a lens or a read hook in play the Record is refreshed as + `get()` reads, after the write and outside the queue (`#emitUpdate`, in + order) — never await a read hook inside the queue. - **A purge is a scrub, not a delete.** `purge` wins at any stamp and a purged entry takes nothing but another purge or a merge (`applyOp` returns early). The payload leaves the log through `OpLog.scrub(id)`, - which rewrites files in place; `persist` in `src/apply.js` scrubs every id - of the batch whose entry is purged, after the batch is durable and before - the relay. Never log a payload for a purged id through another path. + which rewrites files in place; `persist` in `src/apply.js` and `mergeBatch` + in `src/bootstrap.js` scrub every id of the batch whose entry is purged, + after the batch is durable and before the relay. Never log a payload for a purged id through another path. - **`packOps` is 17 positional slots, both halves together.** `v` (13), `actor` (14), `sig`, `digest` and `nonce` (15–17) were appended; a reader of fewer slots ignores the rest, a reader of more takes an absent slot as @@ -165,21 +199,74 @@ docs/ VitePress site ops). Once the package is published, an older reader counts an unknown type as malformed and its stream stalls there — a new type, or a slot an older reader cannot take as null, is then a `PROTOCOL` bump (`src/protocol.js`). -- **Every write is an op, and a field is a register, a counter OR a set.** +- **Every write is an op, and a field is a register, a counter, a set, a list, a text OR a tree.** `insert`/`set` → `write`, `update`/`swap`/`record.save` → `set`, `inc` → - `inc`, `add`/`remove` → `add`/`remove`, `delete` → tombstone. Registers are - last-writer-wins by `(ts, origin)`; counters are per-origin sums; a set is - `meta.sets[field][elementKey][origin] = [added, removed]` — the newest add - ts and the newest add ts a remove observed — live while `added > removed`, - read sorted by element text (a same-origin re-add moves the stamp, so there - is no insertion order to keep). A `remove` carries the tags this node - observed (`observedTags`, built in `prepareOp` INSIDE the queue — outside it - a concurrent apply could add between the two); `applyBase` drops a slot - below `base` and clamps a surviving component below it, which is what makes - meta — not only data — order-free. Mixing two kinds on one field throws - locally (`FIELD_KIND`) and is refused (`OP_REFUSED` event) from a peer; the - snapshot merge lets a register, then a counter, win over a set. Do not add - a path that lets kinds meet — it cannot be made to commute. + `inc`, `add`/`remove` → `add`/`remove`, `splice`/`push`/`updateText` → + `splice`, `setAt` → `assign`, `move` → `move`, `mark`/`unmark`/`unmarkById` + → `mark`, `treeMove`/`treeRemove` → `tree`, `delete` → tombstone. Registers + are last-writer-wins by `(ts, origin)`; counters are per-origin sums, each + within ±`MAX_COUNT` and added up in origin order; a set + is `meta.sets[field][elementKey][origin] = [added, removed]`, live while + `added > removed`, read sorted by element text; a list or a text is + `meta.seqs[field]`, a block of `src/sequence.js`; a tree is + `meta.trees[field]`, a block of `src/tree.js` — how a block merges (Fugue, + the run-length content, the pending ops, the movable list, the marks; the + move log and its fold) is the header of its module, not repeated here. + What crosses modules: a `remove` carries the tags this node observed + (`observedTags`, built in `prepareOp` INSIDE the queue — outside it a + concurrent apply could add between the two); `applyBase` drops a slot below + `base` and clamps a surviving component below it, `applySeq` prunes a block + at once (`prune`, the same on either side of a reset) and a block's `ts` is + derived from what it holds, so a cut never bumps it — which is what makes + meta, not only data, order-free. Mixing two kinds on one field throws + locally (`FIELD_KIND`, `kindClash` in `src/commit.js`, before the stamp); + from a peer the kinds' PRECEDENCE decides, on the op path (`yieldTo` in + `src/ops.js`) as in the snapshot merge: a register, then a counter, wins + over a set, a set over a list, a list over a text, a text over a tree — + where the winner arrives the loser's state is dropped, where the winner is + held the loser's op is `OP_REFUSED`. The rule reads kinds alone, never + stamps, which is what makes it commute; do not add a path that decides a + clash another way. The wire: `splice` is code `/` with `{ cut, at | + run/from, content }` (`content` naming the kind even when empty), `assign` + `=` `{ elem, value }`, `move` `^` `{ elem, cut, at, value }`, `mark` `*` + `{ start, end, key, value, many? } | { off }`, `tree` `~` `{ node, parent, + fi }`. An index, a sibling or a range is resolved in `prepareOp` INSIDE the + queue (`resolveSplice`/`resolveMove`/`resolveMark`/`resolveTree` in + `src/commit.js`: `OUT_OF_RANGE`, `CYCLE`, `NOT_FOUND`; `resolveTree` folds + the tree ONCE and hands back `null` for a move to where the node is, and + `commitLocal` returns before `applyOp` on it). `sync.pendingMax` + (`PENDING_MAX`), `MAX_RUN` (`src/rle.js`), `MAX_FIELD`, `MAX_FI`, `MAX_DEPTH` + and `MAX_COUNT` bound what a peer can make a node hold or read, on the op path + and in `normalizeMeta` alike — a snapshot buys nothing an op could not. The + fuzz in `tests/suites/sync-engine.js` compares the blocks and the move logs + across nodes, not only what they read as. `compact(id)` (`src/versions.js`) + = `applyPatches(id, diffViews(null, {}, reading, kinds))` laid in runs of + `MAX_RUN`, then a text's marks as `mark` patches (`marksIn` in + src/sequence.js): a reset plus one op per field, one move per node — + convergent through the reset rule. +- **A payload the engine does not admit is never silent.** `applyOp` reports + `reason: 'malformed'` in `rejected` (`malformed()` in `src/ops.js`): on the + local path `refuse()` in `src/commit.js` throws a `TypeError` — the + arguments were checked, so this is a bug — and from a peer it is + `OP_REFUSED{reason:'malformed'}`, counted in `syncom.ops.rejected`, the op + still logged and relayed as a no-op (a dirk, never a gap). A `bound` + refusal carries `detail` (`pending`/`run`/`field`/`counter`), `count` and `max`. +- **A sequence block is never mutated in place; its index is cached by + identity.** Every change goes through `cloneSeq`, so a changed block is a + new object, and `seqIndexOf(entry, field)` in `src/ops.js` keeps + `buildIndex`'s result on the entry (`entry.seqIndex[field] = { seq, base, + index }`, carried by `cloneEntry` and into `EntryStore`'s cache, never into + the file) while `meta.seqs[field]` is the SAME object under the same base. + Read a sequence field through `seqIndexOf`, never through a bare + `buildIndex` on a path that runs per op; a keystroke carries the held index + over (`extendIndex`, on the `extended` hint `applySplice` returns — its + conditions are at its definition) and `render` keeps the reading on the + index, so never mutate an index or the array `render` hands back. + `parsedRun` memoizes a run tuple's parse by the tuple's identity (an + unchanged run shares its tuple between blocks; a tuple under another key is + parsed again). `latestClock(meta, true)` is the trusted path for an entry + this node holds — a peer's raw meta takes the untrusted one, which + normalizes first. - **Ops from one origin apply in seq order; seqs run per origin AND scope** (`origin@scope` vector keys), and ops born from a snapshot (`merge`, `advance`) run in their own `origin@~` stream. `applyRemote` skips @@ -190,7 +277,21 @@ docs/ VitePress site The storage opens its log with `sealApplied: true` and every write path calls `log.applied(from, to)` after `EntryStore.write`/`#persistBatch`; a roll then seals nothing from the oldest batch not yet applied onwards. A new path - that appends must mark what it applied, or the tail grows forever. The + that appends must mark what it applied, or the tail grows forever. Under + `open({ defer })` a local write stops at the append: `persistLocal` hands + the entry to `Storage#deferWrite` — `EntryStore.hold` (cache, catalog, + held), a slot per record with its positions, one timer — and + `#flushDeferred` lands every slot through `#land` (one block, the files) + and marks the positions applied; `tag()` and `close()` flush first, and + `#land` on any other path (a peer's page, a replay) settles a slot of a + record it wrote, so an older deferred entry is never laid over a newer + file. Never write a record file from a deferred slot outside `#land`. + `droppedTs` is the newest stamp among what `#compact` dropped or an open + could not read (an unreadable segment, an index without the field and a + floor above 1, no index and a floor above 1 — all `latestTs`): a reader of + the past is exact only from a reset stamped above it. `pin(keep)` holds + segments for such a reader and adds up; `retain` is one slot and the + SyncManager's. The append is the commit point: a chain block or record file that fails after it is never thrown through — `core.persist` (`Storage#land`) and `persistLocal` hold the entries in memory (`EntryStore.hold`, never @@ -298,6 +399,87 @@ docs/ VitePress site scope among them) at span start AND end and records a failure's `code` rather than the exception. A new `event:` name goes into the table in `docs/guide/telemetry.md` — the names are a surface. +- **A version is a fold of the retained log, exact only from a reset above + `droppedTs`.** The model is the header of `src/history.js`; the glue is + `src/versions.js` (`past`/`applied`/`present`), outside the queue: + `log.entriesOf(id)` filtered by `appliedHere` (skipped out, held in once + the node runs the version), folded with `applyOp`. With `droppedTs > 0` the + fold of everything retained must `deepEqual` the present (`present`) — the + counter straddle `repair()` also has — or it is `HISTORY_GONE compacted` / + `exact: false`; a stamp below a retained `merge` is `reason: 'merge'`, a + cut before a purge `'scrubbed'`. Never answer a version from the record + file or the catalog, and never fold from an entry below the exact start. +- **A tag is copy-on-write at the EntryStore seam.** The layout, the read + walk, the untag move-down and the lazy read of an older tag are the header + of `src/tags.js`. What crosses modules: record files are copied by + `EntryStore.#changing` (`before(id, file)`) ahead of + `write`/`writeSealed`/`remove`, into the NEWEST tag only, once per record, + only for a record the tag's catalog holds live — so the newest tag is read + at open and after every drop, an older one at its first use, and the sync + readers `hasAt`/`rowAt`/`catalogOf` come after `ensure(name)` (a Cursor at + a tag awaits its source's `ready`). `tag()` flushes deferred files first and + refuses (`UNPERSISTED`) while a record is held — its file is behind the + cache and a copy from memory would carry no block to verify; a record held + since the tag reads as its file, still the tag's state. An unreadable + `tags.json` never fails `open()` (`tags.corrupt.json`, `stats().tags`, + `take()` refuses `CORRUPT`, the copies wait under `tags/.pending/data/` for + the next good open); never rebuild `tags.json` from the disk — fsi lists + files, not directories. Every purge path goes through `core.scrub` (log + + tags) — never `core.log.scrub` alone. The error field is `tag`, never + `name` (it would overwrite `Error.name`). A view at a tag (`at(name)`, + `src/versions.js`) reads through `sourceAt`, a `storage[SOURCE]`-shaped seam + over the tag's catalog (`searches` false, `subscribe` throws), and pages + through `snapshotPage` with a core of the tag's catalog, entries and + position — the same `pageOf`, never a second page writer. +- **A patch is in the write API's terms, and the only way back is new ops.** + The vocabulary and the per-kind diff are the header of `src/patch.js`; + `realize(patch)` maps to `[type, payload]` for `commit()`. `applyPatches` + (`src/versions.js`) rehearses, then runs them in ONE `#run` turn; + `restore(id, at)` = `applyPatches(diff(id, undefined, at))` — never rewrite + the log or the file. A `remove` patch with `removals` goes out as given + (`prepareOp` takes `payload.removals` over `observedTags`), one with `own: + true` keeps this node's tags alone (`ownTags`): an undo revokes this node's + adds and leaves a peer's. The `commit` event (`announce` in src/commit.js) + is LOCAL ops only, built only when `listenerCount('commit') > 0`, with + `before`/`after` as copies and the kinds of both sides; a peer's op raises + `change`, never `commit`. +- **Undo is a subpath over `commit`, never inside the engine.** The model — + steps of inverse patches, refuse-to-overwrite, the exact and the loose + inversion — is the header of `src/undo.js`. What crosses modules: the + manager applies through `storage.applyPatches(id, kept, { source: token })` + and tells its own commits by the token; every `remove` it makes is `own: + true`; conflicts are read from `log.entriesOf(id)` after the step's + `lastSeq` (an `inc` or an `add`/`remove` by others commutes with the + inverse, `conflicting`), so it `log.pin`s from its oldest step. `kindsOf` + lives in ops.js (it needs the predicates) and `deepEqual` in shape.js, so + patch.js and the subpath ship without the op model — keep it that way. The + stack is memory only; a `purge` forgets the record on every step. +- **A branch is a storage; a merge is a push.** `src/branch.js` opens the + fork through `parent.constructor` (the subpath bundles no engine), never + copies the directory (a copied identity is `SELF`/`REWOUND`), gives it + `retention: { ops: 0 }` + `autoStart: false`, joins the parent's (or the + tag's) snapshot pages and adopts the vector. `merge` = `outbound(source.log, + { exclude: into.nodeId, vector: into.vector, own: source.nodeId, + allStreams: true })` → `into.applyRemote(page, { via: { url: null, node }, + adoptFrom: source })` — the one selection `#pushFromLog` uses too + (`src/outbound.js`; change one, change both: a push asks `first` of the + sender's own streams, a merge of EVERY stream, and joins as state when one + is out of reach). No-echo is `exclude` on `origin`/`from` plus `vectorHas`; + never add a third filter. `drop()` refuses a storage without `branch.json` + (`sweep` is the shared remover, used by a failed `branch()` too), and + `branch()` refuses a path that holds files: another storage is never taken + over. +- **Arguments are checked once, in `src/args.js`.** `spliceArgs`, `treeArgs`, + `markArgs`, `setElements`, `jsonValue` and `isNodeId` serve the public + methods of `Storage` AND `realize()` in `src/patch.js` (no requires — the + undo subpath ships with it; `tree.js` takes `isNodeId` from it), so a patch + never reaches the engine with a shape a call could not have. `prepareOp` + refuses a sequence or tree op on a field of another kind (`kindClash`) + before a stamp is minted or an index read — no placeholder payloads for + `applyOp` to refuse later — and `applyPatches` rehearses every op over a + scratch record (`rehearse` in `src/commit.js`) before the first append: a + restore lands whole or not at all. `copyPayload` copies `data`, `value`, + `content` AND `removals` — nothing of a caller's is kept by reference. - **`repair(id)` replays the log, it does not guess.** `Storage.#repair` starts at the first whole-record op the log retains for the id (`WHOLE`: `write`/`merge`/`delete`, or a `purge` even scrubbed), folds the rest with @@ -412,6 +594,15 @@ bootstrap stream never reaches the node it bootstrapped from. Coverage thresholds (95/95/90/95) are enforced in CI. They currently pass with headroom; do not lower them to land a change. +`INVARIANTS.md` names, for every guarantee (`SYNC-n`, `OPS-n`, `LIST-n`, `SEQ-n`, `RLE-n`, +`TREE-n`, `BOUND-n`, `LOG-n`, `VER-n`, `TAG-n`, `PATCH-n`, `UNDO-n`, `BRANCH-n`), the tests +that hold it; `tests/package/invariants.test.js` fails when an id names no test or a named +title is gone. Rename a test and its row together; a new +guarantee gets an id, a test and a row. Only plain-string titles are citable. The +hazard walk in `tests/suites/sync-engine.js` (drops, duplicate pages through +`tests/utils/memory-link.js`'s `duplicate` hook, a reopen) is the one that must stay +green whatever else moves. + ## Benchmarks and the comparison - **The alternatives are not devDependencies.** They live in `bench/compare/package.json`, installed with @@ -422,10 +613,12 @@ headroom; do not lower them to land a change. `` / `:end -->` in README.md and docs/guide/benchmarks.md and the SVGs in docs/public/benchmarks/; edit `bench/results/*.json` or the model, never the regions. The docs components (`docs/.vitepress/theme/components/Bench*.vue`) import the same `bench/report/model.mjs`. -- **The matrix is data.** Every cell of `docs/comparison.json` keeps the `source` it was decided from (and a - `detail` for a footnote left out of the table); `pnpm docs:comparison` renders it. A cell about another - project is decided from its published source and official docs, not its README; the syncom column from - this repository. Keep the prose of docs/guide/comparison.md and why.md in step with the counts. +- **The matrix is data.** Every cell of `docs/comparison.json`, `docs/comparison-rxdb.json` and + `docs/comparison-crdt.json` keeps the `source` it was decided from (and a `detail` for a footnote left out of + the table); `pnpm docs:comparison` renders them. A cell about another project is decided from its source and + official docs, not its README — the CRDT page from the clones in `../../open-source/local-first-alternatives/` + at the commits it names; the syncom column from this repository. Keep the prose of docs/guide/comparison.md, + vs-rxdb.md, vs-crdt.md and why.md in step with the counts. ## Documentation diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f762cee..8cb315f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -21,9 +21,10 @@ pnpm test:coverage # c8 over src/ — thresholds 95 lines / 95 statements / pnpm test:types # tsd against the .d.ts files pnpm lint # oxlint pnpm format # oxfmt (format:check in CI) -pnpm size # bundle-size report; the browser budget fails the run +pnpm size # bundle-size report; a browser budget exceeded fails the run pnpm bench # every script in bench/ (--save → bench/results/engine.json) pnpm bench:compare # against the alternatives (pnpm bench:compare:install first) +pnpm bench:size # the browser bundles of syncom and the alternatives (the same install) pnpm bench:report # charts and tables of docs/guide/benchmarks.md and the README pnpm docs:comparison # the matrix of docs/guide/comparison.md from docs/comparison.json pnpm docs:dev # the VitePress site (docs:build / docs:preview too) @@ -43,8 +44,10 @@ with `node --test --test-name-pattern 'swap'`. `package.json#browser` plus `src/index.browser.js`. A new split module needs an entry in both. - **A `.d.ts` change and its `tsd` test land together.** `index.d.ts` is the - Node surface; `browser.d.ts` re-exports it and overrides only the three - crypto functions that are async in the browser. + Node surface; `browser.d.ts` re-exports it and overrides only the five + crypto functions that are async in the browser (`calculateHash`, `encrypt`, + `decrypt`, `sign`, `verify`). Each subpath has its own `.d.ts` and + `tests/.test-d.ts`. - **Anything in `tests/` not named `*.test.js` is a helper** and is skipped by `node --test` discovery — that is why `tests/suites/`, `tests/utils/` and `tests/package/` work the way they do. @@ -54,9 +57,19 @@ with `node --test --test-name-pattern 'swap'`. `tests/wrpc.integration.test.js` only, which skips when it is missing; nothing under `src/` may `require` it. The adapter in `src/wrpc.js` duck-types the client it is handed. -- **A field is a register, a counter or a set.** `set`/`update` write - registers, `inc` writes counters, `add`/`remove` write sets, and no two of - them meet on one field — do not add a path that lets them. +- **A field is one kind: a register, a counter, a set, a list, a text or a + tree.** `set`/`update` write registers, `inc` counters, `add`/`remove` sets, + `splice`/`push`/`updateText` lists and texts (`setAt`/`move` a list, + `mark`/`unmark`/`unmarkById` a text), `treeMove`/`treeRemove` trees. A local + write of another kind is refused before it is stamped; between peers the + kinds' precedence decides — register, counter, set, list, text, tree — by + the kinds alone, the same on the op path and in a snapshot merge. Do not add + a path that decides a clash another way. +- **An adapted algorithm brings its notice.** Code or a rule taken from an MIT + project is named in `THIRD_PARTY_NOTICES.md` — the project, the commit, the + files that carry it — with the upstream notice. Projects under other + licences (json-joy, Collabs, `fugue/fugue`) are read as specifications only; + nothing is copied from them. ## Tests @@ -74,6 +87,16 @@ bundler honouring `package.json#browser` would. The sync engine is tested over client at the boundary (JSON across, resume from the last event id on reconnect), and once more over real wrpc in `tests/wrpc.integration.test.js`. +`INVARIANTS.md` names every guarantee — `SYNC-n`, `OPS-n`, `LIST-n`, `SEQ-n`, +`TREE-n`, `RLE-n`, `BOUND-n`, `LOG-n`, `VER-n`, `TAG-n`, `PATCH-n`, `UNDO-n`, +`BRANCH-n` — with the tests that hold it, and `tests/package/invariants.test.js` +fails when an id names no test or a named title is gone: rename a test and its +row together, and give a new guarantee an id, a test and a row. The seeded +hazard walk in `tests/suites/sync-engine.js` — dropped and duplicated pages, a +reopen mid-walk — is the one that must stay green whatever else moves. +`tests/package/telemetry-events.test.js` keeps every logged `event:`, span and +metric in the table of `docs/guide/telemetry.md`. + `tests/package/consistency.test.js` is the machine-checked half of the release checklist: it fails if `exports`, `files`, the `browser` map, the `.d.ts` files, the `scripts/size.js` rows or the docs keywords/version drift from @@ -94,9 +117,14 @@ Benchmarks are not run in CI: they measure a machine, not a change. ## Bundle size `pnpm size` prints one row per `exports` entry and enforces a min+gzip budget -on the browser entry. The budget is a **ratchet, not a target**: raise it -deliberately, in the same change that earns the bytes, and say why in a comment -next to the number. +on every browser entry — the main one and each subpath. The budget is a +**ratchet, not a target**: raise it deliberately, in the same change that earns +the bytes, and say why in a comment next to the number. The main entry's v1 +ceiling of 64 KB was passed on 2026-10-24 for the module split of the second +review (the comment in `scripts/size.js` names it; the budget is 66 KB now); +every raise still names its feature, and a raise of more than a kilobyte is a +design decision (an engine subpath, an injected kind), taken in a plan, not in +the number. ## Release checklist diff --git a/INVARIANTS.md b/INVARIANTS.md new file mode 100644 index 0000000..4322357 --- /dev/null +++ b/INVARIANTS.md @@ -0,0 +1,55 @@ +# Invariants + +What syncom guarantees, each with the tests that hold it. The ids are stable; a test named here +is a plain-string title in `tests/` (`tests/package/invariants.test.js` fails when one is renamed +without its row, or when an id names no test). The prose behind them is +[Guarantees and limits](docs/guide/sync.md#guarantees-and-limits) and +[Versions and history](docs/guide/versions.md). + +| id | guarantee | tests | +| --- | --- | --- | +| SYNC-1 | Two nodes that have applied the same set of ops hold the same records, whatever the order — apart from the documented limits: the counter/reset case, and an op a bound refused on one node and not on another. | `two nodes converge in both directions`; `concurrent writes converge: registers by last writer, counters by sum`; `a random workload over a mesh converges (seeded)`; `a random walk under hazards — drops, duplicate pages, a reopen — still converges (seeded)` | +| SYNC-2 | An op may arrive more than once; applying it again changes nothing. | `applying the same op twice is a no-op`; `applyRemote: in order per origin, duplicates and gaps skipped` | +| SYNC-3 | A peer's op is checked for its shape and its skew before anything reads it; a malformed one is counted, neither logged nor relayed, never observed into the clock. | `an op the engine cannot index by is malformed: counted, never logged`; `a stamp beyond maxSkew is refused and never observed into the clock` | +| SYNC-4 | A node's own sequence never repeats: the log is written before the op is announced, and a reopen recovers the seq from the log. | `reopen restores epoch, next, vector and entries`; `identity, seq and vector survive a reopen; what was logged but not written is replayed` | +| SYNC-5 | Logged means applied, eventually: the append is the commit point, a batch seals only once its records were written, and a write whose file failed is held and reported. | `survives a torn open file, a lost index and a duplicated roll`; `identity, seq and vector survive a reopen; what was logged but not written is replayed`; `a copy that cannot be written stops the write; a torn copy is evidence; a purge reaches the tags` | +| SYNC-6 | A disconnect and a reconnect resume from the last event id, both ways; a failed push leaves the peer behind and the next sign of life catches it up. | `a disconnect and reconnect resumes from the last event id, both ways`; `a page from the host triggers the recovery`; `pushes to one peer never overlap: handshake, relay and sync() take turns` | +| SYNC-7 | A node that cannot be resumed bootstraps from a snapshot; the joined state is transitive and never echoes back to the node it came from. | `a late joiner bootstraps from a snapshot once history is compacted`; `snapshot pages every entry, tombstones included; mergeSnapshot joins them`; `a page of nothing but echoes moves the cursor and announces nothing` | +| SYNC-8 | A host checks, every node decides: a skipped op is still logged and relayed; what stops at the host is out of scope, malformed, skewed, unsigned where required, or a throwing hook. | `beforeApply(false) skips an op here: logged, relayed, replay-safe`; `a skip is local: the op still reaches the peers behind this node`; `the hosting unit refuses what it must`; `scopes: a node syncs only what it asked for, plus everything global` | +| SYNC-9 | Every op is signed by its origin's key and the node id is that key; a signed op relays intact through a node that does not verify. | `signOp fills nonce, digest and sig; verifyOp answers every row of the table`; `a signed op relays intact through a node that does not verify, into one that requires` | +| OPS-1 | A field is a register, a counter, a set, a list, a text or a tree — one of them, never two. A local write of another kind is refused before it is stamped; when kinds meet from peers, the precedence register > counter > set > list > text > tree decides on the op path as in a merge — the loser's state goes where the winner arrives, the loser's op is refused where the winner is held — so every node ends with one kind whatever the order. | `kinds meeting on one field in any order leave the lowest rank everywhere`; `a field is a register, a counter or a set — never two of them`; `a list or a text is never also a register, a counter or a set`; `mergeEntry joins two sequence histories like replaying both; kinds keep their precedence`; `a tree moves through applyOp; concurrent moves that would make a cycle leave one in effect`; `splice, setAt, move, mark and treeMove on the wrong kind are FIELD_KIND whatever the index`; `an array spliced into a text, or a string into a list, is FIELD_KIND and leaves the field whole` | +| OPS-2 | A reset (write, delete) drops what is stamped below it, and the same predicate guards each op as it applies, so meta — not only data — agrees whatever the order. | `a reset drops set tags older than it and clamps the rest, so metas agree too`; `a reset drops counter entries older than it`; `a text converges: concurrent inserts, cuts, extensions and a reset, in any order` | +| OPS-3 | A purge wins at any stamp, forever; the payload leaves the log, the tags and every peer that learns of it. | `purge wins at any stamp, forever; a merge with a purged side is purged`; `after a purge nothing of the record is left, and nothing later revives it`; `a snapshot that carries a purge scrubs what the joiner held of the record` | +| OPS-4 | A peer-shaped meta reaches nothing it should not: own keys only, well-formed stamps alone. | `normalizeMeta rebuilds a merge meta from own keys and well-formed stamps alone`; `fields named after Object.prototype members are fields; __proto__ never is` | +| OPS-5 | Encrypted records are sealed for this node's key alone and do not sync. | `encrypted records stay local`; `encrypted writes go through the same hooks and stay local` | +| OPS-6 | Hooks run inside the operation and may refuse it; events never break a write. | `a throwing beforeWrite refuses the write: nothing stamped, logged or written`; `a change listener that throws does not fail the write it observed`; `a hook cannot write to the storage, but it can read` | +| LIST-1 | Two writers inserting at one spot both keep theirs, each writer's run whole: words typed apart never interleave. | `lists and texts converge: concurrent inserts and cuts, each writer’s run whole`; `three nodes type words apart; after the merge every word is whole` | +| LIST-2 | A concurrent move of one element lands it once, at the newer move's place; a value set follows the element. | `a list moves and assigns through applyOp; concurrent moves leave the element once`; `setAt and move keep an element’s name; a move names where it shows` | +| LIST-3 | A mark is two anchors picked by its expand rule; the newest per name wins, a stacking mark goes only by id, and the two mix on one name in stamp order. | `a mark rides on a text through applyOp; a list, a register or a counter refuses it`; `mark, unmark and spans: styles grow as their expand rule says, stack when many`; `a plain mark and a stacking mark of one name mix in stamp order, never throwing` | +| LIST-4 | A cursor names a place that survives the edits around it. | `a cursor names a place that survives the edits around it` | +| LIST-5 | A tree's moves fold in stamp order and a cycle never forms: of two concurrent moves that would make one, the older stands on every node. | `concurrent moves that would make a cycle: exactly one takes, the same one everywhere`; `treeMove and treeRemove keep a movable tree; a cycle is refused inside the queue` | +| LOG-1 | The log knows what it dropped: `droppedTs` is the newest stamp compaction took or an open could not read, and a pin holds a segment past retention. | `droppedTs is the newest stamp compaction dropped; a pin holds a segment past retention` | +| LOG-2 | A node's own ops are kept for the nodes it dials until they hold them; a host behind what the log retains is reported, not fed gaps. | `the outbox is bounded: beyond maxOutbox the backlog is dropped and the peer is behind`; `a push answers with the keys it touched; a second snapshot page carries no vector` | +| VER-1 | A version read from the log is exact while nothing was dropped, and from a reset stamped above `droppedTs` after; what cannot be given exactly is refused, never guessed. | `compaction: the past is exact from a reset above what was dropped, gone before it`; `a counter incremented across the reset compaction crossed makes the past inexact`; `a snapshot joined after a stamp hides what it carried: HISTORY_GONE merge` | +| VER-2 | An event id is what this node held; a stamp and a vector are what was written by then, the same on every node that holds the ops. | `an event id is what this node held; a stamp and a vector are what was written by then`; `history() lists one version per op, newest first; get(id, { at }) reads each back` | +| VER-3 | A tag reads records as they were: a copy lands in the newest tag only, once per record, before its first change, and moves down when the tag goes. | `a tag reads records as they were: after an update, a delete, an insert, through the lens`; `copies land in the newest tag only, once per record, and move down when it goes`; `queries, the list and a snapshot at a tag read the storage as it was` | +| VER-4 | A restore is new ops that lead back: a version of its own that merges like any write. | `restore() reads as the version, through every kind, and merges like any write`; `diff() names what turns one version into another, per field by kind` | +| VER-5 | An undo is new ops too: a field anyone else changed since is left alone, and an undone add revokes this node's adds alone. | `a field a peer changed since is left alone; a whole-record step needs force`; `an undone add revokes its own tag alone; counters undo across nodes` | +| VER-6 | A branch is a storage under a new identity; a merge hands over exactly what the other side lacks, and a second merge has nothing to hand over. | `a fork stands at the parent under a new identity; merges go both ways`; `a parent that compacted past the fork is joined as state; signatures are required and met` | +| RLE-1 | A run holds at most `MAX_RUN` slots and a field `MAX_FIELD`, tombstones included, on the op path and the snapshot path alike: a block past them is dropped, an op refused — a gap in a peer's block never costs more than its length says. | `a run and a field are bounded in slots: a gap bomb is dropped at the door, typing past the cap starts a new run`; `a hostile snapshot is dropped at the door: a gap bomb, pending work past the bound, a stranger extending a run` | +| SEQ-1 | A run is extended by its origin alone; another origin's extension is malformed, so what a field reads as never depends on the order two extensions arrived in. | `a run is extended by its origin alone: a stranger’s extension is malformed, whichever arrives first`; `a hostile snapshot is dropped at the door: a gap bomb, pending work past the bound, a stranger extending a run` | +| SEQ-2 | A pending cut is waited for once, and a mark switched off before it arrives fills in when it does: the same op twice, and a join of two pictures of one field, leave the block as it was. | `join and a repeated cut are idempotent: a pending cut is waited for once`; `a mark switched off before it arrives leaves one block, whichever came first` | +| BOUND-1 | Pending work — runs, cuts, extensions, registers of elements not here, marks switched off before they arrived — is bounded per field (`sync.pendingMax`) on every path a peer's state arrives by; a block past the bound is not taken as state, an op past it is refused with the count. | `the snapshot path is bounded like the op path: pending work past it, self-anchored runs, typed-off origins`; `registers of elements not here and marks switched off before they arrive count against the bound`; `applyOp takes { pendingMax }; a refusal names the bound, the count and the limit`; `open({ sync: { pendingMax } }) bounds a peer’s pending work; a refusal is reported with its count` | +| BOUND-2 | One origin's running sum of a counter stays within ±MAX_COUNT (`Number.MAX_SAFE_INTEGER`): an inc past it is a bound refusal on every node alike, a snapshot slot past it is not taken, data a merge's meta does not back is not kept, and the sums add up in origin order, so every node reads the same number. | `an inc past MAX_COUNT is a bound refusal in any order; the sums add up in origin order`; `a merge drops a counter sum past MAX_COUNT and data its meta does not back`; `an inc past a counter’s range is refused here, and from a peer, and a reopen reads the same` | +| TREE-1 | The fold orders moves by stamp, then by node id: two moves with one forged stamp fold the same on every node. A place is a fractional index, from an op or a snapshot, or the move is malformed. | `moves with one forged stamp fold the same on every node: ties go by node id`; `a place is a fractional index, or the move is malformed — from an op and from a snapshot alike` | +| TREE-2 | A tree reads `MAX_DEPTH` levels: what hangs deeper is folded, not shown, the same on every node, and a local move that would reach deeper — or a place with no room left between its neighbours — is refused. | `a tree reads MAX_DEPTH levels and no deeper: a 4 000-deep chain renders, folds and joins`; `treeMove refuses a node that would read below MAX_DEPTH levels, and a place with no room between its neighbours` | +| PATCH-1 | `applyPatches()` is all or nothing up to the first append: every patch is resolved and applied over a scratch record first, so an index out of range, a kind clash, a cycle or a type change refuses the batch whole — a `restore()` lands whole or not at all. | `applyPatches() is all or nothing before the first append: a patch out of range leaves the record as it was` | +| PATCH-2 | A patch is checked as the method of its name checks its arguments, and the data it carries is copied before anything keeps it: a malformed patch is a TypeError with nothing minted, and `removals` in the log never alias the caller's array. | `a patch is checked as the method of its name checks its arguments; removals are copied, not kept by reference` | +| BRANCH-1 | A merge hands the receiver every stream it can take or joins the source's state — a third node's stream the source compacted past the receiver included; it never reports success over a gap. | `a merge hands over every stream or joins as state: a third node’s ops the parent compacted are never skipped` | +| BRANCH-2 | `drop()` removes only what `branch()` made, `branch()` takes only a new directory, and a fork that fails leaves nothing behind. | `drop() takes a branch alone: a storage with no branch.json is refused and keeps its files`; `branch() opens a new directory: a path that holds files — another storage — is refused`; `a failed branch() leaves nothing behind: the next attempt starts clean` | +| TAG-1 | A tag's name is checked where it enters — the call and the file alike — before it becomes a path under `tags/`. | `a tags.json line whose name is not a record id is dropped at open, never read as a path` | +| UNDO-1 | A failed undo leaves the records it took back taken back, with their way back on the redo stack, and the rest on the undo stack for another try: a record is never inverted twice — nor a part of one, where the failure stopped inside it. | `a failed undo leaves what it took back taken back; the retry inverts the rest, never twice`; `a failed undo inside one record leaves its landed ops landed; the retry takes back the rest` | +| TAG-2 | A `tags.json` that cannot be read never blocks `open()`: the storage runs without tags, keeps the file aside as `tags.corrupt.json`, refuses `tag()` until it is repaired, and keeps every changed record's file under `tags/.pending/` so the tags a repair brings back read as they did. | `a torn tags.json never blocks open: no tags, the file kept aside, tag() refused until repaired` | +| TAG-3 | An open reads `tags.json` and the newest tag alone; an older tag's catalog and copies are read at its first use, once, and a tag whose catalog is gone is dropped then — the newest at open — as an untag would drop it (what the older tag needs of its copies moves there, the rest go), with a warning, its reader told `TAG_NOT_FOUND`; the next newest takes the write path. | `an older tag is read at its first use: the open reads the newest alone` | +| LOG-3 | Under `open({ defer })` a record file lags the log by the window at most, never the reads: the log is durable at once, the cache and the catalog read every write, `tag()` and `close()` flush first, a peer's write of the record lands it whole and settles what waited, and a crash before the flush replays the batch. | `writes within the window coalesce into one file and one block; reads see every write at once`; `tag() and close() flush first; the file is whole on disk after close`; `a peer’s op on a deferred record writes the file whole and settles what waited` | +| VER-7 | `compact()` rewrites a record as one op per field — a text or a list as one run (one per MAX_RUN elements), a tree as one move per node, a text's marks laid again — and the record and its spans read the same here and on every peer, with the history under it gone by the reset rule. | `compact() rewrites a record as one op per field: a text of many runs is one, a tree one move per node, the reading the same`; `compact() lays a text’s marks again, keeps a counter at 0, and lays a field longer than a run in several` | diff --git a/README.md b/README.md index 634b6dd..780a1f3 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,8 @@ Fast, zero-dependency distributed data sync engine 🌐 disk — or in the browser's origin-private filesystem — commits every write to a hash-linked log so tampering is detectable, and replicates every write to peers as an operation that converges without coordination: last-writer-wins -registers, per-origin counters, observed-remove sets, tombstones. The transport is injected — +registers, per-origin counters, observed-remove sets, lists and text that merge by +position, trees whose moves never form a cycle, tombstones. The transport is injected — [wrpc](https://wrpc.vercel.app) out of the box — and the package has no runtime dependencies. @@ -33,7 +34,7 @@ const main = async () => { record.on('update', (data, delta) => console.log('changed', delta)); - record.title = 'Ada Lovelace'; + record.name = 'Ada Lovelace'; // changed { name: 'Ada Lovelace' } await record.save(); await storage.inc(id, 'hits', 1); // a counter: increments add up across nodes @@ -73,17 +74,21 @@ npm install @alexify/syncom ### Bundle size -Measured by `pnpm size` on every CI run. The browser entry carries a budget -that fails the build when exceeded. +Measured by `pnpm size` on every CI run. Every browser entry carries a budget +that fails the build when exceeded; a subpath is off the main entry, so an app +pays for what it imports. | Entry | raw | min | min+gzip | budget | | ----- | ---:| ---:| --------:| ------:| -| main entry — browser (@alexify/syncom) | 296.0 KB | 122.3 KB | 41.9 KB | 42.0 KB | -| main entry — node (@alexify/syncom) | 279.9 KB | 121.3 KB | 41.6 KB | — | +| main entry — browser (@alexify/syncom) | 456.0 KB | 193.2 KB | 66.0 KB | 66.0 KB | +| main entry — node (@alexify/syncom) | 432.8 KB | 192.2 KB | 65.6 KB | — | | wrpc adapter — browser (@alexify/syncom/wrpc) | 3.4 KB | 1.4 KB | 0.7 KB | 2.0 KB | | contracts — browser (@alexify/syncom/contracts) | 3.7 KB | 1.6 KB | 0.8 KB | 1.0 KB | | idb backend — browser (@alexify/syncom/idb) | 3.7 KB | 1.7 KB | 0.8 KB | 2.0 KB | | search — browser (@alexify/syncom/search) | 4.2 KB | 1.7 KB | 0.9 KB | 2.0 KB | +| order — browser (@alexify/syncom/order) | 4.9 KB | 2.3 KB | 1.0 KB | 1.5 KB | +| undo — browser (@alexify/syncom/undo) | 36.1 KB | 16.3 KB | 6.1 KB | 6.5 KB | +| branch — browser (@alexify/syncom/branch) | 14.8 KB | 7.4 KB | 3.2 KB | 3.5 KB | ## Quick Start @@ -141,44 +146,50 @@ nodes converges. | Area | What you get | | --- | --- | -| **Storage** | Records with live properties and per-record deltas, schema-driven collections, compare-and-swap writes, counters and sets, encryption at rest (AES-256-GCM under an RSA-wrapped key), `purge()` for the right to be forgotten | +| **Storage** | Records with live properties and per-record deltas, schema-driven collections, compare-and-swap writes, encryption at rest (AES-256-GCM under an RSA-wrapped key), `purge()` for the right to be forgotten, `repair(id)` from the log | +| **Field kinds** | A field is a register, a counter (`inc()`), a set (`add()`/`remove()`), a list or a text (`splice()`, `push()`, `updateText()`; `setAt()`/`move()` on a list), or a tree (`treeMove()`/`treeRemove()`). A text carries marks (`mark()`, `unmark()`, `unmarkById()`, `spans()`), and `cursor()`/`index()` name a place that survives edits; `@alexify/syncom/order` mints fractional indexes for ordering records. See [Lists, text and trees](https://syncomjs.vercel.app/guide/lists) | | **Schema** | A metaschema model, any Standard Schema (zod, valibot, arktype), ajv or a compiled JSON Schema validates every write before it is stamped — locally and again on the hub. Versions ride on ops; an older record reads through a lens (migrations, then the validator's aliases and defaults); `migrate()` rewrites it. See [Schema and validation](https://syncomjs.vercel.app/guide/schema) | | **Queries** | `select({ orgId, status: { $in } }).order('name').limit(20).fetch()` over an in-memory record catalog: indexed queries with no I/O, unique keys, `watch()` for live views. See [Queries](https://syncomjs.vercel.app/guide/query) | | **Full-text search** | `select({ $text: 'quick fox' })`: fuzzy, typo-tolerant, TF-IDF ranked trigram search over an entity's `search` fields, best first, narrowed by any equality — the index is `@alexify/syncom/search` (under 1 KB), injected at `open({ search })`. See [Full-text search](https://syncomjs.vercel.app/guide/query#full-text-search) | | **Backends** | node:fs, OPFS, IndexedDB (`@alexify/syncom/idb`) or memory — five functions injected at `open({ fsi })`; `durable: true` for fsync, one instance per directory. See [Backends](https://syncomjs.vercel.app/guide/backends) | | **Replication** | Every write is an op in a persisted log; peers exchange ops over an injected link, resume from where they stopped, and bootstrap from a streamed snapshot when they cannot; scopes for partial replication | -| **Convergence** | Last-writer-wins registers, per-origin counters, observed-remove sets, tombstones with revival — any order, any number of times, same result | +| **Versions** | `history(id)` and `get(id, { at })` by event id, stamp or vector, folded from the log; tags of the whole storage (`tag()`, `at(name)`); `diff()`, `restore()`, `applyPatches()` and `compact(id)`, every way back new ops that merge; an undo stack (`@alexify/syncom/undo`) and branches that merge back (`@alexify/syncom/branch`). See [Versions and history](https://syncomjs.vercel.app/guide/versions) | +| **Convergence** | Last-writer-wins registers, per-origin counters, observed-remove sets, lists and text that merge by position (Fugue — each writer's run whole), trees whose concurrent moves never form a cycle, tombstones with revival — any order, any number of times, same result. A field two nodes make as different kinds at once ends as the same kind everywhere, by a fixed precedence; what a peer can make a node hold is bounded. See [Sync](https://syncomjs.vercel.app/guide/sync#registers-counters-sets-lists-and-text) | | **Transport** | `@alexify/syncom/wrpc` adapts a `WrpcClient` — over a WebSocket, or a WebRTC `PeerLink`'s `remote` from `@alexify/wrpc/webrtc`; `storage.sync.unit()` is the router unit a wrpc server, or a browser's `PeerHost`, mounts. Reconnects, resumable subscriptions, binary streams and rooms come from wrpc. See [WebRTC](https://syncomjs.vercel.app/guide/webrtc) | | **Integrity** | Every write appends a hash-linked block (`chain: false` turns it off for a hub); reads re-verify the record against the block that committed it and throw on a mismatch; `repair(id)` rebuilds a record from the log | | **Signed ops** | Every op is signed with the node's Ed25519 key, and the node id *is* that key: a peer verifies an op against its origin through any relay, a purge leaves a header that still verifies, `sync.signatures: 'require'` refuses unsigned ops. See [Signatures](https://syncomjs.vercel.app/guide/sync#signatures) | | **CRDT** | `diff` / `merge` / `apply` over plain objects — nested maps merge recursively, arrays are grow-only sets, numbers add so counters converge | | **Contracts** | `@alexify/syncom/contracts`: store a procedure in the chain by hash, load it anywhere, run it against a read-only view of storage; failures are recorded as blocks. A subpath, so the package root never runs a block's code | | **Portability** | One API over `node:fs` + `node:crypto` on the server and OPFS + Web Crypto in the browser, chosen by the `browser` export condition | -| **Hooks and events** | `open({ hooks })` runs your code inside a write, a read or a peer's batch — validate, stamp, redact, skip — and rejects with a typed `HookError` when it throws; `on()` events (`change`, `apply`, `peer`, `sync`, …) observe everything and can never break it. See [Hooks and events](https://syncomjs.vercel.app/guide/hooks) | -| **Observability** | Injected OpenTelemetry (`open({ telemetry })`): spans for local writes, applied batches, pushes, bootstraps and log rolls, plus outcome and lag metrics — nested under wrpc's spans, which carry the trace across the wire. See [OpenTelemetry](https://syncomjs.vercel.app/guide/telemetry) | +| **Hooks and events** | `open({ hooks })` runs your code inside a write, a read or a peer's batch — validate, stamp, redact, skip — and rejects with a typed `HookError` when it throws; `on()` events (`change`, `commit`, `apply`, `peer`, `sync`, …) observe everything and can never break it. See [Hooks and events](https://syncomjs.vercel.app/guide/hooks) | +| **Observability** | Injected OpenTelemetry (`open({ telemetry })`): spans for local writes, applied batches, pushes, bootstraps, log rolls, tags, history reads and patches, plus outcome and lag metrics — nested under wrpc's spans, which carry the trace across the wire. `stats()` reports the queue, the cache, the log, the tags and the files that lag the log. See [OpenTelemetry](https://syncomjs.vercel.app/guide/telemetry) | | **Scaling** | Hubs that dial each other, scopes as buckets, per-peer push limits, a reserved shared-log contract for the broker-backed hub. See [Scaling](https://syncomjs.vercel.app/guide/scaling) | | **Dependencies** | None. wrpc, metaschema and fake-indexeddb are devDependencies for the integration tests; everything is injected at runtime | ## Compared with alternatives -syncom sits where the CRDT libraries (Yjs, Automerge, Loro, Jazz), the local-first databases -(RxDB, PouchDB, TinyBase, Dexie Cloud, Fireproof), the peer-to-peer and event-sourced stores +syncom sits where the CRDT libraries (Yjs, Automerge, Loro), the local-first databases +(Jazz, RxDB, PouchDB, TinyBase, Dexie Cloud, Fireproof), the peer-to-peer and event-sourced stores (GUN, OrbitDB, Evolu, LiveStore) and the sync services (ElectricSQL, PowerSync, Zero, Triplit, InstantDB) meet. The [comparison](https://syncomjs.vercel.app/guide/comparison) puts all -nineteen in one table of forty features, each cell decided from the published source. In short: -syncom is a database that merges field by field, not a text CRDT. It syncs with no service to -run, its hosts validate and scope what peers send, and its log is verifiable and purgeable. It -has no text or list CRDT and no UI bindings, and it is a pre-release, not on npm yet. +nineteen in one table of forty-eight features, each cell decided from the published source. In short: +syncom is a database that merges field by field — and whose list, text and tree fields merge by +position, with marks, cursors and moves, like a text CRDT's — with tags, history, undo and +branches on top. It syncs with no service to run, its hosts +validate and scope what peers send, and its log is verifiable and purgeable. It has no editor +bindings and no UI bindings yet, and it is a pre-release, not on npm yet. RxDB, the closest of them, is compared [feature by feature](https://syncomjs.vercel.app/guide/vs-rxdb) -in 300 rows, from its code and every page of its docs. +in 300 rows, from its code and every page of its docs, and Yjs, Automerge and Loro +[as CRDT libraries](https://syncomjs.vercel.app/guide/vs-crdt) in 141 rows, from their code. ## Benchmarks Measured against Yjs, Automerge, Loro, TinyBase, PouchDB, RxDB, Fireproof, GUN and OrbitDB on the same scenarios: insert, update, read, a converging counter, a batch, replicating a store, -in memory and on disk. Each library runs in a fresh process, is used the way its documentation -shows, and is checked to give the right answer before it is timed. The method and the -interactive charts are on the [Benchmarks](https://syncomjs.vercel.app/guide/benchmarks) page. +in memory and on disk, and typing into a text, in memory. Each library runs in a fresh process, +is used the way its documentation shows, and is checked to give the right answer before it is +timed. The method and the interactive charts are on the +[Benchmarks](https://syncomjs.vercel.app/guide/benchmarks) page. @@ -188,7 +199,7 @@ interactive charts are on the [Benchmarks](https://syncomjs.vercel.app/guide/ben Insert a record, on disk: Add one record to a store of 1,000. Each library's own persistence on node:fs, every write awaited. ops/s, log scale, higher is better. Not shown: RxDB (Node.js disk storages are Premium).
-10 more scenarios +13 more scenarios Insert a record, in memory: Add one record to a store of 1,000. Nothing touches the disk. ops/s, log scale, higher is better. Not shown: GUN (needs its storage adapter to answer in Node.js). @@ -210,29 +221,35 @@ interactive charts are on the [Benchmarks](https://syncomjs.vercel.app/guide/ben Replicate a store, on disk: An empty replica pulls a store of 1,000 records, in process, through the library's own sync protocol, until every record reads back. Each library's own persistence on node:fs, every write awaited. records/s, log scale, higher is better. Not shown: RxDB (Node.js disk storages are Premium); Fireproof (syncs through a cloud or self-hosted gateway only); GUN (syncs over its WebSocket/WebRTC mesh only). +Type at the end of a text, in memory: Insert one character at the end of a text that merges by position — the common keystroke. Nothing touches the disk. ops/s, log scale, higher is better. Not shown: TinyBase (no text that merges by position); PouchDB (no text that merges by position); RxDB (no text that merges by position); Fireproof (no text that merges by position); GUN (needs its storage adapter to answer in Node.js); OrbitDB (no text that merges by position). + +Type at the start of a text, in memory: Insert one character at the start of the text — every insert anchors into the same spot. Nothing touches the disk. ops/s, log scale, higher is better. Not shown: TinyBase (no text that merges by position); PouchDB (no text that merges by position); RxDB (no text that merges by position); Fireproof (no text that merges by position); GUN (needs its storage adapter to answer in Node.js); OrbitDB (no text that merges by position). + +Type anywhere in a text, in memory: Insert one character at a random position of the text (seeded). Nothing touches the disk. ops/s, log scale, higher is better. Not shown: TinyBase (no text that merges by position); PouchDB (no text that merges by position); RxDB (no text that merges by position); Fireproof (no text that merges by position); GUN (needs its storage adapter to answer in Node.js); OrbitDB (no text that merges by position). +
**In memory** -| Library | Insert a record
ops/s | Update a field
ops/s | Read a record
ops/s | Increment a counter
ops/s | Insert 1,000 at once
records/s | Replicate a store
records/s | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | -| **syncom**
default, createMemoryFsi() · v0.0.1 | 16,200 | 17,100 | 972,000 | 16,700 | 16,000 | 25,200 | -| **syncom**
chain: false, createMemoryFsi() · v0.0.1 | 17,500 | 18,800 | 929,000 | 18,600 | 17,200 | 26,200 | -| Yjs
Y.Doc, a Y.Map per record · v13.6.33 | 235,000 | 382,000 | 3,680,000 | — | 275,000 | 99,700 | -| Automerge
Automerge.change, Counter · v3.5.0 | 512 | 1,990 | 10,600,000 | 1,950 | 8,960 | 20,900 | -| Loro
LoroDoc, a LoroMap per record · v1.16.4 | 31,300 | 412,000 | 313,000 | 349,000 | 83,100 | 105,000 | -| TinyBase
MergeableStore · v10.0.1 | 22,900 | 108,000 | 1,460,000 | — | 23,700 | 11,500 | -| PouchDB
pouchdb-adapter-memory · v9.0.0 | 20,300 | 8,310 | 30,100 | — | 74,000 | 17,800 | -| RxDB
memory storage · v17.6.0 | 16,700 | 50,400 | 281,000 | — | 2,220,000 | 64,100 | -| Fireproof
memory:// store · v0.24.19 | 310 | 377 | 726 | — | 22,300 | — | -| OrbitDB
documents, memory blockstore · v4.0.0 | 1,080 | 31.1 | 81.4 | — | 1,090 | 220 | +| Library | Insert a record
ops/s | Update a field
ops/s | Read a record
ops/s | Increment a counter
ops/s | Insert 1,000 at once
records/s | Replicate a store
records/s | Type at the end of a text
ops/s | Type at the start of a text
ops/s | Type anywhere in a text
ops/s | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| **syncom**
default, createMemoryFsi() · v0.0.1 | 15,200 | 17,200 | 845,000 | 17,100 | 15,800 | 25,100 | 1,800 | 388 | 537 | +| **syncom**
chain: false, createMemoryFsi() · v0.0.1 | 16,700 | 19,400 | 885,000 | 18,800 | 16,900 | 25,800 | 1,820 | 391 | 550 | +| Yjs
Y.Doc, a Y.Map per record · v13.6.33 | 235,000 | 382,000 | 3,680,000 | — | 275,000 | 99,700 | 346,000 | 525,000 | 30,400 | +| Automerge
Automerge.change, Counter · v3.5.0 | 512 | 1,990 | 10,600,000 | 1,950 | 8,960 | 20,900 | 23,700 | 21,600 | 20,600 | +| Loro
LoroDoc, a LoroMap per record · v1.16.4 | 31,300 | 412,000 | 313,000 | 349,000 | 83,100 | 105,000 | 619,000 | 444,000 | 311,000 | +| TinyBase
MergeableStore · v10.0.1 | 22,900 | 108,000 | 1,460,000 | — | 23,700 | 11,500 | — | — | — | +| PouchDB
pouchdb-adapter-memory · v9.0.0 | 20,300 | 8,310 | 30,100 | — | 74,000 | 17,800 | — | — | — | +| RxDB
memory storage · v17.6.0 | 16,700 | 50,400 | 281,000 | — | 2,220,000 | 64,100 | — | — | — | +| Fireproof
memory:// store · v0.24.19 | 310 | 377 | 726 | — | 22,300 | — | — | — | — | +| OrbitDB
documents, memory blockstore · v4.0.0 | 1,080 | 31.1 | 81.4 | — | 1,090 | 220 | — | — | — | **On disk** | Library | Insert a record
ops/s | Update a field
ops/s | Read a record
ops/s | Increment a counter
ops/s | Insert 1,000 at once
records/s | Replicate a store
records/s | | --- | ---: | ---: | ---: | ---: | ---: | ---: | -| **syncom**
default, node:fs · v0.0.1 | 1,520 | 1,340 | 958,000 | 1,480 | 1,690 | 5,830 | -| **syncom**
chain: false, node:fs · v0.0.1 | 2,060 | 3,080 | 951,000 | 3,060 | 2,740 | 5,820 | +| **syncom**
default, node:fs · v0.0.1 | 1,320 | 1,530 | 891,000 | 1,210 | 1,620 | 5,310 | +| **syncom**
chain: false, node:fs · v0.0.1 | 2,420 | 2,840 | 875,000 | 2,960 | 2,640 | 5,910 | | Yjs
y-leveldb · v13.6.33 | 19,200 | 22,400 | 3,690,000 | — | 155,000 | 53,700 | | Automerge
automerge-repo, NodeFS storage · v3.5.0 | 490 | 1,250 | 10,500,000 | 1,200 | 7,820 | 3,100 | | Loro
updates appended to a file · v1.16.4 | 19,500 | 43,300 | 285,000 | 42,600 | 66,900 | 108,000 | @@ -242,7 +259,7 @@ interactive charts are on the [Benchmarks](https://syncomjs.vercel.app/guide/ben | GUN
RAD on node:fs · v0.2020.1241 | 3.63 | 3.77 | 749 | — | failed | — | | OrbitDB
documents, LevelDB · v4.0.0 | 952 | 17.4 | 82.4 | — | 921 | 193 | -_Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 1 s samples (5 rounds for a batch or a replication), each library in a fresh process · 2026-10-22. Medians, rounded to three significant figures; — where the library has no such operation, no such storage, or no in-process sync, failed where it did not finish twice (the chart above says why)._ +_Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 1 s samples (5 rounds for a batch or a replication), each library in a fresh process · 2026-10-24. Medians, rounded to three significant figures; — where the library has no such operation, no such storage, or no in-process sync, failed where it did not finish twice (the chart above says why)._ @@ -268,7 +285,7 @@ _Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 1 s samples (5 roun | TinyBase
tinybase@10.0.1 | MergeableStore + IndexedDB persister + custom synchronizer | 56.4 KB | 23.8 KB | | | Yjs
yjs@13.6.33, y-indexeddb@9.0.12, y-protocols@1.0.7 | yjs + y-indexeddb + y-protocols/sync | 80.2 KB | 24.5 KB | | | PouchDB
pouchdb-browser@9.0.0 | pouchdb-browser | 117.8 KB | 40.4 KB | pouchdb-browser carries its HTTP adapter (the CouchDB client) with it | -| **syncom**
@alexify/syncom@0.0.1 | open() over OPFS + the wrpc link adapter | 124.1 KB | 42.6 KB | storage, schema, queries, the op log and the sync protocol are one entry | +| **syncom**
@alexify/syncom@0.0.1 | open() over OPFS + the wrpc link adapter | 195.0 KB | 66.6 KB | storage, schema, queries, the op log and the sync protocol are one entry | | RxDB
rxdb@17.6.0, dexie@4.4.6, rxjs@7.8.2 | rxdb + Dexie storage + replication | 282.3 KB | 90.1 KB | | | Fireproof
@fireproof/core@0.24.19 | @fireproof/core | 784.2 KB | 204.0 KB | | | OrbitDB
@orbitdb/core@4.0.0, helia@6.1.4, libp2p@3.5.1, @libp2p/gossipsub@15.0.23, blockstore-idb@4.0.1 | @orbitdb/core + helia + libp2p, IndexedDB blockstore | 1185.6 KB | 353.7 KB | createHelia brings Helia's default libp2p configuration with it | @@ -295,8 +312,8 @@ configuration needed. import { open, diff, merge } from '@alexify/syncom'; ``` -Crypto is asynchronous there (`calculateHash`, `encrypt` and `decrypt` return -promises), and storage lives in the origin-private filesystem — or in +Crypto is asynchronous there (`calculateHash`, `encrypt`, `decrypt`, `sign` and +`verify` return promises), and storage lives in the origin-private filesystem — or in IndexedDB through `open({ fsi: createIdbFsi() })` from `@alexify/syncom/idb`. `browser.d.ts` declares those signatures separately. A browser node dials the mesh through `@alexify/syncom/wrpc` like any other, and hosts for its peers over WebRTC. @@ -306,29 +323,35 @@ See [Browser](https://syncomjs.vercel.app/guide/browser). 0.0.1 — a pre-release, not published to npm, and there is no release until the package does what it is meant to; the API may still change. Everything on this page works today and is covered by the test suite, including -replication over real wrpc WebSockets. Known limits — a counter straddling a concurrent reset, a field written -as two kinds at once, encrypted records staying local, no proof of key possession at `hello` — are listed under -[Guarantees and limits](https://syncomjs.vercel.app/guide/sync#guarantees-and-limits). +replication over real wrpc WebSockets. Known limits — a counter straddling a concurrent reset, encrypted records +staying local, no proof of key possession at `hello` — are on the +[Sync](https://syncomjs.vercel.app/guide/sync#guarantees-and-limits) page. ## Documentation - [Why syncom?](https://syncomjs.vercel.app/guide/why) · [Compared with alternatives](https://syncomjs.vercel.app/guide/comparison) · + [Compared with RxDB](https://syncomjs.vercel.app/guide/vs-rxdb) · + [Compared with CRDT libraries](https://syncomjs.vercel.app/guide/vs-crdt) · [Benchmarks](https://syncomjs.vercel.app/guide/benchmarks) · [Architecture](https://syncomjs.vercel.app/guide/architecture) - [Getting started](https://syncomjs.vercel.app/guide/getting-started) - [Storage](https://syncomjs.vercel.app/guide/storage) · + [Schema and validation](https://syncomjs.vercel.app/guide/schema) · + [Queries](https://syncomjs.vercel.app/guide/query) · + [Lists, text and trees](https://syncomjs.vercel.app/guide/lists) · + [Versions and history](https://syncomjs.vercel.app/guide/versions) · + [Hooks and events](https://syncomjs.vercel.app/guide/hooks) · [CRDT deltas](https://syncomjs.vercel.app/guide/crdt) · [Blockchain log](https://syncomjs.vercel.app/guide/chain) · [Smart contracts](https://syncomjs.vercel.app/guide/contracts) - [Sync](https://syncomjs.vercel.app/guide/sync) · [wrpc transport](https://syncomjs.vercel.app/guide/wrpc) · [WebRTC](https://syncomjs.vercel.app/guide/webrtc) · - [Schema and validation](https://syncomjs.vercel.app/guide/schema) · - [Queries](https://syncomjs.vercel.app/guide/query) · + [Scaling](https://syncomjs.vercel.app/guide/scaling) +- [Browser](https://syncomjs.vercel.app/guide/browser) · [Backends](https://syncomjs.vercel.app/guide/backends) · - [Scaling](https://syncomjs.vercel.app/guide/scaling) · - [Browser](https://syncomjs.vercel.app/guide/browser) + [OpenTelemetry](https://syncomjs.vercel.app/guide/telemetry) - [API reference](https://syncomjs.vercel.app/reference/api) ## Contributing @@ -342,10 +365,11 @@ pnpm test:coverage # c8 over src/ pnpm test:types # tsd against the .d.ts files pnpm lint # oxlint pnpm format # oxfmt (format:check in CI) -pnpm size # bundle-size report; the browser budget fails the run +pnpm size # bundle-size report; a browser budget exceeded fails the run pnpm bench # every script in bench/ (--save writes bench/results/engine.json) pnpm bench:compare:install && pnpm bench:compare # against the alternatives pnpm bench:report # the charts and the tables of the docs and this README +pnpm docs:comparison # the comparison matrix from docs/comparison.json pnpm docs:dev # the VitePress site ``` diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md new file mode 100644 index 0000000..d25a727 --- /dev/null +++ b/THIRD_PARTY_NOTICES.md @@ -0,0 +1,178 @@ +# Third-party notices + +syncom has no runtime dependencies. A few of its algorithms were adapted from +other MIT-licensed projects, read at the commits named below; this file keeps +their notices, as the MIT license asks, and says which syncom files carry the +adaptation. Nothing was copied from json-joy (AGPL-3.0), Collabs or the +`fugue/` directory of the Fugue repository (Apache-2.0), or from the +`automerge-perf` editing trace (CC-BY-4.0): those were read as specifications +only. + +## Fugue — `fugue-simple` (reference implementation) + +Repository: https://github.com/mweidner037/fugue — `fugue-simple/src/index.ts` +(commit 31e74fe). The placement rule (`placeInsert`), the sibling order and +the iterative walk of `src/sequence.js` are adapted from it. The paper: +Matthew Weidner and Martin Kleppmann, "The Art of the Fugue: Minimizing +Interleaving in Collaborative Text Editing" (2023), arXiv:2305.00583. + +> The MIT License (MIT) +> +> Copyright (c) 2023 Matthew Weidner and Martin Kleppmann. +> +> Permission is hereby granted, free of charge, to any person obtaining a copy +> of this software and associated documentation files (the "Software"), to deal +> in the Software without restriction, including without limitation the rights +> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +> copies of the Software, and to permit persons to whom the Software is +> furnished to do so, subject to the following conditions: +> +> The above copyright notice and this permission notice shall be included in +> all copies or substantial portions of the Software. +> +> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +> SOFTWARE. + +## Yjs + +Repository: https://github.com/yjs/yjs (commit 0e0dc34). Three ideas of +`src/sequence.js` follow Yjs: deleted content collapsed to a length that +keeps the place (`ContentDeleted`; the run-length form is `src/rle.js`), consecutive inserts by one writer stored +as one run (`Item.mergeWith`), and an op whose dependency has not arrived +kept until it does (`pendingStructs`). The undo manager of `src/undo.js` +follows the rules of Yjs's `UndoManager` (`src/utils/UndoManager.js`): +changes tracked by origin, merged into one step within a capture timeout, +an undo that refuses to overwrite what others changed since. No code was +copied. + +> The MIT License (MIT) +> +> Copyright (c) 2023 +> - Kevin Jahns . +> - Chair of Computer Science 5 (Databases & Information Systems), RWTH Aachen University, Germany +> +> Permission is hereby granted, free of charge, to any person obtaining a copy +> of this software and associated documentation files (the "Software"), to deal +> in the Software without restriction, including without limitation the rights +> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +> copies of the Software, and to permit persons to whom the Software is +> furnished to do so, subject to the following conditions: +> +> The above copyright notice and this permission notice shall be included in +> all copies or substantial portions of the Software. +> +> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +> SOFTWARE. + +## Loro and loro-js + +Repository: https://github.com/loro-dev/loro (commit 75e4bf2). Three rules of +`src/sequence.js` follow Loro's: the movable list (a move cuts the slot the +element showed at, adds a one-element position run and points the element's +position register at it; `crates/loro-internal/src/state/movable_list_state.rs`, +`context/loro-js-movable-list.md`), the expand rule that picks a mark's anchor +sides and reverses under removal (`crates/loro-internal/src/container/richtext.rs`), +and the stable cursor as an element id with a side (`crates/loro-internal/src/cursor.rs`). +`src/order.js` is an adaptation of `loro-js/src/runtime/fractional-index.ts` +from bytes to a 62-letter alphabet. `src/tree.js` follows Loro's movable tree +(`crates/loro-internal/src/state/tree_state.rs`, `loro-js/src/runtime/tree-moves.ts`) +in its rules — a removal is a move under a trash node, siblings order by a +fractional index then by stamp, a move that would make a cycle is kept and +skipped — over the algorithm of Kleppmann, Mulligan, Beresford and Gomes, "A +highly-available move operation for replicated trees" (2021); the fold itself +is written for syncom's stamped move log and shares no code with Loro's +undo-do-redo. + +> MIT License +> +> Copyright (c) 2023 Loro +> +> Copyright (c) Loro Developers (loro-js) +> +> Permission is hereby granted, free of charge, to any person obtaining a copy +> of this software and associated documentation files (the "Software"), to deal +> in the Software without restriction, including without limitation the rights +> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +> copies of the Software, and to permit persons to whom the Software is +> furnished to do so, subject to the following conditions: +> +> The above copyright notice and this permission notice shall be included in +> all copies or substantial portions of the Software. +> +> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +> SOFTWARE. + +## Peritext + +Repository: https://github.com/inkandswitch/peritext (commit 89c162d). The +stacking marks of `src/sequence.js` — a mark flagged `many` that never +replaces another of its key and goes only by its own id — follow Peritext's +`allowMultiple` (`src/schema.ts`, `src/peritext.ts`). No code was copied. + +> MIT License +> +> Copyright (c) 2021 Ink & Switch LLC +> +> Permission is hereby granted, free of charge, to any person obtaining a copy +> of this software and associated documentation files (the "Software"), to deal +> in the Software without restriction, including without limitation the rights +> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +> copies of the Software, and to permit persons to whom the Software is +> furnished to do so, subject to the following conditions: +> +> The above copyright notice and this permission notice shall be included in +> all copies or substantial portions of the Software. +> +> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +> SOFTWARE. + +## list-positions + +Repository: https://github.com/mweidner037/list-positions (commit 383b844). +The run-length form of a sequence's content in `src/rle.js` (read by +`src/sequence.js`) — `["ab", 3, "c"]`, present items alternating with gap lengths — is the save +format of its `List` and `Text` (`src/lists/list.ts`, `src/lists/text.ts`), +and the rule that a writer extends its own run instead of anchoring a new one +follows `Order.createPositions` (`src/order/order.ts`). + +> The MIT License (MIT) +> +> Copyright © 2024 Matthew Weidner +> +> Permission is hereby granted, free of charge, to any person obtaining a copy +> of this software and associated documentation files (the "Software"), to deal +> in the Software without restriction, including without limitation the rights +> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +> copies of the Software, and to permit persons to whom the Software is +> furnished to do so, subject to the following conditions: +> +> The above copyright notice and this permission notice shall be included in +> all copies or substantial portions of the Software. +> +> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +> SOFTWARE. diff --git a/bench/branch.js b/bench/branch.js new file mode 100644 index 0000000..483fa1e --- /dev/null +++ b/bench/branch.js @@ -0,0 +1,57 @@ +/** + * Branches — what a fork costs (the parent's snapshot paged into a new + * storage, a new key pair with it) and what a merge back costs per op + * (the ops the other side lacks, through applyRemote in pages). + * + * Run with `node bench/branch.js` or `pnpm bench branch`. + */ + +'use strict'; + +const fs = require('node:fs/promises'); +const os = require('node:os'); +const path = require('node:path'); +const { bench } = require('./support/harness.js'); +const { open } = require('../src/storage.js'); +const { branch, merge, drop } = require('../src/branch.js'); + +const RECORDS = 200; + +const main = async () => { + console.log(`Branch benchmark — Node ${process.version}\n`); + const root = await fs.mkdtemp(path.join(os.tmpdir(), 'syncom-bench-branch-')); + try { + const parent = await open({ path: path.join(root, 'parent'), lock: false }); + for (let i = 0; i < RECORDS; i++) await parent.insert({ n: i, title: `record ${i}` }); + let forks = 0; + await bench( + `branch — fork at the present (${RECORDS} records)`, + async () => { + const child = await branch(parent, { path: path.join(root, `fork-${forks++}`), lock: false }); + await drop(child); + }, + { warmup: 2, measureMs: 1000 }, + ); + const child = await branch(parent, { path: path.join(root, 'child'), lock: false }); + const ids = await child.list(); + let n = 0; + await bench( + 'merge — 20 ops from the branch into the parent', + async () => { + for (let i = 0; i < 20; i++) await child.update(ids[i % ids.length], { n: n++ }); + await merge(child, { into: parent }); + }, + { warmup: 3, measureMs: 1000 }, + ); + await bench('merge — nothing to hand over', () => merge(child, { into: parent }), { warmup: 5 }); + await drop(child); + await parent.close(); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } +}; + +main().catch((error) => { + console.error(error); + process.exit(1); +}); diff --git a/bench/compare/adapters/automerge.js b/bench/compare/adapters/automerge.js index 999b765..ff7b18d 100644 --- a/bench/compare/adapters/automerge.js +++ b/bench/compare/adapters/automerge.js @@ -49,6 +49,15 @@ const wrap = (A, state) => { return ids; }, close: () => state.close(), + text: async () => { + await state.change((doc) => { + doc.text = ''; + }); + return { + insert: (index, chars) => state.change((doc) => A.splice(doc, ['text'], index, 0, chars)), + read: () => state.doc().text, + }; + }, }; }; @@ -70,6 +79,7 @@ const memory = { const A = local('@automerge/automerge'); return wrap(A, memoryState(A)); }, + text: (store) => store.text(), openReplica: async () => { const A = local('@automerge/automerge'); return wrap(A, memoryState(A, A.init())); diff --git a/bench/compare/adapters/loro.js b/bench/compare/adapters/loro.js index c4f0b9c..ea8c71a 100644 --- a/bench/compare/adapters/loro.js +++ b/bench/compare/adapters/loro.js @@ -82,6 +82,16 @@ const wrap = (L, doc, file) => { close: async () => { if (file) await file.close(); }, + text: () => { + const text = doc.getText('text'); + return { + insert: (index, chars) => { + text.insert(index, chars); + return commit(); + }, + read: () => text.toString(), + }; + }, }; }; @@ -91,6 +101,7 @@ const memory = { const L = local('loro-crdt'); return wrap(L, new L.LoroDoc(), null); }, + text: (store) => store.text(), }; const disk = { diff --git a/bench/compare/adapters/syncom.js b/bench/compare/adapters/syncom.js index 4e18fc6..837567f 100644 --- a/bench/compare/adapters/syncom.js +++ b/bench/compare/adapters/syncom.js @@ -52,6 +52,14 @@ const variant = (chain, memory) => { const units = new WeakMap(); return { open: async (dir) => wrap(await engine().open(options(dir))), + // A text is a field of a record: one record, typed into with splice(). + text: async (store) => { + const id = await store.storage.insert({ kind: 'text' }); + return { + insert: (index, chars) => store.storage.splice(id, 'body', index, 0, chars), + read: async () => (await store.storage.get(id)).body ?? '', + }; + }, openReplica: async (dir, source) => { if (!units.has(source)) units.set(source, source.storage.sync.unit({ access: 'public' })); const unit = units.get(source); diff --git a/bench/compare/adapters/yjs.js b/bench/compare/adapters/yjs.js index e67ac11..fc3c014 100644 --- a/bench/compare/adapters/yjs.js +++ b/bench/compare/adapters/yjs.js @@ -59,6 +59,10 @@ const wrap = (Y, doc, persist, close) => { }, flush, close, + text: () => { + const text = doc.getText('text'); + return { insert: (index, chars) => write(() => text.insert(index, chars)), read: () => text.toString() }; + }, }; }; @@ -68,6 +72,7 @@ const memory = { const Y = local('yjs'); return wrap(Y, new Y.Doc(), null, async () => {}); }, + text: (store) => store.text(), }; const disk = { diff --git a/bench/compare/scenarios.js b/bench/compare/scenarios.js index 9cb1e8d..8c55986 100644 --- a/bench/compare/scenarios.js +++ b/bench/compare/scenarios.js @@ -15,6 +15,11 @@ * - sync: a replica that has nothing pulls a store of SEED records and * can read every one of them; the time from the first message to * the last record in place + * - append, prepend, random: type one character into a text — at its end, + * at its start, anywhere (seeded) — the keystrokes of an editor; + * in memory only, and only the libraries with a text that merges + * by position take part (syncom's text field, Y.Text, Automerge's + * text, LoroText); the others show "—" * * Each runs on two lines: * @@ -78,6 +83,30 @@ const KINDS = [ unit: 'records/s', type: 'rounds', }, + { + id: 'append', + title: 'Type at the end of a text', + description: 'Insert one character at the end of a text that merges by position — the common keystroke.', + unit: 'ops/s', + type: 'ops', + lines: ['memory'], + }, + { + id: 'prepend', + title: 'Type at the start of a text', + description: 'Insert one character at the start of the text — every insert anchors into the same spot.', + unit: 'ops/s', + type: 'ops', + lines: ['memory'], + }, + { + id: 'random', + title: 'Type anywhere in a text', + description: 'Insert one character at a random position of the text (seeded).', + unit: 'ops/s', + type: 'ops', + lines: ['memory'], + }, ]; const LINES = [ @@ -92,6 +121,7 @@ const LINES = [ const SCENARIOS = []; for (const kind of KINDS) { for (const line of LINES) { + if (kind.lines && !kind.lines.includes(line.id)) continue; SCENARIOS.push({ id: `${kind.id}-${line.id}`, kind: kind.id, diff --git a/bench/compare/size.js b/bench/compare/size.js index 864409e..213fe90 100644 --- a/bench/compare/size.js +++ b/bench/compare/size.js @@ -11,6 +11,10 @@ * package instead would charge a modular library for what an app never * imports. Writes bench/results/size.json. * + * `cores` measures the CRDT libraries a second way, for /guide/vs-crdt: the + * whole package entry of each (`export *`), with no persistence or sync + * around it — what a page that only merges documents ships. + * * Run: pnpm bench:size (after pnpm bench:compare:install) */ @@ -134,6 +138,26 @@ const ENTRIES = [ }, ]; +// The whole package of each CRDT library and syncom's whole entry, nothing around them. +const CORES = [ + { id: 'syncom', name: 'syncom', entry: "export * from '@alexify/syncom'", packages: ['@alexify/syncom'] }, + { id: 'yjs', name: 'Yjs', entry: "export * from 'yjs'", packages: ['yjs'] }, + { + id: 'automerge', + name: 'Automerge', + entry: "export * from '@automerge/automerge'", + packages: ['@automerge/automerge'], + wasm: ['@automerge/automerge/dist/automerge.wasm'], + }, + { + id: 'loro', + name: 'Loro', + entry: "export * from 'loro-crdt'", + packages: ['loro-crdt'], + wasm: ['loro-crdt/browser/loro_wasm_bg.wasm'], + }, +].map((spec) => ({ ...spec, source: `${spec.entry};` })); + const versionOf = (name) => { if (name === '@alexify/syncom') return require(path.join(ROOT, 'package.json')).version; return JSON.parse(fs.readFileSync(path.join(MODULES, name, 'package.json'), 'utf8')).version; @@ -214,7 +238,13 @@ const main = async () => { libraries.push(row); console.log(`${spec.name.padEnd(12)} ${kb(row.gzip).padStart(10)} min+gzip ${kb(row.min).padStart(10)} min`); } - const report = { generatedAt: new Date().toISOString(), esbuild: esbuild.version, libraries }; + const cores = []; + for (const spec of CORES) { + const row = await measure(spec); + cores.push(row); + console.log(`core ${spec.name.padEnd(7)} ${kb(row.gzip).padStart(10)} min+gzip ${kb(row.min).padStart(10)} min`); + } + const report = { generatedAt: new Date().toISOString(), esbuild: esbuild.version, libraries, cores }; fs.mkdirSync(path.dirname(OUT), { recursive: true }); fs.writeFileSync(OUT, `${JSON.stringify(report, null, 2)}\n`); console.log(`\nWrote ${path.relative(ROOT, OUT)}`); diff --git a/bench/compare/worker.js b/bench/compare/worker.js index 43b6de3..66bedee 100644 --- a/bench/compare/worker.js +++ b/bench/compare/worker.js @@ -20,6 +20,16 @@ const { getScenario, SEED, BULK, recordOf } = require('./scenarios.js'); const { measureOps, measureRounds } = require('./measure.js'); const FIELDS = ['title', 'done', 'priority', 'owner', 'hits']; +const TEXT_KINDS = new Set(['append', 'prepend', 'random']); + +// A seeded generator, so every library types at the same random places. +const lcg = (seed) => { + let state = seed >>> 0; + return () => { + state = (Math.imul(state, 1664525) + 1013904223) >>> 0; + return state / 0x100000000; + }; +}; // The answer goes to the file bench/compare.js names (stdout belongs to the // libraries' chatter), or to stdout when the worker runs by hand. @@ -109,6 +119,23 @@ const measureKind = async (adapter, scenario, line, store, ids, dir, method) => method, ); } + if (TEXT_KINDS.has(kind)) { + if (!line.text) return { unsupported: 'no text that merges by position' }; + const text = await line.text(store); + await settle(text.insert(0, 'ab')); + await settle(text.insert(1, 'c')); + const read = await settle(text.read()); + if (read !== 'acb') fail(adapter, scenario, `a text reads "${read}" after inserting "ab" then "c" at 1`); + let length = 3; + const random = lcg(SEED); + const at = + kind === 'append' ? () => length : kind === 'prepend' ? () => 0 : () => Math.floor(random() * (length + 1)); + return measureOps(() => { + const wait = text.insert(at(), 'x'); + length++; + return wait; + }, method); + } if (kind === 'sync') { if (!line.replicate) return { unsupported: line.noSync ?? 'no in-process sync' }; return measureRounds( diff --git a/bench/history.js b/bench/history.js new file mode 100644 index 0000000..8dfe7f5 --- /dev/null +++ b/bench/history.js @@ -0,0 +1,37 @@ +/** + * History: the past read out of the log — `history()` over a record with + * two thousand ops, a version a thousand ops back, and the diff against it + * — over the memory backend. The fold is the whole record's; `limit` bounds + * what is handed out, not what is folded. + * + * Run with `node bench/history.js` or `pnpm bench`. + */ + +'use strict'; + +const { bench } = require('./support/harness.js'); +const { createMemoryFsi } = require('../src/backend.js'); +const { open } = require('../src/storage.js'); + +const main = async () => { + console.log(`History benchmark — Node ${process.version}\n`); + const storage = await open({ path: '/bench', fsi: createMemoryFsi(), logger: false }); + const id = await storage.insert({ n: 0, title: 'a' }); + for (let i = 1; i <= 2000; i++) await storage.update(id, { n: i }); + await storage.splice(id, 'body', 0, 0, 'hello world'); + const all = await storage.history(id, { limit: Infinity }); + const back = all.versions[1000].eventId; + await bench('history(id, { limit: 50 }) — a record with 2 000 ops', () => storage.history(id, { limit: 50 })); + await bench('history(id, { limit: Infinity }) — every version materialized', () => + storage.history(id, { limit: Infinity })); + await bench('get(id, { at: eventId }) — 1 000 ops back', () => storage.get(id, { at: back })); + await bench('get(id, { at: { vector } }) — the present, folded', () => + storage.get(id, { at: { vector: storage.vector } })); + await bench('diff(id, at) — against 1 000 ops back', () => storage.diff(id, back)); + await storage.close(); +}; + +main().catch((error) => { + console.error(error); + process.exitCode = 1; +}); diff --git a/bench/report/model.mjs b/bench/report/model.mjs index 04372ae..ba46ab2 100644 --- a/bench/report/model.mjs +++ b/bench/report/model.mjs @@ -230,6 +230,38 @@ export const sizeModel = (size, only = null) => { }; }; +/** + * The CRDT libraries' whole packages next to syncom's whole entry, nothing + * around them (bench/compare/size.js `CORES`): what /guide/vs-crdt shows. + */ +export const coreSizeModel = (size, only = null) => { + const kb = (bytes) => `${(bytes / 1024).toFixed(1)} KB`; + const rows = pick(size.cores ?? [], only) + .map((library) => ({ + id: library.id, + label: library.name, + sublabel: library.entry, + value: library.gzip / 1024, + tone: TONES[library.id], + details: [ + `${kb(library.gzip)} min+gzip`, + `${kb(library.min)} minified${library.wasm ? `, ${kb(library.wasm)} of it WebAssembly` : ''}`, + library.packages.join(', '), + ], + })) + .sort(byValueAsc); + return { + id: 'cores', + title: 'Bundle size of the whole package', + subtitle: + 'Every export of the package (export *), with no persistence or sync around it (esbuild, minified), ' + + "min+gzip, WebAssembly included; syncom's entry carries its storage, queries and sync too. Lower is better.", + unit: 'KB', + scale: 'linear', + groups: [{ id: 'all', label: null, rows }], + }; +}; + // ─── syncom's own hot paths ───────────────────────────────────────────────── /** The scripts of `pnpm bench`, in the order the toggle lists them. */ @@ -237,6 +269,11 @@ export const ENGINE_SUBJECTS = [ { id: 'storage', title: 'Storage' }, { id: 'search', title: 'Search' }, { id: 'apply', title: 'Apply' }, + { id: 'sequence', title: 'Lists and text' }, + { id: 'tree', title: 'Trees' }, + { id: 'history', title: 'History' }, + { id: 'tags', title: 'Tags' }, + { id: 'undo', title: 'Undo' }, { id: 'sync', title: 'Sync' }, { id: 'log', title: 'Op log' }, { id: 'chain', title: 'Chain' }, @@ -249,6 +286,13 @@ const SUBJECT_NOTES = { storage: 'The public API end to end on node:fs: a write is an op appended to the log, a record file and a block.', search: 'The @alexify/syncom/search trigram index: a put per write, $text answered from the catalog in rank order.', apply: "A peer's page once it has arrived: admit, fold, append, write, announce. The applyRemote rows count ops.", + sequence: + 'The sequence engine behind list and text fields: a fold per op over Fugue runs, a reading, a join, a diff.', + tree: 'The movable tree behind tree fields: one move folded over a block of thousands, the fold read anew and read kept.', + history: + 'The past out of the log: history() over a record with two thousand ops, a version a thousand ops back, its diff.', + tags: 'Named versions: a tag over a thousand records, writes under it (the first copies the file), reads at the tag.', + undo: 'The @alexify/syncom/undo manager: what capturing a commit costs a write, what one step back and forward costs.', sync: 'Replication over the in-process link (JSON across the boundary, like a wrpc client).', log: 'OpLog, the persisted operation log every write and every replicated op goes through.', chain: 'The hash-linked chain a write appends a block to (open({ chain: false }) skips it).', diff --git a/bench/results/compare.json b/bench/results/compare.json index 5ff4d57..8b38bcd 100644 --- a/bench/results/compare.json +++ b/bench/results/compare.json @@ -1,5 +1,5 @@ { - "generatedAt": "2026-10-22T19:17:30.850Z", + "generatedAt": "2026-10-24T17:17:53.619Z", "machine": { "cpu": "Apple M3 Max", "cores": 16, @@ -149,14 +149,14 @@ "unit": "ops/s", "results": { "syncom": { - "median": 16158.767242467984, - "min": 15427.693967881063, - "max": 16335.408128686247 + "median": 15151.097414745667, + "min": 14904.66255882781, + "max": 15552.003455535532 }, "syncom-nochain": { - "median": 17478.30819234039, - "min": 17077.209756476783, - "max": 17922.31956174327 + "median": 16673.24695462492, + "min": 16274.081773811795, + "max": 16865.327806717673 }, "yjs": { "median": 234529.6807032105, @@ -215,14 +215,14 @@ "unit": "ops/s", "results": { "syncom": { - "median": 1522.782599268753, - "min": 870.7789145651154, - "max": 1791.154874022326 + "median": 1317.3578188954496, + "min": 950.2611408948047, + "max": 1725.0194544609406 }, "syncom-nochain": { - "median": 2058.9807180673656, - "min": 1899.6581584021314, - "max": 2681.7215423345933 + "median": 2417.3692978616027, + "min": 1769.5105198873337, + "max": 2614.4699912453634 }, "yjs": { "median": 19246.796884022235, @@ -281,14 +281,14 @@ "unit": "ops/s", "results": { "syncom": { - "median": 17097.2224719563, - "min": 16455.99233010987, - "max": 17342.47878112807 + "median": 17242.84831364442, + "min": 17205.531481866048, + "max": 17315.366656374925 }, "syncom-nochain": { - "median": 18832.160250456855, - "min": 18730.591564198567, - "max": 19729.09057764994 + "median": 19357.71257045609, + "min": 19117.982604604313, + "max": 19385.181268453507 }, "yjs": { "median": 382443.6455842893, @@ -347,14 +347,14 @@ "unit": "ops/s", "results": { "syncom": { - "median": 1342.4902569368933, - "min": 994.1276564981987, - "max": 1785.5020077001432 + "median": 1528.678118244573, + "min": 1222.635060664519, + "max": 1649.4870695842033 }, "syncom-nochain": { - "median": 3076.354903983142, - "min": 2747.367021581029, - "max": 3081.166102992538 + "median": 2835.733284806349, + "min": 2718.3785986194475, + "max": 3026.6886638253036 }, "yjs": { "median": 22410.120467865163, @@ -413,14 +413,14 @@ "unit": "ops/s", "results": { "syncom": { - "median": 971873.3309273649, - "min": 964350.8042652654, - "max": 981332.403112029 + "median": 845124.1843550695, + "min": 685279.293974933, + "max": 886859.9657641001 }, "syncom-nochain": { - "median": 929272.3044637601, - "min": 926181.565504644, - "max": 946058.4976424875 + "median": 885127.0987193184, + "min": 882711.0608264786, + "max": 893258.6450470127 }, "yjs": { "median": 3675329.173463649, @@ -479,14 +479,14 @@ "unit": "ops/s", "results": { "syncom": { - "median": 958203.2939642053, - "min": 956925.5030225046, - "max": 962537.8664417432 + "median": 890676.8143517048, + "min": 884625.4855073391, + "max": 895251.6255942554 }, "syncom-nochain": { - "median": 951488.4894516196, - "min": 935275.5523628481, - "max": 961568.0319946897 + "median": 875392.8688821379, + "min": 824518.1420159993, + "max": 887428.5220306918 }, "yjs": { "median": 3688013.505908745, @@ -545,14 +545,14 @@ "unit": "ops/s", "results": { "syncom": { - "median": 16711.29691002396, - "min": 16229.341348079113, - "max": 16871.947092656297 + "median": 17125.547284222685, + "min": 16540.53399503686, + "max": 17355.73108508751 }, "syncom-nochain": { - "median": 18589.830567249475, - "min": 17960.3878465164, - "max": 18722.10628376218 + "median": 18846.07135974315, + "min": 18484.470278960198, + "max": 19189.23373280474 }, "automerge": { "median": 1948.709360655863, @@ -587,14 +587,14 @@ "unit": "ops/s", "results": { "syncom": { - "median": 1477.6751249626168, - "min": 1005.8749634054353, - "max": 1912.8041824122345 + "median": 1213.548883673148, + "min": 1167.2258374633018, + "max": 1802.2578949488534 }, "syncom-nochain": { - "median": 3061.5297282005586, - "min": 2980.962152147103, - "max": 3166.8103123249207 + "median": 2964.8028025723006, + "min": 2751.190000746668, + "max": 3180.871612768841 }, "automerge": { "median": 1196.502626020144, @@ -629,14 +629,14 @@ "unit": "records/s", "results": { "syncom": { - "median": 16009.264112882725, - "min": 15523.531166197667, - "max": 16424.878763863613 + "median": 15788.798422231685, + "min": 15456.229087528834, + "max": 16013.141280470678 }, "syncom-nochain": { - "median": 17208.668006074655, - "min": 16814.859385402095, - "max": 17407.96816319372 + "median": 16869.332863601143, + "min": 16069.624411739253, + "max": 17073.39301895595 }, "yjs": { "median": 274583.8338767806, @@ -695,14 +695,14 @@ "unit": "records/s", "results": { "syncom": { - "median": 1690.1571524683907, - "min": 903.3901754381551, - "max": 1734.4466746537935 + "median": 1621.3571651218515, + "min": 1389.372917468065, + "max": 1691.2450375170993 }, "syncom-nochain": { - "median": 2737.240013521964, - "min": 2695.7214544232434, - "max": 2762.2573443243564 + "median": 2635.4503406941544, + "min": 2414.552750884859, + "max": 2733.7890838582616 }, "yjs": { "median": 154946.6805230321, @@ -758,14 +758,14 @@ "unit": "records/s", "results": { "syncom": { - "median": 25223.675991590604, - "min": 25137.75995208943, - "max": 25752.979787155728 + "median": 25090.561244491815, + "min": 24029.869127325273, + "max": 25916.21934192241 }, "syncom-nochain": { - "median": 26248.900367941336, - "min": 22472.54136352146, - "max": 26531.64957825021 + "median": 25836.399834099306, + "min": 23842.093812678617, + "max": 26197.69852170575 }, "yjs": { "median": 99728.64832078399, @@ -820,14 +820,14 @@ "unit": "records/s", "results": { "syncom": { - "median": 5833.321608356902, - "min": 3341.5014442620823, - "max": 6033.179446231833 + "median": 5309.734494478818, + "min": 4852.105801545003, + "max": 6013.409904086109 }, "syncom-nochain": { - "median": 5820.148698280675, - "min": 4255.809617412205, - "max": 6095.829499843959 + "median": 5906.757404304994, + "min": 5293.615939937269, + "max": 5967.290298230251 }, "yjs": { "median": 53737.56015785516, @@ -866,18 +866,156 @@ "gun": "syncs over its WebSocket/WebRTC mesh only" }, "failed": {} + }, + { + "id": "append-memory", + "kind": "append", + "kindTitle": "Type at the end of a text", + "line": "memory", + "lineTitle": "In memory", + "title": "Type at the end of a text, in memory", + "description": "Insert one character at the end of a text that merges by position — the common keystroke. Nothing touches the disk.", + "unit": "ops/s", + "results": { + "syncom": { + "median": 1801.4481331397703, + "min": 1463.0919645736933, + "max": 2805.155384295683 + }, + "syncom-nochain": { + "median": 1823.0736507012377, + "min": 1484.9941253812033, + "max": 2817.772598400425 + }, + "yjs": { + "median": 346453.14085936476, + "min": 335269.5001450474, + "max": 353758.53860873624 + }, + "automerge": { + "median": 23677.620095621172, + "min": 23427.197585406804, + "max": 23922.54409279021 + }, + "loro": { + "median": 618945.3177829481, + "min": 616183.079787133, + "max": 624243.603751006 + } + }, + "unsupported": { + "tinybase": "no text that merges by position", + "pouchdb": "no text that merges by position", + "rxdb": "no text that merges by position", + "fireproof": "no text that merges by position", + "gun": "needs its storage adapter to answer in Node.js", + "orbitdb": "no text that merges by position" + }, + "failed": {} + }, + { + "id": "prepend-memory", + "kind": "prepend", + "kindTitle": "Type at the start of a text", + "line": "memory", + "lineTitle": "In memory", + "title": "Type at the start of a text, in memory", + "description": "Insert one character at the start of the text — every insert anchors into the same spot. Nothing touches the disk.", + "unit": "ops/s", + "results": { + "syncom": { + "median": 388.4270585571998, + "min": 303.27659675996125, + "max": 631.7405707122994 + }, + "syncom-nochain": { + "median": 391.109876942166, + "min": 295.5667419357193, + "max": 620.929696690396 + }, + "yjs": { + "median": 524545.354540025, + "min": 452698.8299459338, + "max": 529378.5036914867 + }, + "automerge": { + "median": 21562.760180164805, + "min": 21345.56826642622, + "max": 22273.131756077335 + }, + "loro": { + "median": 443522.96217898326, + "min": 442433.48015506344, + "max": 449082.669706808 + } + }, + "unsupported": { + "tinybase": "no text that merges by position", + "pouchdb": "no text that merges by position", + "rxdb": "no text that merges by position", + "fireproof": "no text that merges by position", + "gun": "needs its storage adapter to answer in Node.js", + "orbitdb": "no text that merges by position" + }, + "failed": {} + }, + { + "id": "random-memory", + "kind": "random", + "kindTitle": "Type anywhere in a text", + "line": "memory", + "lineTitle": "In memory", + "title": "Type anywhere in a text, in memory", + "description": "Insert one character at a random position of the text (seeded). Nothing touches the disk.", + "unit": "ops/s", + "results": { + "syncom": { + "median": 536.6881165549418, + "min": 406.36055189612205, + "max": 824.3789945850202 + }, + "syncom-nochain": { + "median": 550.3935460789885, + "min": 399.0188869892625, + "max": 851.708777701791 + }, + "yjs": { + "median": 30398.223116685596, + "min": 28438.17880877493, + "max": 31809.964215128417 + }, + "automerge": { + "median": 20597.41226358537, + "min": 20535.090149786203, + "max": 21091.554177814367 + }, + "loro": { + "median": 310723.24949627114, + "min": 308747.1732514568, + "max": 313146.29320890346 + } + }, + "unsupported": { + "tinybase": "no text that merges by position", + "pouchdb": "no text that merges by position", + "rxdb": "no text that merges by position", + "fireproof": "no text that merges by position", + "gun": "needs its storage adapter to answer in Node.js", + "orbitdb": "no text that merges by position" + }, + "failed": {} } ], "coldStart": { "syncom": { - "open": 36.004875, - "firstRead": 0.9018124999999984, - "total": 37.0136665 + "open": 40.29775, + "firstRead": 1.057874499999997, + "total": 41.319208 }, "syncom-nochain": { - "open": 35.974396, - "firstRead": 0.3473954999999975, - "total": 36.328208 + "open": 39.389958, + "firstRead": 0.5151250000000012, + "total": 39.905083 }, "yjs": { "open": 82.67010450000001, diff --git a/bench/results/engine.json b/bench/results/engine.json index 3badcdf..c1b3041 100644 --- a/bench/results/engine.json +++ b/bench/results/engine.json @@ -1,5 +1,5 @@ { - "generatedAt": "2026-10-22T19:16:04.287Z", + "generatedAt": "2026-10-24T15:16:19.982Z", "machine": { "cpu": "Apple M3 Max", "cores": 16, @@ -8,170 +8,315 @@ "node": "v24.14.1" }, "results": [ + { + "subject": "apply", + "name": "applyOp — set one field on a 5-field entry", + "opsPerSec": 3878877 + }, + { + "subject": "apply", + "name": "applyOp — inc a counter", + "opsPerSec": 3785335 + }, + { + "subject": "apply", + "name": "applyOp — add to a 20-element set", + "opsPerSec": 215539 + }, + { + "subject": "apply", + "name": "applyOp — remove from a 20-element set", + "opsPerSec": 225366 + }, + { + "subject": "apply", + "name": "applyOp — write the whole record", + "opsPerSec": 749830 + }, + { + "subject": "apply", + "name": "applyOp — merge a peer entry", + "opsPerSec": 583583 + }, + { + "subject": "apply", + "name": "packOps — 500 writes", + "opsPerSec": 63907 + }, + { + "subject": "apply", + "name": "unpackOps — 500 writes", + "opsPerSec": 40107 + }, + { + "subject": "apply", + "name": "applyRemote — 100 writes, new records", + "opsPerSec": 6376 + }, + { + "subject": "apply", + "name": "applyRemote — 500 writes, new records", + "opsPerSec": 6638 + }, + { + "subject": "apply", + "name": "applyRemote — 100 signed writes, new records", + "opsPerSec": 5025 + }, + { + "subject": "apply", + "name": "verifyOp — 100 signed writes", + "opsPerSec": 37012 + }, + { + "subject": "apply", + "name": "applyRemote — 100 sets on cached records", + "opsPerSec": 10087 + }, + { + "subject": "apply", + "name": "applyRemote — 100 incs on cached records", + "opsPerSec": 9121 + }, + { + "subject": "apply", + "name": "applyRemote — 100 adds on cached records", + "opsPerSec": 6395 + }, + { + "subject": "apply", + "name": "applyRemote — 100 writes, new records (chain: false)", + "opsPerSec": 6742 + }, + { + "subject": "apply", + "name": "applyRemote — 500 writes, new records (chain: false)", + "opsPerSec": 6846 + }, + { + "subject": "apply", + "name": "applyRemote — 100 signed writes, new records (chain: false)", + "opsPerSec": 5092 + }, + { + "subject": "apply", + "name": "applyRemote — 100 sets on cached records (chain: false)", + "opsPerSec": 10696 + }, + { + "subject": "apply", + "name": "applyRemote — 100 incs on cached records (chain: false)", + "opsPerSec": 10091 + }, + { + "subject": "apply", + "name": "applyRemote — 100 adds on cached records (chain: false)", + "opsPerSec": 7060 + }, + { + "subject": "branch", + "name": "branch — fork at the present (200 records)", + "opsPerSec": 9 + }, + { + "subject": "branch", + "name": "merge — 20 ops from the branch into the parent", + "opsPerSec": 72 + }, + { + "subject": "branch", + "name": "merge — nothing to hand over", + "opsPerSec": 13398 + }, { "subject": "chain", "name": "addBlock — small payload", - "opsPerSec": 5107 + "opsPerSec": 4704 }, { "subject": "chain", "name": "addBlock — 4 KB payload", - "opsPerSec": 4283 + "opsPerSec": 4915 }, { "subject": "chain", "name": "readBlock — single block", - "opsPerSec": 13117 + "opsPerSec": 13199 }, { "subject": "chain", "name": "validate — last 1 block", - "opsPerSec": 11287 + "opsPerSec": 10929 }, { "subject": "chain", "name": "validate — last 10 blocks", - "opsPerSec": 1098 + "opsPerSec": 1062 }, { "subject": "chain", "name": "validate — last 100 blocks", - "opsPerSec": 112 + "opsPerSec": 115 }, { "subject": "chain", "name": "validate — full 200-block chain", - "opsPerSec": 57 + "opsPerSec": 53 }, { "subject": "crdt", "name": "diff — 20 flat keys, all changed", - "opsPerSec": 324832 + "opsPerSec": 346219 }, { "subject": "crdt", "name": "diff — 20 flat keys, one changed", - "opsPerSec": 444578 + "opsPerSec": 454782 }, { "subject": "crdt", "name": "diff — depth 3, width 4", - "opsPerSec": 35563 + "opsPerSec": 35140 }, { "subject": "crdt", "name": "diff — array as a grow-only set", - "opsPerSec": 2830451 + "opsPerSec": 2818983 }, { "subject": "crdt", "name": "merge — 20 flat keys", - "opsPerSec": 2130062 + "opsPerSec": 600095 }, { "subject": "crdt", "name": "merge — depth 3, width 4", - "opsPerSec": 4959915 + "opsPerSec": 2954591 }, { "subject": "crdt", "name": "merge — numeric counter increment", - "opsPerSec": 5398362 + "opsPerSec": 6532773 }, { "subject": "crdt", "name": "apply — 100 inc records", - "opsPerSec": 207946 + "opsPerSec": 131050 }, { "subject": "crdt", "name": "apply — 100 delta records", - "opsPerSec": 8824 + "opsPerSec": 8357 }, { "subject": "emitter", "name": "emit awaited — 0 listeners", - "opsPerSec": 14423606 + "opsPerSec": 15752588 }, { "subject": "emitter", "name": "emit awaited — 1 sync listener", - "opsPerSec": 3747799 + "opsPerSec": 3823928 }, { "subject": "emitter", "name": "emit awaited — 1 listener, (data, delta)", - "opsPerSec": 3781384 + "opsPerSec": 3825533 }, { "subject": "emitter", "name": "emit awaited — 1 async listener", - "opsPerSec": 3402202 + "opsPerSec": 3554343 }, { "subject": "emitter", "name": "emit awaited — 2 sync listeners", - "opsPerSec": 3221211 + "opsPerSec": 3253767 }, { "subject": "emitter", "name": "emit fire-and-forget — 1 sync listener", - "opsPerSec": 4688543 + "opsPerSec": 4661204 }, { "subject": "hash", "name": "calculateHash — empty genesis block", - "opsPerSec": 1166378 + "opsPerSec": 1199887 }, { "subject": "hash", "name": "calculateHash — small record (5 x 16 B)", - "opsPerSec": 1040941 + "opsPerSec": 1059801 }, { "subject": "hash", "name": "calculateHash — medium record (50 x 64 B)", - "opsPerSec": 138038 + "opsPerSec": 134989 }, { "subject": "hash", "name": "calculateHash — large record (200 x 256 B)", - "opsPerSec": 14570 + "opsPerSec": 14218 }, { "subject": "hash", "name": "calculateHash — deeply nested record", - "opsPerSec": 633815 + "opsPerSec": 629742 + }, + { + "subject": "history", + "name": "history(id, { limit: 50 }) — a record with 2 000 ops", + "opsPerSec": 1817 + }, + { + "subject": "history", + "name": "history(id, { limit: Infinity }) — every version materialized", + "opsPerSec": 665 + }, + { + "subject": "history", + "name": "get(id, { at: eventId }) — 1 000 ops back", + "opsPerSec": 2997 + }, + { + "subject": "history", + "name": "get(id, { at: { vector } }) — the present, folded", + "opsPerSec": 1814 + }, + { + "subject": "history", + "name": "diff(id, at) — against 1 000 ops back", + "opsPerSec": 2976 }, { "subject": "log", "name": "append — one op", - "opsPerSec": 5605 + "opsPerSec": 5351 }, { "subject": "log", "name": "append — batch of 100", - "opsPerSec": 312484 + "opsPerSec": 315328 }, { "subject": "log", "name": "since — last 500 entries", - "opsPerSec": 629795 + "opsPerSec": 623608 }, { "subject": "log", "name": "since — everything (no cursor)", - "opsPerSec": 1708 + "opsPerSec": 1573 }, { "subject": "log", "name": "since — 20 from the middle, scoped", - "opsPerSec": 3511359 + "opsPerSec": 3430272 }, { "subject": "log", "name": "append with a roll every 8 batches", - "opsPerSec": 4931 + "opsPerSec": 4989 }, { "subject": "log", @@ -181,147 +326,167 @@ { "subject": "search", "name": "index.put — a 43-word record", - "opsPerSec": 16128 + "opsPerSec": 16389 }, { "subject": "search", "name": "index.query — one word over 2000", - "opsPerSec": 1733 + "opsPerSec": 1769 }, { "subject": "search", "name": "index.query — two words over 2000", - "opsPerSec": 1442 + "opsPerSec": 1434 }, { "subject": "search", "name": "index.query — a typo over 2000", - "opsPerSec": 7824 + "opsPerSec": 7733 }, { "subject": "search", "name": "insert — 2 search fields (43 words)", - "opsPerSec": 1561 + "opsPerSec": 1482 }, { "subject": "search", "name": "select — $text one word over 2000, limit 20", - "opsPerSec": 1348 + "opsPerSec": 1342 }, { "subject": "search", "name": "select — $text + eq narrowing over 2000", - "opsPerSec": 1129 + "opsPerSec": 1080 }, { "subject": "search", "name": "count — $text over 2000", - "opsPerSec": 1174 + "opsPerSec": 1129 }, { "subject": "search", "name": "open — catalog reloaded with text (2000 records)", - "opsPerSec": 10 + "opsPerSec": 11 }, { "subject": "storage", "name": "insert — small record", - "opsPerSec": 1156 + "opsPerSec": 1649 }, { "subject": "storage", "name": "get — cached", - "opsPerSec": 161791 + "opsPerSec": 187466 }, { "subject": "storage", "name": "get — cold (re-verifies the block)", - "opsPerSec": 218 + "opsPerSec": 162 }, { "subject": "storage", "name": "update — one field", - "opsPerSec": 1855 + "opsPerSec": 1863 + }, + { + "subject": "storage", + "name": "history — 50 versions", + "opsPerSec": 19862 + }, + { + "subject": "storage", + "name": "tag", + "opsPerSec": 986 + }, + { + "subject": "storage", + "name": "get — at a tag (a copy)", + "opsPerSec": 6936 + }, + { + "subject": "storage", + "name": "get — at an event id (25 ops back)", + "opsPerSec": 85541 }, { "subject": "storage", "name": "set — whole record", - "opsPerSec": 1855 + "opsPerSec": 1599 }, { "subject": "storage", "name": "swap — match (writes)", - "opsPerSec": 1927 + "opsPerSec": 1829 }, { "subject": "storage", "name": "swap — mismatch (reads only)", - "opsPerSec": 166997 + "opsPerSec": 192470 }, { "subject": "storage", "name": "has — existing id", - "opsPerSec": 184719 + "opsPerSec": 212612 }, { "subject": "storage", "name": "record — cached Record instance", - "opsPerSec": 8901096 + "opsPerSec": 9078536 }, { "subject": "storage", "name": "list — 2000 ids (catalog)", - "opsPerSec": 11325130 + "opsPerSec": 11799479 }, { "subject": "storage", "name": "hasRecord — 2000 ids (catalog)", - "opsPerSec": 13477594 + "opsPerSec": 14482625 }, { "subject": "storage", "name": "select — covered eq over 2000", - "opsPerSec": 21362 + "opsPerSec": 21151 }, { "subject": "storage", "name": "select — eq + order + limit over 2000", - "opsPerSec": 11000 + "opsPerSec": 11185 }, { "subject": "storage", "name": "select — uncovered filter over 2000", - "opsPerSec": 980 + "opsPerSec": 972 }, { "subject": "storage", "name": "open — catalog caught up from the log (2000 records)", - "opsPerSec": 31 + "opsPerSec": 3 }, { "subject": "storage", "name": "open — catalog rebuilt from a scan (2000 records, integrity lazy)", - "opsPerSec": 30 + "opsPerSec": 3 }, { "subject": "sync", "name": "write on A, applied on B (one hop)", - "opsPerSec": 539 + "opsPerSec": 739 }, { "subject": "sync", "name": "write on A, applied on C (two hops)", - "opsPerSec": 679 + "opsPerSec": 787 }, { "subject": "sync", "name": "update on A, applied on C (two hops)", - "opsPerSec": 741 + "opsPerSec": 585 }, { "subject": "sync", "name": "snapshot — one page of 500 out of 2000 (host side)", - "opsPerSec": 8920 + "opsPerSec": 9406 }, { "subject": "sync", @@ -331,112 +496,137 @@ { "subject": "sync", "name": "append of another scope's op while a scoped feed listens", - "opsPerSec": 1711 + "opsPerSec": 1549 }, { - "subject": "apply", - "name": "applyOp — set one field on a 5-field entry", - "opsPerSec": 4717320 + "subject": "tags", + "name": "tag() + untag() — over 1 000 records (memory fsi)", + "opsPerSec": 1932 }, { - "subject": "apply", - "name": "applyOp — inc a counter", - "opsPerSec": 4421459 + "subject": "tags", + "name": "update — under a tag; the first write of a record copies its file", + "opsPerSec": 17511 }, { - "subject": "apply", - "name": "applyOp — add to a 20-element set", - "opsPerSec": 236399 + "subject": "tags", + "name": "at(tag).get — a record copied at the tag", + "opsPerSec": 142544 }, { - "subject": "apply", - "name": "applyOp — remove from a 20-element set", - "opsPerSec": 246187 + "subject": "tags", + "name": "get(id, { at: { tag } }) — through the read hooks", + "opsPerSec": 157133 }, { - "subject": "apply", - "name": "applyOp — write the whole record", - "opsPerSec": 930456 + "subject": "tags", + "name": "at(tag).list — the tag’s catalog, 1 000 ids", + "opsPerSec": 1364874 }, { - "subject": "apply", - "name": "applyOp — merge a peer entry", - "opsPerSec": 667351 + "subject": "tree", + "name": "applyOp — one move on a tree of 5 000 moves", + "opsPerSec": 277 }, { - "subject": "apply", - "name": "packOps — 500 writes", - "opsPerSec": 71667 + "subject": "tree", + "name": "resolve — the fold of 5 000 moves, anew", + "opsPerSec": 309 }, { - "subject": "apply", - "name": "unpackOps — 500 writes", - "opsPerSec": 40294 + "subject": "tree", + "name": "renderTree — the fold kept by the block", + "opsPerSec": 13978 }, { - "subject": "apply", - "name": "applyRemote — 100 writes, new records", - "opsPerSec": 6549 + "subject": "tree", + "name": "storage.treeMove — among 300 siblings (memory fsi)", + "opsPerSec": 1528 }, { - "subject": "apply", - "name": "applyRemote — 500 writes, new records", - "opsPerSec": 6997 + "subject": "undo", + "name": "update — no undo manager", + "opsPerSec": 1983 }, { - "subject": "apply", - "name": "applyRemote — 100 signed writes, new records", - "opsPerSec": 5154 + "subject": "undo", + "name": "update — captured by the undo manager", + "opsPerSec": 1919 }, { - "subject": "apply", - "name": "verifyOp — 100 signed writes", - "opsPerSec": 36659 + "subject": "undo", + "name": "undo + redo — one step of four kinds", + "opsPerSec": 340 }, { - "subject": "apply", - "name": "applyRemote — 100 sets on cached records", - "opsPerSec": 10621 + "subject": "sequence", + "name": "applyOp — append one char to a 1 000-char text (one run)", + "opsPerSec": 158717 }, { - "subject": "apply", - "name": "applyRemote — 100 incs on cached records", - "opsPerSec": 10077 + "subject": "sequence", + "name": "applyOp — insert one char into a 1 000-char text (new run)", + "opsPerSec": 25814 }, { - "subject": "apply", - "name": "applyRemote — 100 adds on cached records", - "opsPerSec": 6702 + "subject": "sequence", + "name": "applyOp — delete 10 chars from a 1 000-char text", + "opsPerSec": 30325 }, { - "subject": "apply", - "name": "applyRemote — 100 writes, new records (chain: false)", - "opsPerSec": 7175 + "subject": "sequence", + "name": "applyOp — prepend one char to a 1 000-run text", + "opsPerSec": 2310 }, { - "subject": "apply", - "name": "applyRemote — 500 writes, new records (chain: false)", - "opsPerSec": 7173 + "subject": "sequence", + "name": "applyOp — merge two pictures of a 1 000-char text", + "opsPerSec": 434 }, { - "subject": "apply", - "name": "applyRemote — 100 signed writes, new records (chain: false)", - "opsPerSec": 5015 + "subject": "sequence", + "name": "buildIndex + render — a 1 000-char text", + "opsPerSec": 38401 }, { - "subject": "apply", - "name": "applyRemote — 100 sets on cached records (chain: false)", - "opsPerSec": 10910 + "subject": "sequence", + "name": "buildIndex + render — a 100 KB text (one run)", + "opsPerSec": 376 }, { - "subject": "apply", - "name": "applyRemote — 100 incs on cached records (chain: false)", - "opsPerSec": 9994 + "subject": "sequence", + "name": "spans — 1 000 chars, 50 overlapping marks", + "opsPerSec": 11429 }, { - "subject": "apply", - "name": "applyRemote — 100 adds on cached records (chain: false)", - "opsPerSec": 6736 + "subject": "sequence", + "name": "diff — one keystroke in a 1 KB text", + "opsPerSec": 315083 + }, + { + "subject": "sequence", + "name": "storage.push — one char (memory fsi)", + "opsPerSec": 8051 + }, + { + "subject": "sequence", + "name": "storage.updateText — one keystroke in a 1 KB text (memory fsi)", + "opsPerSec": 2143 + }, + { + "subject": "sequence", + "name": "storage.move — in a 100-element list (memory fsi)", + "opsPerSec": 1216 + }, + { + "subject": "sequence", + "name": "storage.push — one char into a 10 KB text (memory fsi)", + "opsPerSec": 3233 + }, + { + "subject": "sequence", + "name": "storage.push — one char into a 10 KB text, defer: 50 (memory fsi)", + "opsPerSec": 3143 } ] } diff --git a/bench/results/size.json b/bench/results/size.json index cc350be..19ffef1 100644 --- a/bench/results/size.json +++ b/bench/results/size.json @@ -1,5 +1,5 @@ { - "generatedAt": "2026-10-23T19:12:12.497Z", + "generatedAt": "2026-10-24T19:09:32.026Z", "esbuild": "0.28.2", "libraries": [ { @@ -9,8 +9,8 @@ "packages": [ "@alexify/syncom@0.0.1" ], - "min": 127030, - "gzip": 43609, + "min": 199647, + "gzip": 68221, "note": "storage, schema, queries, the op log and the sync protocol are one entry" }, { @@ -117,5 +117,49 @@ "gzip": 362166, "note": "createHelia brings Helia's default libp2p configuration with it" } + ], + "cores": [ + { + "id": "syncom", + "name": "syncom", + "entry": "export * from '@alexify/syncom'", + "packages": [ + "@alexify/syncom@0.0.1" + ], + "min": 198395, + "gzip": 67767 + }, + { + "id": "yjs", + "name": "Yjs", + "entry": "export * from 'yjs'", + "packages": [ + "yjs@13.6.33" + ], + "min": 93287, + "gzip": 28753 + }, + { + "id": "automerge", + "name": "Automerge", + "entry": "export * from '@automerge/automerge'", + "packages": [ + "@automerge/automerge@3.5.0" + ], + "min": 3724185, + "gzip": 1161526, + "wasm": 3644225 + }, + { + "id": "loro", + "name": "Loro", + "entry": "export * from 'loro-crdt'", + "packages": [ + "loro-crdt@1.16.4" + ], + "min": 3493645, + "gzip": 1143024, + "wasm": 3398779 + } ] } diff --git a/bench/sequence.js b/bench/sequence.js new file mode 100644 index 0000000..a308e5f --- /dev/null +++ b/bench/sequence.js @@ -0,0 +1,103 @@ +/** + * Lists and text: the sequence engine behind the two kinds — what one + * keystroke costs the model (a fold over a block with a run per insert), a + * reading, a snapshot join, a diff, and the same through the storage over + * the memory backend. + * + * Run with `node bench/sequence.js` or `pnpm bench`. + */ + +'use strict'; + +const { bench } = require('./support/harness.js'); +const S = require('../src/sequence.js'); +const { applyOp } = require('../src/ops.js'); +const { diff } = require('../src/myers.js'); +const { createMemoryFsi } = require('../src/backend.js'); +const { open } = require('../src/storage.js'); + +const op = (type, ts, origin, key, value) => ({ origin, seq: ts, ts, type, id: 'r', key, value }); + +const main = async () => { + console.log(`Sequence benchmark — Node ${process.version}\n`); + const text = 'x'.repeat(1000); + // One run of 1 000 characters, and 1 000 runs (a prepend chain). + const oneRun = applyOp(null, op('splice', 1, 'A', 'body', { at: '^', content: text })).entry; + let chain = null; + let prev = null; + for (let i = 0; i < 1000; i++) { + const ts = 2 + i; + const at = prev === null ? '^' : `<${S.elemKey(prev, 0)}`; + chain = applyOp(chain, op('splice', ts, 'B', 'body', { at, content: 'y' })).entry; + prev = S.runKey(ts, 'B'); + } + let ts = 5000; + const run = S.runKey(1, 'A'); + await bench('applyOp — append one char to a 1 000-char text (one run)', () => + applyOp(oneRun, op('splice', ++ts, 'A', 'body', { run, from: 1000, content: 'z' }))); + await bench('applyOp — insert one char into a 1 000-char text (new run)', () => + applyOp(oneRun, op('splice', ++ts, 'C', 'body', { at: `<${S.elemKey(run, 500)}`, content: 'z' }))); + await bench('applyOp — delete 10 chars from a 1 000-char text', () => + applyOp(oneRun, op('splice', ++ts, 'C', 'body', { cut: [[run, 400, 10]], content: '' }))); + await bench('applyOp — prepend one char to a 1 000-run text', () => + applyOp(chain, op('splice', ++ts, 'C', 'body', { at: `<${S.elemKey(prev, 0)}`, content: 'z' }))); + await bench('applyOp — merge two pictures of a 1 000-char text', () => + applyOp( + oneRun, + op('merge', ++ts, 'M', undefined, undefined) && { ...op('merge', ts, 'M'), data: chain.data, meta: chain.meta }, + )); + const seq = oneRun.meta.seqs.body; + await bench('buildIndex + render — a 1 000-char text', () => S.render(S.buildIndex(seq))); + const big = applyOp(null, op('splice', 1, 'A', 'body', { at: '^', content: 'x'.repeat(100_000) })).entry.meta.seqs + .body; + await bench('buildIndex + render — a 100 KB text (one run)', () => S.render(S.buildIndex(big))); + let marked = oneRun; + for (let i = 0; i < 50; i++) { + const start = `<${S.elemKey(run, i * 10)}`; + const end = `<${S.elemKey(run, i * 10 + 20)}`; + marked = applyOp( + marked, + op('mark', 100 + i, 'A', 'body', { start, end, key: i % 2 ? 'bold' : 'em', value: true }), + ).entry; + } + const markedSeq = marked.meta.seqs.body; + await bench('spans — 1 000 chars, 50 overlapping marks', () => S.spans(S.buildIndex(markedSeq), markedSeq)); + const before = 'the quick brown fox '.repeat(50); + const after = `${before.slice(0, 500)}X${before.slice(500)}`; + await bench('diff — one keystroke in a 1 KB text', () => diff(before, after)); + + const storage = await open({ path: '/bench', fsi: createMemoryFsi(), logger: false }); + const id = await storage.insert({}); + await storage.updateText(id, 'body', before); + await bench('storage.push — one char (memory fsi)', () => storage.push(id, 'body', 'z')); + let n = 0; + await bench('storage.updateText — one keystroke in a 1 KB text (memory fsi)', async () => { + const current = (await storage.get(id)).body; + n++; + await storage.updateText(id, 'body', `${current.slice(0, 500 + (n % 7))}Q${current.slice(500 + (n % 7))}`); + }); + const list = await storage.insert({}); + await storage.push( + list, + 'items', + Array.from({ length: 100 }, (_, i) => i), + ); + await bench('storage.move — in a 100-element list (memory fsi)', () => storage.move(list, 'items', n++ % 100, 50)); + // The record file written whole on every keystroke, against the same + // keystrokes under `defer`: the file and the block wait for the window. + const long = await storage.insert({}); + await storage.updateText(long, 'body', 'x'.repeat(10_000)); + await bench('storage.push — one char into a 10 KB text (memory fsi)', () => storage.push(long, 'body', 'z')); + await storage.close(); + const deferred = await open({ path: '/bench-defer', fsi: createMemoryFsi(), logger: false, defer: 50 }); + const lazy = await deferred.insert({}); + await deferred.updateText(lazy, 'body', 'x'.repeat(10_000)); + await bench('storage.push — one char into a 10 KB text, defer: 50 (memory fsi)', () => + deferred.push(lazy, 'body', 'z')); + await deferred.close(); +}; + +main().catch((error) => { + console.error(error); + process.exitCode = 1; +}); diff --git a/bench/storage.js b/bench/storage.js index d6f729e..5d96998 100644 --- a/bench/storage.js +++ b/bench/storage.js @@ -57,6 +57,21 @@ const main = async () => { ); await bench('update — one field', () => storage.update(id, { hits: 1 }), { warmup: 20 }); + + // The past is a fold of the log's entries of the record: fifty versions + // read back, and one of them read at its event id. + const versioned = await storage.insert(record); + for (let i = 0; i < 49; i++) await storage.update(versioned, { hits: i }); + const { versions } = await storage.history(versioned); + await bench('history — 50 versions', () => storage.history(versioned)); + // A tag is a catalog file; a record's first change after it copies the file. + let tags = 0; + await bench('tag', () => storage.tag(`t${tags++}`), { warmup: 5, measureMs: 500 }); + const tagged = await storage.insert(record); + await storage.tag('read-at'); + await storage.update(tagged, { hits: 1 }); + await bench('get — at a tag (a copy)', () => storage.get(tagged, { at: { tag: 'read-at' } })); + await bench('get — at an event id (25 ops back)', () => storage.get(versioned, { at: versions[25].eventId })); await bench('set — whole record', () => storage.set(id, record), { warmup: 20 }); await bench( diff --git a/bench/tags.js b/bench/tags.js new file mode 100644 index 0000000..a58cabf --- /dev/null +++ b/bench/tags.js @@ -0,0 +1,40 @@ +/** + * Tags: a tag taken over a thousand records (the catalog copied under the + * log position), writes under a tag (the first one of a record copies its + * file, the rest write as usual), a read at the tag, and a tag dropped — + * over the memory backend. + * + * Run with `node bench/tags.js` or `pnpm bench`. + */ + +'use strict'; + +const { bench } = require('./support/harness.js'); +const { createMemoryFsi } = require('../src/backend.js'); +const { open } = require('../src/storage.js'); + +const main = async () => { + console.log(`Tags benchmark — Node ${process.version}\n`); + const storage = await open({ path: '/bench', fsi: createMemoryFsi(), logger: false }); + const ids = []; + for (let i = 0; i < 1000; i++) ids.push(await storage.insert({ n: i, group: i % 10 })); + let k = 0; + await bench('tag() + untag() — over 1 000 records (memory fsi)', async () => { + await storage.tag(`t${k}`); + await storage.untag(`t${k++}`); + }); + await storage.tag('cow'); + let j = 0; + await bench('update — under a tag; the first write of a record copies its file', () => + storage.update(ids[j++ % ids.length], { n: j })); + await bench('at(tag).get — a record copied at the tag', () => storage.at('cow').get(ids[j++ % ids.length])); + await bench('get(id, { at: { tag } }) — through the read hooks', () => + storage.get(ids[j++ % ids.length], { at: { tag: 'cow' } })); + await bench('at(tag).list — the tag’s catalog, 1 000 ids', async () => (await storage.at('cow').list()).length); + await storage.close(); +}; + +main().catch((error) => { + console.error(error); + process.exitCode = 1; +}); diff --git a/bench/tree.js b/bench/tree.js new file mode 100644 index 0000000..a65f7c0 --- /dev/null +++ b/bench/tree.js @@ -0,0 +1,55 @@ +/** + * Trees: the movable tree behind the tree kind — one move folded over a + * block of thousands of moves, the fold read anew and read from the block + * that keeps it, and a move among many siblings through the storage over + * the memory backend. + * + * Run with `node bench/tree.js` or `pnpm bench`. + */ + +'use strict'; + +const { bench } = require('./support/harness.js'); +const T = require('../src/tree.js'); +const { applyOp } = require('../src/ops.js'); +const { createMemoryFsi } = require('../src/backend.js'); +const { open } = require('../src/storage.js'); + +const op = (ts, origin, value) => ({ origin, seq: ts, ts, type: 'tree', id: 'r', key: 'outline', value }); + +const main = async () => { + console.log(`Tree benchmark — Node ${process.version}\n`); + // A root, 1 000 nodes under it, 4 000 more moves among them: a block of + // 5 000 moves, the fold of which is what every op pays for. + let entry = applyOp(null, op(1, 'A', { node: 'root', parent: null, fi: 'V' })).entry; + let ts = 1; + for (let i = 0; i < 1000; i++) { + entry = applyOp(entry, op(++ts, 'A', { node: `n${i}`, parent: 'root', fi: 'V' })).entry; + } + for (let i = 0; i < 4000; i++) { + const node = `n${i % 1000}`; + const parent = `n${(i * 7 + 3) % 1000}`; + if (node !== parent) entry = applyOp(entry, op(++ts, 'B', { node, parent, fi: 'V' })).entry; + } + const block = entry.meta.trees.outline; + await bench('applyOp — one move on a tree of 5 000 moves', () => + applyOp(entry, op(++ts, 'C', { node: 'n5', parent: 'root', fi: 'V' }))); + await bench('resolve — the fold of 5 000 moves, anew', () => T.resolve(block)); + await bench('renderTree — the fold kept by the block', () => T.renderTree(block)); + + const storage = await open({ path: '/bench', fsi: createMemoryFsi(), logger: false }); + const id = await storage.insert({}); + await storage.treeMove(id, 'outline', 'root'); + for (let i = 0; i < 300; i++) await storage.treeMove(id, 'outline', `n${i}`, 'root'); + let n = 0; + await bench('storage.treeMove — among 300 siblings (memory fsi)', () => { + n++; + return storage.treeMove(id, 'outline', `n${n % 300}`, 'root', { after: `n${(n + 7) % 300}` }); + }); + await storage.close(); +}; + +main().catch((error) => { + console.error(error); + process.exitCode = 1; +}); diff --git a/bench/undo.js b/bench/undo.js new file mode 100644 index 0000000..f7244d8 --- /dev/null +++ b/bench/undo.js @@ -0,0 +1,57 @@ +/** + * Undo — what capturing a commit costs a write (the inverse patches, as + * copies), and what an undo and a redo of one step cost: the patches + * realized as ops through the usual write path. The record carries a + * register, a counter, a set and a short text, so every kind of inverse is + * in the step. + * + * Run with `node bench/undo.js` or `pnpm bench undo`. + */ + +'use strict'; + +const fs = require('node:fs/promises'); +const os = require('node:os'); +const path = require('node:path'); +const { bench } = require('./support/harness.js'); +const { open } = require('../src/storage.js'); +const { createUndo } = require('../src/undo.js'); + +const main = async () => { + console.log(`Undo benchmark — Node ${process.version}\n`); + const root = await fs.mkdtemp(path.join(os.tmpdir(), 'syncom-bench-undo-')); + try { + const storage = await open({ path: root, lock: false }); + const id = await storage.insert({ title: 'a', n: 0 }); + await storage.inc(id, 'hits', 1); + await storage.add(id, 'tags', ['x']); + await storage.splice(id, 'body', 0, 0, 'hello world'); + let n = 0; + await bench('update — no undo manager', () => storage.update(id, { n: n++ }), { warmup: 20 }); + const undo = createUndo(storage, { mergeInterval: 0, limit: 1000 }); + await bench('update — captured by the undo manager', () => storage.update(id, { n: n++ }), { warmup: 20 }); + undo.clear(); + await bench( + 'undo + redo — one step of four kinds', + async () => { + await storage.update(id, { n: n++ }); + await storage.inc(id, 'hits', 1); + await storage.add(id, 'tags', [`t${n}`]); + await storage.splice(id, 'body', 0, 1, 'H'); + await undo.undo(); + await undo.redo(); + undo.clear(); + }, + { warmup: 10, measureMs: 500 }, + ); + undo.stop(); + await storage.close(); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } +}; + +main().catch((error) => { + console.error(error); + process.exit(1); +}); diff --git a/branch.d.ts b/branch.d.ts new file mode 100644 index 0000000..98c29ee --- /dev/null +++ b/branch.d.ts @@ -0,0 +1,42 @@ +// The `@alexify/syncom/branch` subpath: a storage forked from another — a +// new identity standing at the parent's state or at a tag of it — merged +// back, or the parent into it, as ops. Platform-neutral; bundles no engine. + +import type { Storage, StorageOptions, VersionVector } from './index.js'; + +export interface BranchOptions extends Omit { + /** The branch's directory — a new one. */ + path: string; + /** Fork at a tag of the parent instead of its present. */ + from?: { tag: string } | null; +} + +export interface BranchInfo { + /** The parent's node id. */ + parent: string; + forkedAt: { eventId: string; vector: VersionVector; ts: number; tag: string | null }; + createdAt: number; +} + +export interface MergeResult { + /** Ops applied, and the pages they went in. */ + ops: number; + pages: number; + /** The source's log no longer held what the other needed: its state was joined instead, and its vector adopted. */ + joined: boolean; +} + +/** + * A storage forked from `parent` at `path`: a new identity standing at the + * parent's state (or at `from.tag`), opened with the options given (`sync` + * gets `retention: { ops: 0 }` and `autoStart: false`). Rejects with + * `BRANCH_EXISTS` when `path` holds any files — a branch or another storage + * is never taken over — and `TAG_NOT_FOUND` for a `from.tag` the parent lacks. + */ +export function branch(parent: Storage, options: BranchOptions): Promise; +/** Hands `into` everything `source` has that it lacks — ops through `applyRemote`, or a join of state when the source compacted past it. */ +export function merge(source: Storage, options: { into: Storage }): Promise; +/** What a storage opened on a branch's directory was forked from, or `null` for one that is no branch; `CORRUPT` for an unreadable `branch.json`. */ +export function branchInfo(storage: Storage): Promise; +/** Closes a branch and removes its files (the empty directories stay); a storage that is no branch is a `TypeError`, an unreadable `branch.json` `CORRUPT`. */ +export function drop(storage: Storage): Promise; diff --git a/branch.js b/branch.js new file mode 100644 index 0000000..d8d5fd6 --- /dev/null +++ b/branch.js @@ -0,0 +1,3 @@ +'use strict'; + +module.exports = require('./src/branch.js'); diff --git a/browser.d.ts b/browser.d.ts index 7f3f745..33e02fe 100644 --- a/browser.d.ts +++ b/browser.d.ts @@ -91,6 +91,31 @@ export type { SnapshotPage, ApplyRemoteResult, SyncomErrorCode, + SyncomRecordCtor, + StorageStats, + LockOptions, + HostedPeer, + Refusal, + RefusalReason, + SplicePayload, + AssignPayload, + MovePayload, + MarkPayload, + TreePayload, + SequenceBlock, + TreeBlock, + TreeNode, + TextSpan, + CursorSide, + MarkExpand, + Version, + HistoryVersion, + HistoryResult, + Patch, + CommitEvent, + Tag, + TagView, + TagCollection, } from './index.js'; export { diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index 8624606..60f5bcc 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -38,6 +38,11 @@ const keywords = [ 'database', 'query', 'full-text-search', + 'collaborative-text', + 'rich-text', + 'fractional-index', + 'undo', + 'branches', 'schema', 'migrations', 'standard-schema', @@ -148,6 +153,7 @@ export default withMermaid({ { text: 'Why syncom?', link: '/guide/why' }, { text: 'Compared with alternatives', link: '/guide/comparison' }, { text: 'Compared with RxDB', link: '/guide/vs-rxdb' }, + { text: 'Compared with CRDT libraries', link: '/guide/vs-crdt' }, { text: 'Benchmarks', link: '/guide/benchmarks' }, { text: 'Getting Started', link: '/guide/getting-started' }, { text: 'Architecture', link: '/guide/architecture' }, @@ -159,6 +165,8 @@ export default withMermaid({ { text: 'Storage', link: '/guide/storage' }, { text: 'Schema and validation', link: '/guide/schema' }, { text: 'Queries', link: '/guide/query' }, + { text: 'Lists, text and trees', link: '/guide/lists' }, + { text: 'Versions and history', link: '/guide/versions' }, { text: 'Hooks and events', link: '/guide/hooks' }, { text: 'CRDT deltas', link: '/guide/crdt' }, { text: 'Blockchain log', link: '/guide/chain' }, diff --git a/docs/.vitepress/theme/components/BenchFigure.vue b/docs/.vitepress/theme/components/BenchFigure.vue index 4885d45..dce8850 100644 --- a/docs/.vitepress/theme/components/BenchFigure.vue +++ b/docs/.vitepress/theme/components/BenchFigure.vue @@ -2,17 +2,21 @@ import { computed } from 'vue'; import compare from '../../../../bench/results/compare.json'; import size from '../../../../bench/results/size.json'; -import { coldStartModel, machineCaption, sizeModel } from '../../../../bench/report/model.mjs'; +import { coldStartModel, coreSizeModel, machineCaption, sizeModel } from '../../../../bench/report/model.mjs'; import BenchBars from './BenchBars.vue'; // `only`: the library ids to show (a page about one alternative); all of them when absent. -const props = defineProps<{ chart: 'coldstart' | 'size'; only?: string[] }>(); +// `cores`: the CRDT libraries' whole packages (/guide/vs-crdt). +const props = defineProps<{ chart: 'coldstart' | 'size' | 'cores'; only?: string[] }>(); -const figure = computed(() => - props.chart === 'coldstart' - ? { model: coldStartModel(compare, props.only ?? null), caption: machineCaption(compare.machine) } - : { model: sizeModel(size, props.only ?? null), caption: `esbuild ${size.esbuild} · ${size.generatedAt.slice(0, 10)}` }, -); +const sizeCaption = `esbuild ${size.esbuild} · ${size.generatedAt.slice(0, 10)}`; +const figure = computed(() => { + if (props.chart === 'coldstart') { + return { model: coldStartModel(compare, props.only ?? null), caption: machineCaption(compare.machine) }; + } + const model = props.chart === 'cores' ? coreSizeModel(size, props.only ?? null) : sizeModel(size, props.only ?? null); + return { model, caption: sizeCaption }; +});