perf(db): explore incremental update hot paths

[samwillis]

Draft scope

This is the original exploratory branch from the incremental update benchmark investigation. It is intentionally broad and should be reviewed as a bundle of instrumentation, benchmark tooling, and candidate performance changes rather than as a narrow production fix.

The branch currently contains 32 commits over main across 43 files.

What it contains

Instrumentation and benchmark tooling

• Adds opt-in perf tracing helpers for DB live queries and db-ivm operators.
• Adds trace aggregation coverage for the incremental query path, including hashing/operator timings.
• Adds scripts/bench/incremental-update.ts for reproducing the issue-style benchmark shapes and source-size experiments.
• Adds the investigation docs:
  • INCREMENTAL_UPDATE_INSTRUMENTATION_PLAN.md
  • INCREMENTAL_UPDATE_FINDINGS.md

DB collection and sync hot paths

• Avoids full row origin snapshots and avoids storing implicit remote row origins.
• Adds faster single-op and two-row sync commit paths.
• Reduces redundant synced-data/index map lookups during common insert/update/delete paths.
• Trims virtual prop enrichment and simple sync update checks when they are not needed.
• Reduces incremental change delivery allocations and skips empty filtered subscription batches.
• Makes Transaction.applyMutations linear rather than repeatedly shifting work from the front of an array.

Query/live graph hot paths

• Runs immediate live query graph work directly when there is no transaction context.
• Skips redundant pending graph checks.
• Avoids full nested include scans in the benchmarked include shapes.
• Trims live query update allocations.

Index/sorted map/hash/db-ivm changes

• Adds fast paths for key-sorted SortedMap updates and duplicate lookup avoidance.
• Caches compiled index expression evaluators and reduces index timestamp allocation overhead.
• Reduces group aggregate hashing cost.
• Reuses group aggregate entries and specializes simple group aggregates.
• Adds fast paths for single-row joins.

Why

The benchmark shared during the investigation showed very slow steady-state incremental updates for query shapes that look like filtered lists plus joins/includes/counts. The main question was whether the time was in source collection update application, hashing, live query scheduling, or the db-ivm query graph.

This branch keeps the instrumentation used to answer that question and the candidate optimizations that were explored from those traces.

Review notes

• This is a draft PR for review/discussion and not currently shaped as a minimal merge candidate.
• Some production-ready fixes from this investigation have already been split out elsewhere. This branch preserves the full exploratory thread for comparison and review.
• The diff includes tracing and benchmark docs/tooling by design.

Validation performed during the investigation

• Ran the local incremental benchmark repeatedly across multiple source sizes, including the larger issue/user/comment datasets used to approximate the shared benchmark.
• Compared baseline, this branch, the external performance PR branch, and the combined branch.
• Added targeted tests for trace aggregation and db-ivm groupBy incremental behavior.

No fresh full CI run was performed immediately before opening this draft PR.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 548970d2-3d8c-4c4b-9c90-6951ac9e791e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-incremental-snapshot-hot-path

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}