perf: dep_graph: skip edge collection in the marking walk when carrying [-1.54% incr-unchanged]#58
Open
xmakro wants to merge 2 commits into
Open
Conversation
Compaction was triggered only by a fixed generation cap: after eight carried generations the file is rewritten fresh, whatever its actual state. Track the live record bytes exactly in the footer, as an O(changed) delta like the other counts, and rewrite once the record region exceeds twice its live bytes. High-churn graphs compact as often as before, while low-churn graphs carry for up to sixteen generations instead of eight, avoiding the periodic compaction spikes the fixed cap forced on them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With the record region carried forward, promoting a green node no longer consumes its edge list: the record is already in the new file. The marking walk still collected every dependency's index into a buffer purely to hand it to the encoder. Add a non-collecting variant of the walk for carried sessions that verifies dependency colors and promotes with just the color-map insert, eliminating a store per edge and the per-node edge-frame bookkeeping from the hottest incremental path. Compacting sessions keep the collecting walk, whose encoder still needs the edges. Local measurement on top of the region carry: the walk's self cost on a dep-graph-heavy incr-unchanged rebuild drops by about a quarter, and multi-generation medians improve by up to another 1.5% (match-stress) and 0.9% (hyper incr-patched). 12 generations of incr-unchanged, of alternating real edits, and of -Zincremental-verify-ich all pass; tests/incremental passes in full. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 876ef96)
xmakro
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #44 (append-only save via wholesale carry of the record region). Split out of that PR to be measurable on its own; independent of the serve-edges-in-place sibling PR #59.
With the record region carried forward, promoting a green node no longer consumes its edge list: the record is already in the new file. The marking walk still collected every dependency's index into a buffer purely to hand it to the encoder. Add a non-collecting variant of the walk for carried sessions that verifies dependency colors and promotes with just the color-map insert — no store per edge, no per-node edge-frame bookkeeping in the hottest incremental path. Compacting sessions keep the collecting walk, whose encoder still needs the edges.
Local rustc-perf (primary crates, instructions:u), this change alone against the #44 branch:
28 of 60 cells improved by at least 0.25%, 0 regressed (best: cargo debug incr-unchanged -1.98%).
Validation: 20 generations each of incr-unchanged, of alternating real edits, and of alternating edits under
-Zincremental-verify-ich, all crossing the compaction boundary: every generation clean.