SDSTOR-22729: index/wb_cache: Fix recovery corruption after root-split crash#900
Open
JacksonYao287 wants to merge 1 commit into
Open
SDSTOR-22729: index/wb_cache: Fix recovery corruption after root-split crash#900JacksonYao287 wants to merge 1 commit into
JacksonYao287 wants to merge 1 commit into
Conversation
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## stable/v7.x #900 +/- ##
==============================================
Coverage ? 48.15%
==============================================
Files ? 110
Lines ? 13057
Branches ? 6285
==============================================
Hits ? 6288
Misses ? 2579
Partials ? 4190 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
JacksonYao287
force-pushed
the
SDSTOR-22729
branch
2 times, most recently
from
July 24, 2026 06:06
ac1f6d4 to
1f4d59f
Compare
…t crash
A B-tree root split (tree height N → N+1) proceeds in three steps:
1. Allocate new_root and call on_root_changed(new_root) — updates the
in-memory superblock (SB) and calls transact_bufs(meta_buf, new_root_buf),
which links meta_buf → new_root_buf in the CP flush DAG.
2. split_node(new_root, old_root) modifies old_root in memory
(edge_info = EMPTY, next_bnode = child_node2) and calls
transact_nodes({child_node2}, {}, old_root, new_root), which invokes
link_buf(new_root_buf, old_root_buf).
3. The on-disk SB is only written at the very end of CP flush, after all
node buffers complete.
Three bugs combined to corrupt the tree when a SIGKILL landed after step 2
but before the SB write:
Bug A — link_buf Condition 1 created a flat flush DAG
Condition 1 bypassed new_root_buf whenever it was newly created in the
current CP, regardless of whether old_root_buf was new or old. As a result,
old_root_buf linked directly to meta_buf instead of through new_root_buf,
making new_root_buf and old_root_buf siblings under meta_buf with no ordering
between them. old_root could therefore reach disk in its transient split
state (edge_info=EMPTY, next_bnode=child_node2) before new_root, widening
the crash window.
Bug B — Recovery discarded a committed new_root
When the SB write did not complete, the recovery loop saw new_root_buf's
parent (meta_buf) as uncommitted and threw new_root_buf away. The persisted
SB still pointed to old_root.
Bug C — repair_root_node applied an unsafe edge repair
With old_root as the recovered tree root, repair_root_node read
old_root.next_bnode (= child_node2, level N) and set it as old_root's edge
child (also level N). This violated the B-tree invariant
child.level == parent.level − 1, causing validate_node to abort with
"Child node level mismatch" on the next B-tree access.
The same bug sequence applies to the first root split (leaf → level=1) as
well as any subsequent split.
Fix 1 — Restore correct flush-DAG ordering (covers Bug A)
Changed link_buf Condition 1 in wb_cache.cpp to bypass up_buf only when
BOTH up_buf AND down_buf were created in the current CP. When down_buf is
an older node (e.g. old_root), the dependency chain through up_buf
(new_root) is preserved, ensuring old_root cannot be flushed before
new_root:
Before: if (up_buf->m_created_cp_id == icp_ctx->id())
After: if (up_buf->m_created_cp_id == icp_ctx->id() &&
down_buf->m_created_cp_id == icp_ctx->id())
The same fix is mirrored in index_cp.cpp process_txn_record so that journal
recovery rebuilds an identical DAG. The sanity check is tightened: only a
buffer that was itself created in the current CP must not point to another
same-CP-new up_buffer; older downs pointing to new ups are now valid.
Fix 2 — Detect and promote committed new roots during recovery (covers Bug B)
After the main recovery loop, newly introduced new_root_candidates tracking
collects every committed node whose direct parent is an unwritten meta_buf,
provided the index table already has a persisted SB root
(has_persisted_root() == true). For each ordinal, the highest-level
candidate is selected and set_root_from_committed_buf() is called to update
the in-memory SB (root_node, root_link_version, btree_depth) and the btree's
own root pointer before the next forced CP writes the corrected SB to disk.
The first-CP path (SB root still empty) is intentionally excluded so that
recovery_completed() can build a fresh root as before.
Fix 3 — Harden repair_root_node fallback (covers Bug C)
Added two guards before applying the edge repair:
a. Buffer validity: skip if raw_buffer is null, the node magic is invalid,
or node_id does not match blkid — the buffer was never written to disk.
b. Level invariant: read the candidate next_bnode from disk; if its level is
≥ old_root's level (partial root-split state), skip the repair rather than
corrupting the edge.
A third early return handles the case where next_bnode is already empty,
meaning new_root was already promoted and there is nothing to repair.
Fix 4 — Complete node_id validation in set_root_from_committed_buf
repair_root_node and repair_node both perform a three-way check
(raw_buffer != nullptr, is_valid_node, node_id == blkid). The initial
implementation of set_root_from_committed_buf checked only the first two.
Added the node_id != blkid guard to match, preventing a disk block whose
node magic is valid but whose node_id disagrees with its blkid from being
promoted as the new tree root.
- prune_up_buffers: guard against cascading into committed ancestors; only
schedule durable (valid magic + matching node_id) buffers for repair.
- update_up_buffer_counters: skip detaching committed, non-freed buffers from
their parent so they remain available for new-root promotion.
- recover_buf: return early for meta buffers (tracked separately via
new_root_candidates; loading them as vdev nodes corrupts m_dirtied_cp_id).
- read_buf: reject invalid or mismatched blocks before init_node to surface
disk corruption early with a clear error instead of a debug assertion.
- IndexBuffer::to_string: replace bare {...} delimiters with [...] to prevent
fmt from misinterpreting them as format placeholders when the string is later
re-parsed by HS_*_ASSERT_CMP macros.
- btree_common.ipp: include next_bnode value in "Broken child linkage" message;
use enum_name(ret) for node-read failure reasons.
- Version bump: 7.5.14 → 7.5.15.
Added two regression tests to IndexCrashTest:
CrashAtMetaBufOnFirstRootSplit
Phase 1: insert max_keys/2 entries and flush cleanly so the tree is at a
stable leaf-only state (btree_depth=0, SB root = old_leaf).
Phase 2: set crash_flush_on_meta, insert max_keys+1 more entries to force
the first root split (leaf → level=1), trigger CP. The flip fires
after new_root and old_leaf are both durable but before meta_buf
writes, leaving the SB stale.
Recovery: Fix 2 detects new_root (cand_level=1 > btree_depth=0) and promotes
it; Fix 3 would catch candidate.level(0) >= old_leaf.level(0) if
Fix 2 had not applied. Tree integrity verified via reapply_after_crash
and get_all.
CrashAtMetaBufOnSecondRootSplit
Phase 1: insert max_keys+1 entries to trigger the first root split cleanly,
then flush (tree now at level=1, SB up to date).
Phase 2: set crash_flush_on_meta, insert max_keys² more entries to force the
second root split (level=1 → level=2), trigger CP. Same crash
mechanics as above.
Recovery: same verification.
JacksonYao287
force-pushed
the
SDSTOR-22729
branch
from
July 24, 2026 07:40
1f4d59f to
96f4aad
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A B-tree root split (tree height N → N+1) proceeds in three steps:
Allocate new_root and call on_root_changed(new_root) — updates the
in-memory superblock (SB) and calls transact_bufs(meta_buf, new_root_buf),
which links meta_buf → new_root_buf in the CP flush DAG.
2. split_node(new_root, old_root) modifies old_root in memory
(edge_info = EMPTY, next_bnode = child_node2) and calls
transact_nodes({child_node2}, {}, old_root, new_root), which invokes
link_buf(new_root_buf, old_root_buf).
3. The on-disk SB is only written at the very end of CP flush, after all
node buffers complete.
Three bugs combined to corrupt the tree when a SIGKILL landed after step 2
but before the SB write:
Bug A — link_buf Condition 1 created a flat flush DAG
Condition 1 bypassed new_root_buf whenever it was newly created in the
current CP, regardless of whether old_root_buf was new or old. As a result,
old_root_buf linked directly to meta_buf instead of through new_root_buf,
making new_root_buf and old_root_buf siblings under meta_buf with no ordering
between them. old_root could therefore reach disk in its transient split
state (edge_info=EMPTY, next_bnode=child_node2) before new_root, widening
the crash window.
Bug B — Recovery discarded a committed new_root
When the SB write did not complete, the recovery loop saw new_root_buf's
parent (meta_buf) as uncommitted and threw new_root_buf away. The persisted
SB still pointed to old_root.
Bug C — repair_root_node applied an unsafe edge repair
With old_root as the recovered tree root, repair_root_node read
old_root.next_bnode (= child_node2, level N) and set it as old_root's edge
child (also level N). This violated the B-tree invariant
child.level == parent.level − 1, causing validate_node to abort with
"Child node level mismatch" on the next B-tree access.
The same bug sequence applies to the first root split (leaf → level=1) as
well as any subsequent split.
Fix 1 — Restore correct flush-DAG ordering (covers Bug A)
Changed link_buf Condition 1 in wb_cache.cpp to bypass up_buf only when
BOTH up_buf AND down_buf were created in the current CP. When down_buf is
an older node (e.g. old_root), the dependency chain through up_buf
(new_root) is preserved, ensuring old_root cannot be flushed before
new_root:
The same fix is mirrored in index_cp.cpp process_txn_record so that journal
recovery rebuilds an identical DAG. The sanity check is tightened: only a
buffer that was itself created in the current CP must not point to another
same-CP-new up_buffer; older downs pointing to new ups are now valid.
Fix 2 — Detect and promote committed new roots during recovery (covers Bug B)
After the main recovery loop, newly introduced new_root_candidates tracking
collects every committed node whose direct parent is an unwritten meta_buf,
provided the index table already has a persisted SB root
(has_persisted_root() == true). For each ordinal, the highest-level
candidate is selected and set_root_from_committed_buf() is called to update
the in-memory SB (root_node, root_link_version, btree_depth) and the btree's
own root pointer before the next forced CP writes the corrected SB to disk.
The first-CP path (SB root still empty) is intentionally excluded so that
recovery_completed() can build a fresh root as before.
Fix 3 — Harden repair_root_node fallback (covers Bug C)
Added two guards before applying the edge repair:
a. Buffer validity: skip if raw_buffer is null, the node magic is invalid,
or node_id does not match blkid — the buffer was never written to disk.
b. Level invariant: read the candidate next_bnode from disk; if its level is
≥ old_root's level (partial root-split state), skip the repair rather than
corrupting the edge.
A third early return handles the case where next_bnode is already empty,
meaning new_root was already promoted and there is nothing to repair.
Fix 4 — Complete node_id validation in set_root_from_committed_buf
repair_root_node and repair_node both perform a three-way check
(raw_buffer != nullptr, is_valid_node, node_id == blkid). The initial
implementation of set_root_from_committed_buf checked only the first two.
Added the node_id != blkid guard to match, preventing a disk block whose
node magic is valid but whose node_id disagrees with its blkid from being
promoted as the new tree root.
schedule durable (valid magic + matching node_id) buffers for repair.
their parent so they remain available for new-root promotion.
new_root_candidates; loading them as vdev nodes corrupts m_dirtied_cp_id).
disk corruption early with a clear error instead of a debug assertion.
fmt from misinterpreting them as format placeholders when the string is later
re-parsed by HS_*_ASSERT_CMP macros.
use enum_name(ret) for node-read failure reasons.
Added two regression tests to IndexCrashTest:
CrashAtMetaBufOnFirstRootSplit
Phase 1: insert max_keys/2 entries and flush cleanly so the tree is at a
stable leaf-only state (btree_depth=0, SB root = old_leaf).
Phase 2: set crash_flush_on_meta, insert max_keys+1 more entries to force
the first root split (leaf → level=1), trigger CP. The flip fires
after new_root and old_leaf are both durable but before meta_buf
writes, leaving the SB stale.
Recovery: Fix 2 detects new_root (cand_level=1 > btree_depth=0) and promotes
it; Fix 3 would catch candidate.level(0) >= old_leaf.level(0) if
Fix 2 had not applied. Tree integrity verified via reapply_after_crash
and get_all.