Repository navigation
Conversation
# Conflicts: # chainlog/bench_test.go # chainlog/perf_test.go
# Conflicts: # chainlog/bench_test.go # chainlog/perf_test.go
# Conflicts: # chainlog/wal/checkpoint_test.go # chainlog/wal/fileio.go # chainlog/wal/meta_test.go
# Conflicts: # chainlog/wal/recover.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
wal.Opennow recovers the tail of an interrupted write instead of refusing the log. After a crash duringAppend, the last segment can end with part of a record. Until nowOpenreported that as corruption forever. Stacked on #102 (the segmented file log).chainlog/wal/recover.go: the bad record that ends the scan of the last segment is cut when one of these holds:Cutting means:
Opentruncates the file to the end of the last whole record, logswal torn tail cutat Warn withsegment,positionandbytes, and fsyncs it withSyncAlways. The cut runs only after the checkpoint check passes, so anOpenthat fails changes no file. Any other bad record, and any bad record in an earlier segment, staysErrCorrupt, and no file changes: a bad record whose declared end lies inside the file and is followed by more data is damage, not a torn write.How it was verified:
TestRecoverEveryCut: 30 blocks in 1 KiB segments. For every cut length of the last segment,Opensucceeds withHeadequal to the whole records kept, and the entries equal the originals byte for byte. When bytes were cut, the file ends at the last whole record and the warning carries the segment, position and removed bytes. One moreAppendthen works.TestTornVersusCorrupt, fourteen cases:ErrCorruptwith the files unchanged: a flipped byte in the second-to-last record, in the last record of a sealed segment, or in the first segment; trailing bytes whose first byte is non-zero; zeros with one byte past the first 64 KiB; a zero header followed by a record; a damaged last record followed by one byte; a length flip that would cut records the checkpoint covers; a renamed segment; an empty sealed segment; no segments left.TestRecoverSync: withSyncAlways, the cut segment is fsynced after the cut, and a failed fsync failsOpenwith the I/O error.TestRecoverOneSegment: a log that never rolled.FuzzRecover: append up to 40 blocks, then cut the last segment or flip a byte in it or in any segment, and reopen.Opennever panics and fails only withErrCorrupt. Whatever it keeps equals the originals. A cut always opens, withHeadequal to the records wholly kept. A flipped byte is corruption, unless it is in the very last record, which is cut; a damaged length field in the last segment is a cut when its declared end is at or past the end of the file. The 4 seeds run in everygo test;go test -run '^$' -fuzz '^FuzzRecover$' -fuzztime 10s ./chainlog/walfound nothing.fix:commits.-race: 96.4% (CI requires 95%).go mod tidy -diff,go vet(also for windows/amd64), build and tests with and without-race, golangci-lint v2.14.0. All clean.Risk: cutting too much drops entries that
Appendalready returned. The every-cut test and the fuzz target pin the decision to the three cases. A bit flip in a length field of the last segment can look like case 1 and drop the records after it; the spec accepts that, and the checkpoint check stops it from dropping records the checkpoint covers.Closes #10