Repository navigation
Conversation
# Conflicts: # chainlog/wal/hooks.go
# Conflicts: # chainlog/bench_test.go # chainlog/perf_test.go
# Conflicts: # chainlog/bench_test.go # chainlog/perf_test.go
# Conflicts: # chainlog/wal/checkpoint_test.go # chainlog/wal/fileio.go # chainlog/wal/meta_test.go
# Conflicts: # chainlog/wal/hooks.go # chainlog/wal/wal.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
wal.Log.Truncatenow deletes old segment files, so a long-running log stays within a bounded amount of disk instead of growing forever. Stacked on #102 (the segmented file log).chainlog/wal/compact.go(Truncatecalls it once its bound checks pass), withb = min(before, last finalize + 1):b; if there are none, return.Head, so the deleted entries are never needed again.Oldestto the first remaining segment's offset.SyncAlways.wal compactedat Info withbefore,segments_deletedandoldest.A crash at any step leaves a directory
Openaccepts: the checkpoint is durable before the first deletion, and deletions go from the front. A failed deletion stops the compaction and returnswal: remove <path>: <cause>; the log retries the files it could not delete, oldest first, at the start of every laterTruncate, before it deletes any other segment, so the files on disk never have a gap. AReadthat copied a segment just before its file was closed getschainlog.ErrTruncated, never a "file already closed" error.hooks.removedlets a test stop compaction after a deletion, as a crash would.How it was verified:
TestCompaction: 60 blocks with aFinalizeevery 10, in 512-byte segments withSyncAlways.Truncate(1000)deletes exactly the sealed segments below the last finalize offset plus 1.Oldestis the first remaining segment, and the checkpoint hasx == Head. Thewal compactedline carries the three keys, a read belowOldestisErrTruncated, and reopening gives the same log.TestCompactionBound: entries after the lastFinalize. A bound at a segment's last offset keeps that segment, a bound at its first offset drops the ones before, andTruncate(1000)stops at the last finalize offset plus 1. The checkpoint holdsHeadand that offset.TestCompactionOrder: withSyncAlways, the checkpoint and its directory are fsynced before the first deletion, the deletions go oldest first, and the directory is fsynced after the last; dropped files are closed.TestCompactionRemoveFailure: a deletion that keeps failing blocks every later deletion; once it works, the nextTruncatefinishes.TestCompactionNothingToDelete: aTruncatethat selects nothing writes no checkpoint.TestCompactionCrash: compaction stops after its first deletion. Reopening givesHeadandStateunchanged andOldestat the second segment; the nextTruncatefinishes the job.TestCompactionBoundsDisk(skipped with-short): 200,000 blocks in 64 KiB segments, aFinalizeevery 100 andTruncate(Head+1)every 1,000. The directory never exceeds 4 × 65,536 bytes plus the checkpoint plus 4,096.TestTruncateConcurrentRead: 4 readers read 64 entries fromOldestin a loop while 1,100 entries are appended and truncated every 50. Every error isErrTruncated, and every entry read equals the one appended.Truncatesubtests.fix:commits.-race: 96.7% (CI requires 95%).go mod tidy -diff,go vet(also for windows/amd64), build and tests with and without-race, golangci-lint v2.14.0. All clean.Risk: deleting a segment a consumer still needs loses its entries. Compaction honors both the caller's bound and the last finalize offset, and the conformance
Truncatesubtests check entries at or above the bound stay readable.Closes #11