Skip to content

feat(wal): compact whole segments below the bound - #106

Merged
hmdsefi merged 41 commits into
plan/v0.1from
task/E2.6
Oct 20, 2026
Merged

hmdsefi merged 41 commits into
plan/v0.1from
task/E2.6

Conversation

@hmdsefi

@hmdsefi hmdsefi commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

wal.Log.Truncate now deletes old segment files, so a long-running log stays within a bounded amount of disk instead of growing forever. Stacked on #102 (the segmented file log).

chainlog/wal/compact.go (Truncate calls it once its bound checks pass), with b = min(before, last finalize + 1):

  1. Select every segment except the active one whose last offset is below b; if there are none, return.
  2. Write a checkpoint at Head, so the deleted entries are never needed again.
  3. Drop the segments from the index and set Oldest to the first remaining segment's offset.
  4. Close their files, then delete them oldest first, and fsync the directory with SyncAlways.
  5. Log wal compacted at Info with before, segments_deleted and oldest.

A crash at any step leaves a directory Open accepts: the checkpoint is durable before the first deletion, and deletions go from the front. A failed deletion stops the compaction and returns wal: remove <path>: <cause>; the log retries the files it could not delete, oldest first, at the start of every later Truncate, before it deletes any other segment, so the files on disk never have a gap. A Read that copied a segment just before its file was closed gets chainlog.ErrTruncated, never a "file already closed" error. hooks.removed lets a test stop compaction after a deletion, as a crash would.

How it was verified:

  • TestCompaction: 60 blocks with a Finalize every 10, in 512-byte segments with SyncAlways. Truncate(1000) deletes exactly the sealed segments below the last finalize offset plus 1. Oldest is the first remaining segment, and the checkpoint has x == Head. The wal compacted line carries the three keys, a read below Oldest is ErrTruncated, and reopening gives the same log.
  • TestCompactionBound: entries after the last Finalize. A bound at a segment's last offset keeps that segment, a bound at its first offset drops the ones before, and Truncate(1000) stops at the last finalize offset plus 1. The checkpoint holds Head and that offset.
  • TestCompactionOrder: with SyncAlways, the checkpoint and its directory are fsynced before the first deletion, the deletions go oldest first, and the directory is fsynced after the last; dropped files are closed.
  • TestCompactionRemoveFailure: a deletion that keeps failing blocks every later deletion; once it works, the next Truncate finishes. TestCompactionNothingToDelete: a Truncate that selects nothing writes no checkpoint.
  • TestCompactionCrash: compaction stops after its first deletion. Reopening gives Head and State unchanged and Oldest at the second segment; the next Truncate finishes the job.
  • TestCompactionBoundsDisk (skipped with -short): 200,000 blocks in 64 KiB segments, a Finalize every 100 and Truncate(Head+1) every 1,000. The directory never exceeds 4 × 65,536 bytes plus the checkpoint plus 4,096.
  • TestTruncateConcurrentRead: 4 readers read 64 entries from Oldest in a loop while 1,100 entries are appended and truncated every 50. Every error is ErrTruncated, and every entry read equals the one appended.
  • The conformance suite passes in all three configurations, now with real deletion in the Truncate subtests.
  • Review fixes are in the fix: commits.
  • Coverage of the module with -race: 96.7% (CI requires 95%).
  • Gate: gofmt, go mod tidy -diff, go vet (also for windows/amd64), build and tests with and without -race, golangci-lint v2.14.0. All clean.

Risk: deleting a segment a consumer still needs loses its entries. Compaction honors both the caller's bound and the last finalize offset, and the conformance Truncate subtests check entries at or above the bound stay readable.

Closes #11

# Conflicts:
#	chainlog/wal/hooks.go
Base automatically changed from task/E2.4 to plan/v0.1 October 20, 2026 14:33
@hmdsefi
hmdsefi merged commit f333405 into plan/v0.1 Oct 20, 2026
9 checks passed
@hmdsefi
hmdsefi deleted the task/E2.6 branch October 20, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant