Skip to content

feat(wal): add the segmented file log - #102

Merged
hmdsefi merged 32 commits into
plan/v0.1from
task/E2.4
Oct 20, 2026
Merged

hmdsefi merged 32 commits into
plan/v0.1from
task/E2.4

Conversation

@hmdsefi

@hmdsefi hmdsefi commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Adds chainlog/wal, the durable chainlog.Log: the log lives in one directory as a lock file, a meta file, numbered segment files of records and a checkpoint of the validator state. It passes the same conformance suite as the in-memory log.

Stacked on #98 (the Log interface and conformance suite), with #99 (the meta and checkpoint files) merged in, because it needs both. Until #99 lands, this PR's diff also shows #99's files; the new code is wal.go, segment.go, recover.go, hooks.go, lock_*.go and their tests.

  • wal.go: Options, SyncPolicy, Open and the Log methods.
    • Append validates the batch with State.CheckBatch under the write lock. It then encodes the batch into one buffer, writes it with one Write, and fsyncs with SyncAlways, holding only the append lock. Finally it publishes the batch: state, index and head, then it wakes the waiters.
    • When a batch does not fit the active segment, Append rolls first: it writes a checkpoint of the state at Head, then creates the next segment.
    • A failed write or fsync makes every later Append and Truncate fail with ErrFailed (sticky), and leaves Head and State unchanged.
    • Read copies the segments it needs under the read lock and reads with ReadAt without it. Every 256th record is indexed, and Decode checks each record's CRC again. A read error turns into ErrClosed or ErrTruncated when the log closed or dropped that range meanwhile.
  • recover.go: Open's work.
    1. Remove stray .tmp files and classify the directory.
    2. Create a new log, or finish a creation interrupted after meta.
    3. Read meta and the checkpoint.
    4. Scan every segment, checking that offsets are contiguous, that no sealed segment is empty and that every record is good.
    5. Check that the checkpoint fits the segments, then replay only the entries after it.
  • lock_unix.go / lock_other.go: a non-blocking flock on lock; other systems fail with errors.ErrUnsupported.
  • Truncate checks its bounds and the sticky failure; deleting segments comes in feat(wal): compact whole segments below the bound #106 (compaction), and cutting a torn tail on Open in feat(wal): recover a torn tail on open #104. Both should land with this PR.
  • Spec and plan: two interface details in the plan's Deviations. hooks.removed comes with compaction, and scanSegment takes (f, first).

How it was verified:

  • The conformance suite passes three times: never/0, never/256 (a roll every few entries) and always/4096.
  • TestOpenOptions and TestNewDirectory (files, modes 0600 and 0750, random ID, stray .tmp removed, other files kept); TestOpenAfterInterruptedCreate; TestMetaCorruptOpen (8 cases; the directory listing with sizes is unchanged after each failed Open).
  • TestLock: a second Open in the same process fails, and a child process killed with SIGKILL while holding the lock leaves the directory openable.
  • TestPersistence: 50 entries equal a Memory log fed the same entries, and still do after reopening.
  • TestRollAndCheckpoint: segment sizes, a batch larger than a segment gets its own segment, the checkpoint is at the last roll, and reopening replays only the entries after it.
  • TestCheckpointOpen (9 cases, plus a checkpoint just before the first segment that opens) and TestOpenScanErrors (6 cases); TestOpenCorruptActive.
  • TestWriteFailure (EIO through a write hook, sticky error texts, reopen recovers), TestReadCorrupt, TestObservability (wal opened, wal segment rolled), and TestOpenAfterInterruptedRoll: after a roll whose write failed, with or without the new empty segment, reopening gives the same log and Append continues at Head+1.
  • GOOS=windows GOARCH=amd64 go vet ./chainlog/wal: clean.
  • The review fix commits add: wal.MaxWindowCap (10,000,000) so a checkpoint always fits its 1 GiB limit; meta and checkpoint read through a size bound; I/O errors as wal: <op> <path>: <cause>; and tests for the last finalize offset through Append, checkpoint and replay, the checkpoint's lower bound, the index after a reopen (past record 512), the roll boundary, a wrong offset under Read and a segment named 0.
  • Every fsync goes through one hook: TestSyncOrder pins the fsyncs of creation, append and roll, and TestSyncFailure fails each of them. TestReadClose closes the log under 8 readers.
  • Read skips the records before from by their headers: reading one entry 255 records past an index point takes 23 µs (BenchmarkReadTail).
  • ExampleOpen shows the smallest use.
  • Coverage of the module with -race: 96.5% (CI requires 95%).
  • Gate: gofmt, go mod tidy -diff, go vet, build and tests with and without -race, golangci-lint v2.14.0. All clean.

Benchmarks on an M-series Mac (EVMSYNC_PERF=1 go test -run '^TestPerf$' ./chainlog/wal), blocks of 10 logs with 3 topics and 64 bytes of data:

Benchmark Result Target
Append/never 7.1 µs/op, 22 allocs ≤ 15 µs
Append/always 4.2 ms/op (fsync with F_FULLFSYNC) report only
Read, 256 entries per call 382 µs/call, about 670,000 entries/s ≥ 200,000 entries/s
Open of a 256 MiB log without checkpoint 302 ms, about 850 MiB/s ≥ 150 MiB/s

Risk: this is where entries become durable. A bug here loses or corrupts entries after a crash. The interrupted-create and interrupted-roll tests, the corruption cases and the conformance suite in all three configurations guard against that.

Closes #9

Base automatically changed from task/E2.2 to plan/v0.1 October 20, 2026 14:31
# Conflicts:
#	chainlog/wal/checkpoint_test.go
#	chainlog/wal/fileio.go
#	chainlog/wal/meta_test.go
@hmdsefi
hmdsefi merged commit 2dd35a0 into plan/v0.1 Oct 20, 2026
9 checks passed
@hmdsefi
hmdsefi deleted the task/E2.4 branch October 20, 2026 14:33
hmdsefi added a commit that referenced this pull request Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant