Repository navigation
Conversation
# Conflicts: # chainlog/bench_test.go # chainlog/perf_test.go
# Conflicts: # chainlog/bench_test.go # chainlog/perf_test.go
# Conflicts: # chainlog/wal/checkpoint_test.go # chainlog/wal/fileio.go # chainlog/wal/meta_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
chainlog/wal, the durablechainlog.Log: the log lives in one directory as a lock file, ametafile, numbered segment files of records and a checkpoint of the validator state. It passes the same conformance suite as the in-memory log.Stacked on #98 (the Log interface and conformance suite), with #99 (the meta and checkpoint files) merged in, because it needs both. Until #99 lands, this PR's diff also shows #99's files; the new code is
wal.go,segment.go,recover.go,hooks.go,lock_*.goand their tests.wal.go:Options,SyncPolicy,Openand theLogmethods.Appendvalidates the batch withState.CheckBatchunder the write lock. It then encodes the batch into one buffer, writes it with oneWrite, and fsyncs withSyncAlways, holding only the append lock. Finally it publishes the batch: state, index and head, then it wakes the waiters.Appendrolls first: it writes a checkpoint of the state atHead, then creates the next segment.AppendandTruncatefail withErrFailed(sticky), and leavesHeadandStateunchanged.Readcopies the segments it needs under the read lock and reads withReadAtwithout it. Every 256th record is indexed, andDecodechecks each record's CRC again. A read error turns intoErrClosedorErrTruncatedwhen the log closed or dropped that range meanwhile.recover.go:Open's work..tmpfiles and classify the directory.meta.metaand the checkpoint.lock_unix.go/lock_other.go: a non-blockingflockonlock; other systems fail witherrors.ErrUnsupported.Truncatechecks its bounds and the sticky failure; deleting segments comes in feat(wal): compact whole segments below the bound #106 (compaction), and cutting a torn tail onOpenin feat(wal): recover a torn tail on open #104. Both should land with this PR.hooks.removedcomes with compaction, andscanSegmenttakes(f, first).How it was verified:
TestOpenOptionsandTestNewDirectory(files, modes0600and0750, random ID, stray.tmpremoved, other files kept);TestOpenAfterInterruptedCreate;TestMetaCorruptOpen(8 cases; the directory listing with sizes is unchanged after each failedOpen).TestLock: a secondOpenin the same process fails, and a child process killed with SIGKILL while holding the lock leaves the directory openable.TestPersistence: 50 entries equal aMemorylog fed the same entries, and still do after reopening.TestRollAndCheckpoint: segment sizes, a batch larger than a segment gets its own segment, the checkpoint is at the last roll, and reopening replays only the entries after it.TestCheckpointOpen(9 cases, plus a checkpoint just before the first segment that opens) andTestOpenScanErrors(6 cases);TestOpenCorruptActive.TestWriteFailure(EIO through a write hook, sticky error texts, reopen recovers),TestReadCorrupt,TestObservability(wal opened,wal segment rolled), andTestOpenAfterInterruptedRoll: after a roll whose write failed, with or without the new empty segment, reopening gives the same log andAppendcontinues atHead+1.GOOS=windows GOARCH=amd64 go vet ./chainlog/wal: clean.wal.MaxWindowCap(10,000,000) so a checkpoint always fits its 1 GiB limit;metaandcheckpointread through a size bound; I/O errors aswal: <op> <path>: <cause>; and tests for the last finalize offset throughAppend, checkpoint and replay, the checkpoint's lower bound, the index after a reopen (past record 512), the roll boundary, a wrong offset underReadand a segment named 0.TestSyncOrderpins the fsyncs of creation, append and roll, andTestSyncFailurefails each of them.TestReadClosecloses the log under 8 readers.Readskips the records beforefromby their headers: reading one entry 255 records past an index point takes 23 µs (BenchmarkReadTail).ExampleOpenshows the smallest use.-race: 96.5% (CI requires 95%).go mod tidy -diff,go vet, build and tests with and without-race, golangci-lint v2.14.0. All clean.Benchmarks on an M-series Mac (
EVMSYNC_PERF=1 go test -run '^TestPerf$' ./chainlog/wal), blocks of 10 logs with 3 topics and 64 bytes of data:Append/neverAppend/alwaysF_FULLFSYNC)Read, 256 entries per callOpenof a 256 MiB log without checkpointRisk: this is where entries become durable. A bug here loses or corrupts entries after a crash. The interrupted-create and interrupted-roll tests, the corruption cases and the conformance suite in all three configurations guard against that.
Closes #9