Add lz4 and zstd compression support to the CLFUS RAM cache#13257
Open
phongn wants to merge 9 commits into
Open
Add lz4 and zstd compression support to the CLFUS RAM cache#13257phongn wants to merge 9 commits into
phongn wants to merge 9 commits into
Conversation
bryancall
reviewed
Jun 11, 2026
bryancall
left a comment
bryancall
left a comment
Contributor
There was a problem hiding this comment.
Took a pass over this — really nice change. The liblzma lzma_stream_buffer_bound() fix is a real correctness improvement, the per-thread zstd context design is clean, and adding compression-integrity tests where there were none is great. A handful of improvement suggestions inline; none are blockers.
The one I'd most encourage looking at is the sticky-null behavior of the thread-local zstd context: a one-time allocation failure silently degrades that thread to uncompressed forever (and evicts valid entries on read) with no log or metric. Details inline.
Member
|
Since I know you probably cant get to the CI output @phongn , here's the failure. It's in the clfus catch test: |
RamCacheCLFUS allocated its hash table, seen filter, and entries but had no destructor, so destroying an instance leaked them. LeakSanitizer flagged this once the new unit test started creating and destroying instances. Add a destructor that releases each entry's data, returns the entries to the allocator, and frees the table and seen filter. The synchronous RAM cache test never calls TEST_DONE(), so the event threads kept running as the process tore down static state at exit and an ET_NET thread incremented the freed Metrics singleton (heap-use-after-free). Shut the event system down from the shared cache test harness at end of run so the threads stop first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Related PR to add zstd and lz4 to our CI hosts: apache/trafficserver-ci#441 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add lz4 and zstd compression support to the CLFUS RAM cache
Summary
This adds two modern compression backends to the CLFUS RAM cache, as optional build-time dependencies:
proxy.config.cache.ram_cache.compress: 4) — a replacement for fastlz: strictly better compression ratio at substantially higher throughput.proxy.config.cache.ram_cache.compress: 5, level 3) — a replacement for libz/deflate-6: comparable ratio at roughly 10× the compression speed and 3× the decompression speed.The existing fastlz/libz/liblzma backends are unchanged and remain valid for their config values; the docs now recommend lz4 over fastlz and zstd over libz.
Benchmarks (lzbench, silesia XML corpus, Xeon Gold 6338, one thread):
Implementation notes
LZ4_compress_default/LZ4_decompress_safeAPI.thread_localZSTD_CCtx/ZSTD_DCtx) withZSTD_compress2and a sticky compression level, avoiding a context allocation per call — relevant since decompression sits on the cache-hit path. This requires zstd ≥ 1.4.0 (the first release with the advanced one-shot API stable); the version floor is enforced infind_package.Fatal, matching the existing liblzma behavior.cmake/FindLZ4.cmakeandcmake/FindZSTD.cmakemodules. zstd detection previously usedfind_package(zstd CONFIG)only, which fails on distributions that don't shipzstd-config.cmake; it now resolves via the module (thezstd::zstdalias shim is kept for builds usingCMAKE_FIND_PACKAGE_PREFER_CONFIG).traffic_layout infonow reportsTS_HAS_LZ4and the lz4 build/runtime versions, mirroring zstd.Testing
The
RamCacheCLFUSclass definition moved from the.ccinto a new private header so unit tests can drivecompress_entries()synchronously. A new Catch2 test (test_RamCacheCLFUS) does store → compress → read-back roundtrips across all five backends, asserting byte-for-byte equality and the expectedRAM_HIT_COMPRESS_*state, plus incompressible-fallback and small-payload cases. No prior test verified compression data integrity for any backend.Drive-by fixes
e->leninstead oflzma_stream_buffer_bound(e->len), causing encode failures (and spurious "incompressible" marking) for data that didn't shrink; it now matches the other backends, with the existingREQUIRED_COMPRESSION/REQUIRED_SHRINKthresholds deciding what to keep.compress: 2was configured — zlib is a required dependency and always available.CI / packaging
liblz4-dev/lz4-develadded to the deb and yum CI images. The Fedora CI image will needlz4-develadded for build coverage of the new backend (zstd-devel is already present).Future work
Compression is currently implemented inside CLFUS only. A follow-up refactor will extract it into a shared layer so all RAM cache algorithms can use it; the new unit test is structured to migrate there.