Skip to content

perf: use bounded LRU for local cache#9

Merged
lan17 merged 5 commits into
mainfrom
codex/lru-local-cache
Jul 13, 2026
Merged

perf: use bounded LRU for local cache#9
lan17 merged 5 commits into
mainfrom
codex/lru-local-cache

Conversation

@lan17

@lan17 lan17 commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Summary

Replace DialCache's hand-written per-use-case Map/FIFO local cache with the mature lru-cache v11 implementation.

The local layer now provides:

  • per-entry TTLs
  • one global entry-count bound per DialCache instance
  • true least-recently-used eviction
  • cached undefined support through the existing entry wrapper
  • exact expiry checks using a nonzero monotonic clock

Motivation

The previous implementation created one Map per use case and applied localMaxSize independently to each Map. As the number of registered use cases grew, total local entries were not bounded by the configured limit. Eviction followed insertion order, so frequently read entries did not gain retention priority.

Using a well-tested local-cache library keeps TTL and eviction mechanics behind the existing local-cache abstraction while enforcing the configured process-local bound.

Design

LocalCache owns one LRUCache<string, LocalEntry<unknown>> keyed by the existing full DialCache URN. The URN already includes the use case and all key dimensions, so a single cache preserves key isolation while enforcing one aggregate limit.

The cache uses sparse maxSize accounting with every entry weighted as one, avoiding allocation proportional to a configured limit. TTL remains configured per entry:

  • resolved local TTL seconds are converted to integer milliseconds on set
  • ttlResolution: 0 checks expiry on every access
  • a nonzero integer performance.now() clock keeps TTL timing monotonic and avoids the library's zero-timestamp sentinel
  • the stored TTL accounts for lru-cache's strict expiry comparison, preserving DialCache's historical exact-boundary behavior
  • expiry remains absolute rather than sliding on reads

Values remain wrapped in LocalEntry, allowing an application result of undefined to remain distinguishable from a missing cache entry.

Consumer impact

The exported symbols, method signatures, and public type shapes are unchanged. localMaxSize already existed on DialCacheConfig; this PR documents and tightens its contract:

  • the default remains 10,000 entries
  • the limit now applies globally across all use cases in one DialCache instance
  • 0 disables local storage
  • negative, fractional, non-finite, and unsafe integer limits fail fast with RangeError
  • the bound counts entries rather than estimating JavaScript object memory
  • cache hits update LRU recency

The package engine changes from Node >=18.17 to 20 || >=22, matching lru-cache v11. Redis behavior, request coalescing, runtime ramping, metrics APIs, invalidation semantics, cache-key construction, and local TTL units remain unchanged.

Validation

  • pnpm check — typecheck, 99 unit tests with coverage thresholds, build, and packaged ESM/CJS/TypeScript consumer
  • pnpm test:integration — 32 Docker-backed Redis/Valkey tests
  • packaged consumer verified on Node 20
  • pnpm audit --prod — no known vulnerabilities
  • git diff --check
  • iterative review/fix/re-review loops with a final fresh confirmation review returning no findings

@lan17 lan17 marked this pull request as ready for review July 13, 2026 04:01
@lan17 lan17 merged commit 24ca4ee into main Jul 13, 2026
4 checks passed
@lan17 lan17 deleted the codex/lru-local-cache branch July 13, 2026 04:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant