Characterize and guard watermark cardinality and Redis Cluster hot slots

[lan17]

Priority

P1 — High — resolve before broad production adoption or large-scale use.

Problem

Every tracked write creates or extends a baseline watermark. The default floor is four hours, all variants for one keyType/id share one cluster slot, and tracked reads stay on primaries.

Evidence:

• Baseline watermark creation and TTL extension
• Four-hour default floor
• Per-ID shared cluster hash tag
• Tracked node-redis reads are primary-bound

At 10,000 new tracked IDs per second, a four-hour floor can retain roughly 144 million watermark keys. A celebrity ID can also concentrate every use-case/args variant on one primary.

Acceptance criteria

• Benchmark memory and operations for high-cardinality tracked writes.
• Benchmark cluster throughput for uniform and highly skewed IDs.
• Document the distinct ID rate × watermark lifetime capacity model.
• Add stable tracked-write, watermark-suppressed-write, and capacity signals.
• Evaluate sparse-watermark alternatives without weakening correctness.
• Document primary-only and per-ID hot-slot limits.
• Keep logical clock correctness in Remove or explicitly bound Redis wall-clock dependence in invalidation ordering #32.