fix(rocksdb): lower L0 trigger and tighten slowdown/stop thresholds to prevent OOM

With pin_l0_filter_and_index_blocks_in_cache enabled, each L0 SST file
pins its bloom filter and index blocks in the block cache. With the old
settings (trigger=64, 256 MB write buffers), L0 could accumulate to the
stop threshold — at ~9.75 MB pinned per file, this consumed ~3.7 GB
across 3 DBs, overflowing the 2 GB block cache and spilling to heap,
triggering OOM during mainnet initial sync around block 750k.

Lower the trigger to 32 and set slowdown=×3 (96 files) and stop=×4
(128 files). With 128 MB write buffers (~4.9 MB filter blocks per file),
pinned metadata at the stop threshold is ~1.88 GB across 3 DBs — within
the 2 GB cache at all times.

Set the write buffer CLI flag accordingly: --db-write-buffer-size-mb=128

Author: philippem

src/new_index/db.rs

@@ -104,20 +104,24 @@ impl DB {
         // default trigger of 4, while keeping the file count — and therefore
         // bloom-filter memory and lookup cost — bounded.
         //
-        // With bloom filters at 10 bits/key and a 512 MB write buffer, each L0
-        // file has ~7.8 M keys, so its filter block is ~9.75 MB. At 64 files
-        // that is ~625 MB of pinned filter blocks — well within an 8 GB cache.
-        // Each lookup checks 64 bloom filters (fast, in-memory) and reads from
-        // only ~0.64 files on average (1 % false-positive rate × 64 files).
+        // With bloom filters at 10 bits/key and a 128 MB write buffer, each L0
+        // file has ~3.9 M keys, so its filter block is ~4.9 MB. At the slowdown
+        // threshold (96 files) that is ~470 MB of pinned filter blocks per DB,
+        // ~1.41 GB across 3 DBs — within a 2 GB cache. At the stop threshold
+        // (128 files) it is ~628 MB per DB / ~1.88 GB total, still within bounds.
+        // Previously trigger=64 with 256 MB buffers caused pinned metadata to
+        // overflow the 2 GB cache at L0=128 (~3.7 GB), spilling to uncontrolled
+        // heap and triggering OOM. Trigger=32 + slowdown=96 keeps the peak safe
+        // while allowing enough L0 accumulation for good bulk-load throughput.
         //
         // Set slowdown/stop triggers well above the compaction trigger so writes
         // are never stalled while background compaction catches up.
         // Disable the pending-compaction-bytes stall so the large backlog that
         // builds up during the bulk load does not block writes.
-        const L0_BULK_TRIGGER: i32 = 64;
+        const L0_BULK_TRIGGER: i32 = 32;
         db_opts.set_level_zero_file_num_compaction_trigger(L0_BULK_TRIGGER);
-        db_opts.set_level_zero_slowdown_writes_trigger(L0_BULK_TRIGGER * 4);
-        db_opts.set_level_zero_stop_writes_trigger(L0_BULK_TRIGGER * 8);
+        db_opts.set_level_zero_slowdown_writes_trigger(L0_BULK_TRIGGER * 3);
+        db_opts.set_level_zero_stop_writes_trigger(L0_BULK_TRIGGER * 4);
         db_opts.set_hard_pending_compaction_bytes_limit(0);
         db_opts.set_soft_pending_compaction_bytes_limit(0);