Field report (nectar 0.1.3, Windows, ~85k-file repo)
After the index walk finishes (memory plateaus then declines to ~670MB), CPU keeps climbing past 900s with flat/declining memory: a busy-loop, not indexing. /health stays unresponsive throughout.
Root-cause investigation (current main, 2026-07-04): two named suspects
-
Per-batch known-path sweeps scale with store size, not batch size. Every registration cycle batch recomputes knownPaths() and missingPaths() (src/registration/service.ts:303-312 snapshot comment), and missingPaths performs an existsOnDisk stat for EVERY known path. With ~85k rows in the store, a single settled watch event (one file save) triggers ~85k synchronous statSync calls. Any trickle of events (editor autosave, build output, log write) re-runs the sweep, keeping the CPU pegged indefinitely while the event loop starves. This matches the flat-memory, climbing-CPU signature exactly.
-
Shared-ignore cache misses spawn git synchronously per path. createSharedIgnore's gitignore leg resolves a snapshot miss via runGitCheckIgnore = spawnSync("git", ["check-ignore", ...]) per path (src/registration/ignore.ts:187-199, wired at :282-291). Any watch-event storm over paths not in the ls-files snapshot (untracked build artifacts, temp files) spawns a synchronous git process per observation; on Windows each spawn is tens to hundreds of ms of blocked loop.
Compounding: the fs.watch recursive intake on Windows can deliver high event volumes on large trees, and each settled event feeds suspect 1; each raw observation can feed suspect 2 before debouncing (the ignore test runs pre-debounce, src/registration/fs-watch.ts:165-173).
Ask
- Make the known/missing-path reconciliation incremental (indexes/dirty sets) instead of per-batch full sweeps; a single-path settle must be O(1)-ish, never O(store).
- Batch or cache negative check-ignore resolutions (and/or refresh the snapshot instead of per-path spawnSync); never spawn a process per watch event.
- Add a soak test: sustained single-file churn on a large synthetic store must keep CPU bounded and /health responsive.
Related: the /health event-loop starvation during indexing is filed separately.
Field report (nectar 0.1.3, Windows, ~85k-file repo)
After the index walk finishes (memory plateaus then declines to ~670MB), CPU keeps climbing past 900s with flat/declining memory: a busy-loop, not indexing. /health stays unresponsive throughout.
Root-cause investigation (current main, 2026-07-04): two named suspects
Per-batch known-path sweeps scale with store size, not batch size. Every registration cycle batch recomputes
knownPaths()andmissingPaths()(src/registration/service.ts:303-312snapshot comment), andmissingPathsperforms anexistsOnDiskstat for EVERY known path. With ~85k rows in the store, a single settled watch event (one file save) triggers ~85k synchronousstatSynccalls. Any trickle of events (editor autosave, build output, log write) re-runs the sweep, keeping the CPU pegged indefinitely while the event loop starves. This matches the flat-memory, climbing-CPU signature exactly.Shared-ignore cache misses spawn git synchronously per path.
createSharedIgnore's gitignore leg resolves a snapshot miss viarunGitCheckIgnore=spawnSync("git", ["check-ignore", ...])per path (src/registration/ignore.ts:187-199, wired at:282-291). Any watch-event storm over paths not in the ls-files snapshot (untracked build artifacts, temp files) spawns a synchronous git process per observation; on Windows each spawn is tens to hundreds of ms of blocked loop.Compounding: the fs.watch recursive intake on Windows can deliver high event volumes on large trees, and each settled event feeds suspect 1; each raw observation can feed suspect 2 before debouncing (the ignore test runs pre-debounce,
src/registration/fs-watch.ts:165-173).Ask
Related: the /health event-loop starvation during indexing is filed separately.