Summary
An idle nectar daemon (enricher live) reads Deep Lake on every cycle even when there is nothing to enrich, which keeps Activeloop's serverless compute pod from scaling to zero and bills compute (uptime) continuously.
This is one instance of an Apiary-wide pattern: any daemon that polls Deep Lake while idle bills continuously, because doctor keeps the daemons alive 24/7. The honeycomb sibling is legioncodeinc/honeycomb#267. See "Fleet-wide context" below.
Root cause
The steady-state enricher re-seeds its in-memory mirror from Deep Lake on every cycle: refreshWorkingSet() -> DeepLakeEnricherStore.refresh -> a SELECT ... FROM hive_graph_versions. Because that refresh runs unconditionally at the top of runEnricherCycle, before the empty-work early return, an idle repo issues one Deep Lake read every poll interval (30s), indefinitely. A steady read cadence keeps the Activeloop pod from ever reaching its idle window, so it never scales to zero.
(nectar's Deep Lake transport is statelessly per-query, which is good — the cost is the refresh frequency, not a held socket.)
Fix
Fixed in PR #19: legioncodeinc/nectar#19. Gate the refresh on a one-bit dirty signal that each active project's registration StoreBridge marks on a durable write (onDurableWrite -> createProjectContext -> cli.ts), and that the primary enricher consumes via shouldRefresh. An idle repo marks nothing dirty -> the enricher issues zero Deep Lake reads -> the pod scales to zero. A file change arms exactly one refresh on the next tick, so enrichment latency is unchanged. Also adds NECTAR_ENRICHER_ENABLED as an operator kill switch and the primitive for a "one describer device per project" fleet layout.
The PR is rebased onto current main (0.3.3, multi-root architecture) and mergeable; the suite passes.
Impact
- Continuous Activeloop
compute (uptime) billing whenever the enricher is live (Portkey configured), even with an idle repo.
- Latent until the enricher is activated, but that is nectar's primary use case.
Fleet-wide context (why this matters across nectar / honeycomb / doctor)
The Apiary is designed for always-on daemons: doctor keeps honeycomb, hive, and nectar alive 24/7. That makes "cheap when idle" a fleet requirement, not a per-daemon nicety, because any idle Deep Lake polling bills continuously.
- nectar: this issue, fixed by #19.
- honeycomb: the same class of bug on the job-queue poll -> legioncodeinc/honeycomb#267 (idle 30s Deep Lake poll; opt-in idle-suspend PR to follow).
- doctor: not a cost source itself (its
/health probe reads a cached bit, no Deep Lake round-trip), but it is the amplifier — by keeping daemons reliably up, it guarantees any idle-cost bug bills 24/7.
Both fixes are the same shape: stop polling Deep Lake when idle, wake on real work. Landing them together makes the fleet genuinely scale-to-zero when nobody is coding.
Repro
- Configure Portkey + Deep Lake creds so the enricher goes live; bind a project.
- Do nothing (no file changes in the repo).
- Sample the nectar daemon's outbound Deep Lake (
:443) sockets:
lsof -nP -iTCP -sTCP:ESTABLISHED -a -p "$(pgrep -f nectar)" | grep -v 127.0.0.1 | grep -c ':443'
- Observe a read every ~30s and a flat
compute (uptime) line on Activeloop billing.
Summary
An idle nectar daemon (enricher live) reads Deep Lake on every cycle even when there is nothing to enrich, which keeps Activeloop's serverless compute pod from scaling to zero and bills
compute (uptime)continuously.This is one instance of an Apiary-wide pattern: any daemon that polls Deep Lake while idle bills continuously, because doctor keeps the daemons alive 24/7. The honeycomb sibling is legioncodeinc/honeycomb#267. See "Fleet-wide context" below.
Root cause
The steady-state enricher re-seeds its in-memory mirror from Deep Lake on every cycle:
refreshWorkingSet()->DeepLakeEnricherStore.refresh-> aSELECT ... FROM hive_graph_versions. Because that refresh runs unconditionally at the top ofrunEnricherCycle, before the empty-work early return, an idle repo issues one Deep Lake read every poll interval (30s), indefinitely. A steady read cadence keeps the Activeloop pod from ever reaching its idle window, so it never scales to zero.(nectar's Deep Lake transport is statelessly per-query, which is good — the cost is the refresh frequency, not a held socket.)
Fix
Fixed in PR #19: legioncodeinc/nectar#19. Gate the refresh on a one-bit dirty signal that each active project's registration
StoreBridgemarks on a durable write (onDurableWrite->createProjectContext->cli.ts), and that the primary enricher consumes viashouldRefresh. An idle repo marks nothing dirty -> the enricher issues zero Deep Lake reads -> the pod scales to zero. A file change arms exactly one refresh on the next tick, so enrichment latency is unchanged. Also addsNECTAR_ENRICHER_ENABLEDas an operator kill switch and the primitive for a "one describer device per project" fleet layout.The PR is rebased onto current
main(0.3.3, multi-root architecture) and mergeable; the suite passes.Impact
compute (uptime)billing whenever the enricher is live (Portkey configured), even with an idle repo.Fleet-wide context (why this matters across nectar / honeycomb / doctor)
The Apiary is designed for always-on daemons: doctor keeps honeycomb, hive, and nectar alive 24/7. That makes "cheap when idle" a fleet requirement, not a per-daemon nicety, because any idle Deep Lake polling bills continuously.
/healthprobe reads a cached bit, no Deep Lake round-trip), but it is the amplifier — by keeping daemons reliably up, it guarantees any idle-cost bug bills 24/7.Both fixes are the same shape: stop polling Deep Lake when idle, wake on real work. Landing them together makes the fleet genuinely scale-to-zero when nobody is coding.
Repro
:443) sockets:lsof -nP -iTCP -sTCP:ESTABLISHED -a -p "$(pgrep -f nectar)" | grep -v 127.0.0.1 | grep -c ':443'compute (uptime)line on Activeloop billing.