Priority: expedited spike. Embeddings never persist on macOS, so recall stays lexical-only. The individual pieces (describe via Portkey, local embedding, Deep Lake reads/writes) all work in isolation. The daemon just cannot save a computed embedding back to the store.
Env: nectar 0.3.4, honeycomb 0.8.0, doctor 0.4.3, hive 0.7.0, Node 22, macOS (launchd-managed daemons), hosted Deep Lake (api.deeplake.ai).
Bug 1 (primary blocker): the long-lived daemon's Deep Lake transport hangs on stale keep-alive sockets
Steady-state daemon logs:
enricher hydrate failed (non-fatal): request aborted after 15000ms
durable write-back failed (non-fatal): request aborted after 15000ms
project-context.bridge ... op:"touchIdentity" ... TransportError: request aborted after 15000ms
Under multi-project load, even the daemon's own control API on 127.0.0.1:3854 wedges the same way.
It is not the store. Identical SQL against the identical endpoint (POST ${endpoint}/workspaces/${ws}/tables/query, same bearer token + X-Activeloop-Org-Id) from a fresh short-lived Node process is fast every time:
| Query |
Time |
SELECT COUNT(*) |
398ms |
| full 1,789-row table incl. FLOAT4[768] embeddings (3.9 MB) |
1.14s |
INSERT one row |
338ms |
DELETE one row |
929ms |
The daemon's identical calls abort at exactly 15000ms. Tell-tale timing: right after a daemon restart it works for a few seconds (watched auraip go 0 -> 10 embedded), then every subsequent write-back hangs and the count rolls back to 0. That is a fresh connection pool working, then Deep Lake's load balancer dropping the idle socket, and undici reusing the dead socket until the AbortController fires.
Where: hive-graph/deeplake-transport.js -> DeeplakeSqlTransport. DEFAULT_TRANSPORT_TIMEOUT_MS = 15_000 (hardcoded, no env override), AbortController near line ~103.
Spike fixes to try:
- Give the transport its own undici
Agent with keepAliveTimeout well under Deep Lake's idle cutoff, or disable socket reuse for these calls.
- Retry-on-timeout that forces a brand-new connection (new dispatcher /
Connection: close) instead of reusing the pool, especially after N consecutive timeouts.
- Make the timeout configurable (e.g.
NECTAR_DEEPLAKE_TIMEOUT_MS) as a stopgap. Connection handling is the real fix.
Bug 2: honeycomb embed daemon wedges on the onnxruntime mutex on first inference
honeycomb/embeddings/embed-daemon.js (:3851, nomic-embed-text-v1.5 q8, 768-dim). Warmup succeeds and /health reports ready, but the first real POST /embed {"text":...} hangs forever, and afterwards even /health dies ("mutex lock failed: Invalid argument" class of onnxruntime hang).
OMP_NUM_THREADS=1 alone does NOT fix it. OMP_WAIT_POLICY=PASSIVE does. Proof: launching the same daemon standalone with OMP_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVE served 20/20 sequential embeds clean, single embed in 8ms, stayed healthy.
Fix: set these inside the embed-daemon entrypoint itself (process.env.OMP_WAIT_POLICY ??= 'PASSIVE'; process.env.OMP_NUM_THREADS ??= '1') rather than relying on inherited launchd env, or set onnxruntime session options intraOpNumThreads: 1 / interOpNumThreads: 1 explicitly in the transformers.js pipeline. Env inheritance is fragile (see bug 3) and clearly is not reaching the spawned child reliably (defaultSpawnChild / scrubChildEnv in daemon/index.js).
Bug 3: nectar install strips custom launchd env
Regenerating the plist wipes NECTAR_PORTKEY_* (and any OMP vars), which silently disables brooding:
brooding is dormant; reason: "portkey_disabled"; missing: NECTAR_PORTKEY_ENABLED, NECTAR_PORTKEY_API_KEY, NECTAR_PORTKEY_CONFIG
Fix: merge/preserve existing EnvironmentVariables on install, or read the Portkey/OpenRouter creds from honeycomb's vault (~/.apiary/honeycomb/.secrets/<orgId>__default/) directly so nectar and honeycomb share one credential source.
Minor: describe_status decode bug
nectar hive-graph: invalid describe_status value from Deep Lake mapped to failed: "�????". Direct SQL reads clean described/pending, but the daemon reads garbled bytes for some rows and maps them to failed. Looks like an encoding mismatch (string vs bytes) in how describe_status is written or read.
Net
Fix bug 1 and embeddings actually persist. Bug 2 makes the local embedder reliable. Bug 3 stops the config from silently disappearing.
Priority: expedited spike. Embeddings never persist on macOS, so recall stays lexical-only. The individual pieces (describe via Portkey, local embedding, Deep Lake reads/writes) all work in isolation. The daemon just cannot save a computed embedding back to the store.
Env: nectar 0.3.4, honeycomb 0.8.0, doctor 0.4.3, hive 0.7.0, Node 22, macOS (launchd-managed daemons), hosted Deep Lake (api.deeplake.ai).
Bug 1 (primary blocker): the long-lived daemon's Deep Lake transport hangs on stale keep-alive sockets
Steady-state daemon logs:
Under multi-project load, even the daemon's own control API on 127.0.0.1:3854 wedges the same way.
It is not the store. Identical SQL against the identical endpoint (
POST ${endpoint}/workspaces/${ws}/tables/query, same bearer token +X-Activeloop-Org-Id) from a fresh short-lived Node process is fast every time:SELECT COUNT(*)INSERTone rowDELETEone rowThe daemon's identical calls abort at exactly 15000ms. Tell-tale timing: right after a daemon restart it works for a few seconds (watched auraip go 0 -> 10 embedded), then every subsequent write-back hangs and the count rolls back to 0. That is a fresh connection pool working, then Deep Lake's load balancer dropping the idle socket, and undici reusing the dead socket until the AbortController fires.
Where:
hive-graph/deeplake-transport.js->DeeplakeSqlTransport.DEFAULT_TRANSPORT_TIMEOUT_MS = 15_000(hardcoded, no env override), AbortController near line ~103.Spike fixes to try:
AgentwithkeepAliveTimeoutwell under Deep Lake's idle cutoff, or disable socket reuse for these calls.Connection: close) instead of reusing the pool, especially after N consecutive timeouts.NECTAR_DEEPLAKE_TIMEOUT_MS) as a stopgap. Connection handling is the real fix.Bug 2: honeycomb embed daemon wedges on the onnxruntime mutex on first inference
honeycomb/embeddings/embed-daemon.js(:3851, nomic-embed-text-v1.5 q8, 768-dim). Warmup succeeds and/healthreports ready, but the first realPOST /embed {"text":...}hangs forever, and afterwards even/healthdies ("mutex lock failed: Invalid argument" class of onnxruntime hang).OMP_NUM_THREADS=1alone does NOT fix it.OMP_WAIT_POLICY=PASSIVEdoes. Proof: launching the same daemon standalone withOMP_NUM_THREADS=1 OMP_WAIT_POLICY=PASSIVEserved 20/20 sequential embeds clean, single embed in 8ms, stayed healthy.Fix: set these inside the embed-daemon entrypoint itself (
process.env.OMP_WAIT_POLICY ??= 'PASSIVE'; process.env.OMP_NUM_THREADS ??= '1') rather than relying on inherited launchd env, or set onnxruntime session optionsintraOpNumThreads: 1 / interOpNumThreads: 1explicitly in the transformers.js pipeline. Env inheritance is fragile (see bug 3) and clearly is not reaching the spawned child reliably (defaultSpawnChild/scrubChildEnvindaemon/index.js).Bug 3:
nectar installstrips custom launchd envRegenerating the plist wipes
NECTAR_PORTKEY_*(and any OMP vars), which silently disables brooding:Fix: merge/preserve existing
EnvironmentVariableson install, or read the Portkey/OpenRouter creds from honeycomb's vault (~/.apiary/honeycomb/.secrets/<orgId>__default/) directly so nectar and honeycomb share one credential source.Minor: describe_status decode bug
nectar hive-graph: invalid describe_status value from Deep Lake mapped to failed: "�????". Direct SQL reads cleandescribed/pending, but the daemon reads garbled bytes for some rows and maps them tofailed. Looks like an encoding mismatch (string vs bytes) in howdescribe_statusis written or read.Net
Fix bug 1 and embeddings actually persist. Bug 2 makes the local embedder reliable. Bug 3 stops the config from silently disappearing.