Hybrid keyword search: quality refinements (RRF tuning, CJK unigrams, collision noise)

[Rasbandit]

Non-blocking quality follow-ups from the #610 review:

1. Tune fusion. Hybrid currently uses Qdrant default RRF (k=60, equal leg weights). Evaluate weighted RRF (Qdrant ≥1.17) or DBSF (≥1.11) once there's a representative query set — no public nDCG benchmark exists, so this needs real-vault evaluation.
2. Single-char CJK queries. The tokenizer emits character bigrams for CJK runs, so a single-character CJK query (猫) won't match multi-char CJK content (飼猫 only emitted bigram 飼猫). Consider also emitting CJK unigrams, or document the limitation.
3. u32 collision noise at scale. HMAC dims are truncated to u32; per-user vocab in the low-millions starts accruing collisions (two terms → one bucket, weights summed — graceful ranking noise, not a correctness/security issue). Quantify the ranking-noise floor; widen the dim space if it bites.
4. avgdl re-normalization (Backfill worker for notes_fts keyword index (+ Tiger Cloud migration tool) #605): the current ReindexKeyword stub recomputes against drifting avgdl during the pass; the real impl should snapshot avgdl once and pass it via the existing Bm25 injected-param seam.

Refs #595, #605, #610

[Rasbandit]

Decision (how to expose tunables — no env-var sprawl): expose all search tuning (bm25_k1, bm25_b, avgdl_bootstrap, rrf_k, rrf_weights) via a single Elixir application-config namespace config :engram, :keyword_search, [...], matching the existing Application.get_env(:engram, :embedder|:reranker|:qdrant_collection) pattern. Defaults stay in code (current module values); config is override-only; self-host overrides in its config file, SaaS in runtime.exs.

Explicitly NO new env vars — this aligns with the KMS secret-consolidation direction (32 SSM params → 3). Layering: code default → app config → (only if a concrete per-deploy runtime-override need ever arises) fold into an existing consolidated blob, never a new var. Implementation is just wiring the module reads through Application.get_env(:engram, :keyword_search, []) with current values as fallbacks.

[Rasbandit]

Important finding (from the staging repopulate): the current ReindexKeyword stub re-enqueues notes through EmbedNote, but EmbedNote skips any note whose content is unchanged (embed_note.ex: %Note{content_hash: hash, embed_hash: hash} -> skip). So re-enqueuing alone is a no-op for already-embedded notes — it does NOT rebuild Qdrant points or recompute BM25 weights.

The real #605 re-normalizer must force the re-index — e.g. clear embed_hash (set NULL) for the targeted notes before/while enqueuing, or pass a force: true arg that bypasses the EmbedNote content-hash gate. Without that, the worker can't repopulate a dropped collection or re-normalize against a new avgdl.

Confirmed on staging: clearing embed_hash then enqueuing the reindex correctly recreated the named collections and repopulated all points; enqueuing without clearing would have done nothing.