Design hybrid retrieval API for text, vector, filters, and fusion

[dcfocus]

Motivation

lance-context already supports vector search, metadata/lifecycle filters, and relationship payloads. For production agent memory and RAG-style context recall, callers often need a mix of exact lexical matching and semantic similarity:

• exact identifiers such as service names, policy IDs, error strings, source URIs, or tags
• semantic similarity over embeddings
• scalar filters for tenant/user/agent/session/source/content type/lifecycle scope
• predictable score/fusion metadata so callers can explain why a record was returned

Today the public search path is primarily vector-driven: callers pass an embedding vector, optional filters are applied, and results are ranked by vector distance. Applications that also need text/BM25-style recall have to build a second retrieval path outside lance-context, open the Lance dataset directly, and then fuse/rerank results themselves.

Proposal

Add or design a first-class hybrid retrieval API that can combine text, vector, and scalar-filter channels without turning lance-context into a full RAG framework.

Possible shape:

results = ctx.retrieve(
    text="POLICY-123 rollout risk service-a",
    vector=query_embedding,
    filters={
        "content_type": "text/plain",
        "source_type": "runbook",
        "tags": {"contains": "rollout"},
    },
    limit=10,
    fusion="rrf",
)

The API could return enough ranking detail for debugging and downstream context assembly, for example:

• fused_score
• vector_distance or normalized vector score
• lexical/text score when text retrieval is enabled
• matched channels, such as vector, text, filter
• existing record metadata and optional relationships

Suggested scope for a first PR

• Define the Rust/Python/API DTO shape and behavior for hybrid retrieval.
• Preserve the existing vector-only search(...) behavior for compatibility.
• Support scalar filters and lifecycle visibility consistently with current search.
• Start with a simple, dependency-light fusion strategy such as reciprocal rank fusion when both text and vector candidates are available.

Non-goals

• No document loaders, chunkers, prompt templates, model calls, or full RAG pipeline.
• No external search engine dependency as part of the first version.
• No graph traversal or GraphRAG ranking in the first version, though the API should not preclude future relationship expansion/boosting.
• No mandatory reranker dependency.

Relationship to existing roadmap

Issue #37 describes broader agent-facing architecture, including layered context and hybrid search. This issue is intended as a narrower, implementable retrieval primitive that could serve that roadmap incrementally.

Open questions

• Should this be a new retrieve(...) API, or an extension of search(...)?
• Should lexical retrieval use Lance full-text search/indexes, a lightweight in-process scorer over text payloads, or a staged approach?
• What score fields should be stable public API versus best-effort diagnostics?
• Should relationship expansion be a later option on the same API, or a separate graph retrieval API?

[dcfocus]

I verified this against the current implementation.

Two details worth making explicit:

1. Current search is not just "primarily" vector-driven; it is exclusively vector-driven. Python coerces search(query) into a vector, and core search requires the configured embedding dimension before ranking by L2 distance.

2. Current vector search is also an in-process linear scan over visible/filter-matched records, not an indexed ANN path. Because of that, I think the v1 hybrid API should probably use the same MemWAL-aware listing path plus a lightweight in-process lexical scorer and RRF. Lance FTS/vector indexing can be a later optimization once index freshness semantics are clear.

This keeps the issue scoped as a retrieval primitive rather than turning lance-context into a database or RAG platform.

[dcfocus]

Implemented by #76.