Consolidate Embedding Metadata Storage into Vector Database Layer

[kyle65463]

Description

Previously, vCache treated the vector store and the embedding metadata store as separate, pluggable components. But in practice, nearly all production-grade vector databases (like Pinecone, Weaviate, Qdrant, Milvus) already handle metadata natively.

Keeping a second, parallel storage layer inside vCache:

• duplicated functionality already handled by the DB
• added unnecessary serialization and network overhead
• introduced risk of state mismatch (e.g., vector exists but metadata doesn’t)

This PR merges the two into a unified VectorDB abstraction that manages both the vectors and their associated EmbeddingMetadataObj.

For libraries that don’t support metadata (like raw hnswlib), which are typically used for small- to mid-sized in-memory datasets, we fall back to lightweight in-memory storage for metadata.

Impact

Who it affects:

• vCache users: configuration is simpler (vector_db=… is now the only storage setting)
• vCache contributors: one less interface to maintain

What changes:

• Embeddings and metadata are now stored, retrieved, updated, and evicted through the same code path.

Why it matters:

• Faster lookups, fewer moving parts
• Eliminates an entire class of consistency bugs
• Cleaner imports
• Enables atomic vector+metadata operations

Proposed Solution

1. Create a single VectorDB base class with methods:
   • add(embedding, metadata)
   • get_knn(embedding, k)
   • get_metadata(id) / update_metadata(id, metadata)
2. Add a reference HNSWLibVectorDB implementation with in-memory metadata (since hnswlib doesn’t support metadata)
3. Remove EmbeddingMetadataStorage and related strategy classes
4. Replace all EmbeddingStore with VectorDB
5. Extensibility: Users needing separate metadata storage (e.g. for compliance or multi-service sharing) can create their custom VectorDB and override metadata methods. If this becomes common, we can extend the API to support both "combined" and "split" modes.

Acceptance Criteria

[ ] Public APIs continue to work through the new unified path
[ ] All tests updated and passing

Risks & Dependencies

• Some libraries (like raw hnswlib) don’t support metadata, we can handle their metadata with an in-process dict
• If separate storage is needed again in the future, we can support it through subclassing