feat(llm): Promote TwelveLabs video RAG components into xpacks core#256
feat(llm): Promote TwelveLabs video RAG components into xpacks core#256iapoorv01 wants to merge 3 commits into
Conversation
iapoorv01
force-pushed
the
feature/twelvelabs-video-rag
branch
from
a526624 to
f60073e
Compare
This commit promotes the TwelveLabs integration from the template directory into the native pathway.xpacks.llm core library, as proposed in pathwaycom#255. Key changes: - pyproject.toml: Introduced a new [twelvelabs] optional dependency extra. - embedders.py: Promoted MarengoEmbedder into pathway.xpacks.llm.embedders. It inherits from BaseEmbedder and leverages an async pathway (_aembed_one) via asyncio.gather for concurrent embedding generation, preventing thread blocking. It implements the lazy ImportError pattern. - parsers.py: Promoted TwelveLabsVideoParser into pathway.xpacks.llm.parsers. Implemented proper resource cleanup logic using try/finally blocks to explicitly delete TwelveLabs assets (when delete_assets=True) preventing asset flooding on the API. Also implements the lazy ImportError pattern. - test_twelvelabs.py: Ported the unit tests into python/pathway/xpacks/llm/tests/. Contains no-network test coverage via stubbed SDK mocks. The live smoke test is maintained but securely gated behind the TWELVELABS_API_KEY existence check. - CHANGELOG.md: Added an entry under [Unreleased] -> Added documenting the promotion. This effectively extends Pathway's live-sync multimodal indexing capabilities to handle full video ingestion over the TwelveLabs network natively.
| "Office365-REST-Python-Client >= 2.5.3", | ||
| ] | ||
| twelvelabs = [ | ||
| "twelvelabs >= 0.3.0", |
There was a problem hiding this comment.
This floor is too low and is a regression against the template: the original requirements.txt, module docstring, and ImportError message all require twelvelabs>=1.2.8. The code needs at least 1.2.x — AsyncTwelveLabs and twelvelabs.types.video_context don't exist in 0.3.x, and the assets client only appears in 1.2.0.
An environment that already has an older SDK installed satisfies >=0.3.0, so pip won't upgrade it, and the parser then fails at runtime with ModuleNotFoundError (and the embedder raises the misleading "package is required" hint even though the package is installed). Please change to twelvelabs >= 1.2.8.
| ) | ||
|
|
||
|
|
||
| def _resolve_twelvelabs_api_key(api_key: str | None) -> str: |
There was a problem hiding this comment.
These two helpers (_resolve_twelvelabs_api_key, _build_twelvelabs_client) are verbatim copies of the ones added to embedders.py in this same PR — in the template they existed once. Since the split across two modules forces a shared home, please move them to python/pathway/xpacks/llm/_utils.py (both modules already import from it) and import them in both places.
While you're there, consider replacing the hand-rolled try/except ImportError with the xpack's standard with optional_imports("twelvelabs"): — it's what every other optional-SDK component in these files uses and produces the same pip install pathway[twelvelabs] guidance.
| Returns: | ||
| A list of 512-dimensional ``numpy`` arrays, one per input string. | ||
| """ | ||
| import asyncio |
There was a problem hiding this comment.
Minor: in the template these were top-level imports; the port moved import asyncio (and import os/import time in the other helpers) inside the functions. embedders.py already imports asyncio at module level — please keep stdlib imports at the top; only the twelvelabs import needs to stay lazy.
2 participants
Introduction
This PR resolves #255 by officially promoting the
TwelveLabsVideoParserandMarengoEmbedderout of thellm-appexamples template and integrating them natively into thepathway.xpacks.llmcore library.This enables native, first-class Video RAG capabilities (parsing raw video bytes into Pegasus text descriptions and retrieving them using a shared multimodal embedding space) directly out of the box in Pathway.
Context
Currently, Pathway's core LLM xpack handles PDFs, DOCX, and slides exceptionally well, but has lacked native multimodal video handling. Previously introduced as an app template in PR #129, the TwelveLabs components proved stable and highly valuable. Promoting them to the core
xpackslibrary eliminates the need for developers to maintain boilerplate API scaffolding when building multimodal search applications.Design Decisions & Approach:
twelvelabs.pyfile, we strictly adhered to Pathway's architectural patterns by natively integratingMarengoEmbedderintoembedders.pyandTwelveLabsVideoParserintoparsers.py.twelvelabsSDK is not a hard dependency. It sits securely in a new[twelvelabs]optional extra block inpyproject.toml. The components leverage a robusttry/except ImportErrorguidance pattern to safely prompt users to runpip install pathway[twelvelabs]only if they attempt to instantiate the classes without the SDK.asyncio.gatherpipeline for the Embedder to prevent thread-blocking under load. Furthermore, the Parser correctly utilizes atry/finallyblock to proactively delete TwelveLabs assets (delete_assets=Trueby default) so users do not accidentally flood their cloud workspace with zombie assets during repeated pipeline runs.How has this been tested?
All tests have been ported from the template environment into the native test suite at
python/pathway/xpacks/llm/tests/test_twelvelabs.py.__wrapped__async concurrency behaviors, asset cleanup lifecycle, and fallback failure modes using the strict_FakeClientstubbing pattern.sys.modulesstubs) to align cleanly with the core[tests]environment runtime guarantees.@pytest.mark.skipif(not os.environ.get("TWELVELABS_API_KEY"))to guarantee zero disruptions to the standard CI pipeline.isortandblackhooks.Types of changes
Related issue(s):
TwelveLabsVideoParserandMarengoEmbedderfrom the Video RAG template intopathway.xpacks.llm#255Checklist: