feat: 可选 TwelveLabs 视频理解 provider(Pegasus 问答 + Marengo 向量)/ opt-in TwelveLabs video understanding#12
Open
mohit-twelvelabs wants to merge 1 commit into
Conversation
- 新增 TwelveLabsVideoQAService:通过 Pegasus 直接理解视频画面进行问答, 接口与现有 VideoQAService 一致,可作为视频问答的可选 provider - 新增 Marengo 512 维多模态向量接口,用于视频片段语义检索 - 通过 VIDEO_QA_PROVIDER=twelvelabs + TWELVELABS_API_KEY 显式启用, 默认关闭,不配置时现有转录文本 + OpenAI 路径完全不受影响 - twelvelabs 作为可选依赖(uv sync --extra twelvelabs) - 补充 .env.example、README 文档与单元/实时契约测试
zrt-ai-lab
requested changes
Jul 14, 2026
zrt-ai-lab
left a comment
Owner
There was a problem hiding this comment.
Thanks for the contribution — the optional provider direction is useful, but this PR is not ready to merge yet. Please address the following items and update the branch:
- Rebase onto the latest
mainand resolve the current conflicts inREADME.mdandpyproject.toml. - Fix the actual video input flow.
VideoQA.tsxsends the original user input asvideo_url(commonly a YouTube/Bilibili page URL), while TwelveLabsVideoContext_Urlrequires a direct raw media URL and does not accept video-hosting page links. Please resolve a direct media URL or upload an asset and wait until it is ready before analysis. - Regenerate and commit
uv.lock. Currentlyuv sync --frozen --extra twelvelabsfails withExtra twelvelabs is not defined. Also make surestart.sh, Windows startup, and Docker do not remove or omit the optional dependency when the provider is enabled. - Fix the PR's own offline test failure. Current result: 3 passed, 1 failed, 1 skipped;
test_qa_requires_video_urlraisesTwelveLabs API 不可用before the expectedValueError. - Preserve the default OpenAI QA behavior. Empty
transcriptshould only be accepted when the active provider is TwelveLabs; with the default provider it must still return HTTP 400 instead of opening an SSE stream that later emits an error. - Make the repository release checks pass. The
TWELVELABS_API_KEY=your-twelvelabs-keyexample is currently flagged byscripts/scan_release.py. Please use an accepted placeholder and run both release validation scripts. - Align with the clean release repository policy: do not add
backend/tests/to the published tree. - Either implement and wire the advertised Marengo video-vector functionality, or narrow the PR title/docs to the functionality actually delivered. At present only an unused
embed_texthelper is added; there is no video embedding integration. Prefer the current Embed v2 API if embeddings remain in scope.
Please re-run and report:
python3 scripts/validate_release.pypython3 scripts/scan_release.py- the TwelveLabs offline tests
- frontend production build
- smoke tests for both the default OpenAI path and the enabled TwelveLabs path
Once these are fixed, we can review the updated PR again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
简介(中文)
为 ViNote 新增可选的 TwelveLabs 视频理解 provider,让「视频问答」可以直接理解视频画面(动作、场景、画面中的物体等转录文本无法表达的信息),而不仅仅依赖转录文本。
TwelveLabsVideoQAService,接口与现有VideoQAService完全一致,可无缝替换)。完全可选、非破坏性:通过
VIDEO_QA_PROVIDER=twelvelabs+TWELVELABS_API_KEY显式启用;不配置时,现有「转录文本 + OpenAI」路径完全不受任何影响。依赖以可选 extra 形式提供(uv sync --extra twelvelabs)。可在 https://twelvelabs.io 免费获取 API key,有较慷慨的免费额度。
What this adds (English)
This PR adds an opt-in TwelveLabs video-understanding provider so ViNote's video Q&A can reason over the actual video (actions, scenes, on-screen objects — things a transcript can't capture), not just the transcript text.
TwelveLabsVideoQAServicemirrors the existingVideoQAServiceinterface (answer_question_stream/is_available), so it slots in as a drop-in QA provider.Why it helps ViNote
ViNote already does transcript-based Q&A. For visually-rich videos (tutorials, demos, sports, anything where what's on screen matters), transcript-only answers miss a lot. Pegasus closes that gap, and Marengo gives a foundation for "find the clip that matches this query" semantic search.
Opt-in / non-breaking
VIDEO_QA_PROVIDER=twelvelabsandTWELVELABS_API_KEYare set.get_video_qa_service()returns the existingVideoQAServiceunchanged — zero behavior change.twelvelabsis an optional dependency ([project.optional-dependencies]→uv sync --extra twelvelabs), so existing installs are untouched.How it was tested
twelvelabs>=1.2.8):marengo3.0text embedding returns a 512-dim vector.pegasus1.5analyze(...)returns a text answer (verified via theasset_idpath on an uploaded clip).backend/tests/test_twelvelabs_service.py): no-network unit tests for config toggles, provider selection, sentence-chunking, and the empty-URL guard, plus a live Marengo 512-dim contract test that auto-skips withoutTWELVELABS_API_KEY.Notes / verified gotchas baked in
video_id— this uses a public URL or an uploadedasset_id(direct upload capped at 200MB, documented in code + README).max_tokens≥ 512 (clamped in config).You can grab a free API key at https://twelvelabs.io — there's a generous free tier.