Skip to content

feat: 可选 TwelveLabs 视频理解 provider(Pegasus 问答 + Marengo 向量)/ opt-in TwelveLabs video understanding#12

Open
mohit-twelvelabs wants to merge 1 commit into
zrt-ai-lab:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

feat: 可选 TwelveLabs 视频理解 provider(Pegasus 问答 + Marengo 向量)/ opt-in TwelveLabs video understanding#12
mohit-twelvelabs wants to merge 1 commit into
zrt-ai-lab:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

简介(中文)

为 ViNote 新增可选TwelveLabs 视频理解 provider,让「视频问答」可以直接理解视频画面(动作、场景、画面中的物体等转录文本无法表达的信息),而不仅仅依赖转录文本。

  • Pegasus —— 直接对视频内容进行问答(TwelveLabsVideoQAService,接口与现有 VideoQAService 完全一致,可无缝替换)。
  • Marengo —— 生成 512 维多模态向量,用于对抽取出的视频片段做语义检索。

完全可选、非破坏性:通过 VIDEO_QA_PROVIDER=twelvelabs + TWELVELABS_API_KEY 显式启用;不配置时,现有「转录文本 + OpenAI」路径完全不受任何影响。依赖以可选 extra 形式提供(uv sync --extra twelvelabs)。

可在 https://twelvelabs.io 免费获取 API key,有较慷慨的免费额度。


What this adds (English)

This PR adds an opt-in TwelveLabs video-understanding provider so ViNote's video Q&A can reason over the actual video (actions, scenes, on-screen objects — things a transcript can't capture), not just the transcript text.

  • Pegasus — answers questions directly over the video. TwelveLabsVideoQAService mirrors the existing VideoQAService interface (answer_question_stream / is_available), so it slots in as a drop-in QA provider.
  • Marengo — produces 512-dim multimodal embeddings for semantic search over extracted clips.

Why it helps ViNote

ViNote already does transcript-based Q&A. For visually-rich videos (tutorials, demos, sports, anything where what's on screen matters), transcript-only answers miss a lot. Pegasus closes that gap, and Marengo gives a foundation for "find the clip that matches this query" semantic search.

Opt-in / non-breaking

  • Disabled by default. Only active when VIDEO_QA_PROVIDER=twelvelabs and TWELVELABS_API_KEY are set.
  • With no key configured, get_video_qa_service() returns the existing VideoQAService unchanged — zero behavior change.
  • twelvelabs is an optional dependency ([project.optional-dependencies]uv sync --extra twelvelabs), so existing installs are untouched.

How it was tested

  • Live-verified the TwelveLabs contracts against the real API with the official SDK (twelvelabs>=1.2.8):
    • Marengo marengo3.0 text embedding returns a 512-dim vector.
    • Pegasus pegasus1.5 analyze(...) returns a text answer (verified via the asset_id path on an uploaded clip).
  • 5 pytest tests pass (backend/tests/test_twelvelabs_service.py): no-network unit tests for config toggles, provider selection, sentence-chunking, and the empty-URL guard, plus a live Marengo 512-dim contract test that auto-skips without TWELVELABS_API_KEY.

Notes / verified gotchas baked in

  • Pegasus 1.5 doesn't accept a bare video_id — this uses a public URL or an uploaded asset_id (direct upload capped at 200MB, documented in code + README).
  • Pegasus requires the analyzed window ≥ 4s, and max_tokens ≥ 512 (clamped in config).

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

- 新增 TwelveLabsVideoQAService:通过 Pegasus 直接理解视频画面进行问答,
  接口与现有 VideoQAService 一致,可作为视频问答的可选 provider
- 新增 Marengo 512 维多模态向量接口,用于视频片段语义检索
- 通过 VIDEO_QA_PROVIDER=twelvelabs + TWELVELABS_API_KEY 显式启用,
  默认关闭,不配置时现有转录文本 + OpenAI 路径完全不受影响
- twelvelabs 作为可选依赖(uv sync --extra twelvelabs)
- 补充 .env.example、README 文档与单元/实时契约测试

@zrt-ai-lab zrt-ai-lab left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution — the optional provider direction is useful, but this PR is not ready to merge yet. Please address the following items and update the branch:

  1. Rebase onto the latest main and resolve the current conflicts in README.md and pyproject.toml.
  2. Fix the actual video input flow. VideoQA.tsx sends the original user input as video_url (commonly a YouTube/Bilibili page URL), while TwelveLabs VideoContext_Url requires a direct raw media URL and does not accept video-hosting page links. Please resolve a direct media URL or upload an asset and wait until it is ready before analysis.
  3. Regenerate and commit uv.lock. Currently uv sync --frozen --extra twelvelabs fails with Extra twelvelabs is not defined. Also make sure start.sh, Windows startup, and Docker do not remove or omit the optional dependency when the provider is enabled.
  4. Fix the PR's own offline test failure. Current result: 3 passed, 1 failed, 1 skipped; test_qa_requires_video_url raises TwelveLabs API 不可用 before the expected ValueError.
  5. Preserve the default OpenAI QA behavior. Empty transcript should only be accepted when the active provider is TwelveLabs; with the default provider it must still return HTTP 400 instead of opening an SSE stream that later emits an error.
  6. Make the repository release checks pass. The TWELVELABS_API_KEY=your-twelvelabs-key example is currently flagged by scripts/scan_release.py. Please use an accepted placeholder and run both release validation scripts.
  7. Align with the clean release repository policy: do not add backend/tests/ to the published tree.
  8. Either implement and wire the advertised Marengo video-vector functionality, or narrow the PR title/docs to the functionality actually delivered. At present only an unused embed_text helper is added; there is no video embedding integration. Prefer the current Embed v2 API if embeddings remain in scope.

Please re-run and report:

  • python3 scripts/validate_release.py
  • python3 scripts/scan_release.py
  • the TwelveLabs offline tests
  • frontend production build
  • smoke tests for both the default OpenAI path and the enabled TwelveLabs path

Once these are fixed, we can review the updated PR again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants