Skip to content

feat(backends): add LongCat video and avatar generation#10792

Merged
mudler merged 2 commits into
masterfrom
codex/avatar-video-backend
Jul 12, 2026
Merged

feat(backends): add LongCat video and avatar generation#10792
mudler merged 2 commits into
masterfrom
codex/avatar-video-backend

Conversation

@localai-bot

@localai-bot localai-bot commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

Description

Adds a dedicated longcat-video Python gRPC backend for LocalAI, supporting LongCat-Video text-to-video, image-to-video, and LongCat-Video-Avatar 1.5 audio-driven avatar generation.

The change extends /video with safely staged audio and video-generation parameters, wires distributed audio staging, supports CUDA 12/13 x86_64 and CUDA 13 ARM64 backend images, and exposes audio/reference-image controls in the React Studio UI.

Capability design

  • Adds declarative known_input_modalities and known_output_modalities model-config fields.
  • Capability discovery merges declared modalities with existing inference in canonical order.
  • LongCat gallery and importer recipes declare their actual I/O contract; generic model-capability code no longer branches on the LongCat backend or checkpoint name.
  • The model configuration editor and API-discovery documentation expose the same declarations.

Model gallery

  • longcat-video: text + image input, video output.
  • longcat-video-avatar-1.5: text + image + audio input, video output.
  • Hugging Face imports emit the same self-describing recipe metadata.

Documentation

Adds a dedicated LongCat Video guide covering gallery installation, DGX Spark and NVIDIA ARM64 guidance, BF16/SDPA versus INT8 tradeoffs, Studio usage, text-to-video, image-to-video, avatar API examples, load options, request parameters, troubleshooting, and current limits. The general backend, video-generation, model-configuration, API-discovery, feature index, and whats-new pages link to it.

Backend notes

  • Pins the upstream LongCat-Video source and applies a minimal SDPA compatibility patch verified against that exact revision.
  • Avatar mode supports multi-segment generation, BF16, optional INT8 quantization, and distillation settings.
  • Input staging is bounded to 128 MiB with redirect, HTTP-status, path, and cleanup handling.
  • Fixes the patch context that caused the first PR backend-image jobs to fail; replacement CUDA 12/13 amd64 and CUDA 13 arm64 jobs are running.

Testing

  • Full pre-commit make lint and make test-coverage-check; composite Go coverage 52.1% versus the 48.5% baseline.
  • Pre-commit make test-ui-coverage-check from the initial implementation (215 Playwright tests; UI line coverage 47.95%).
  • python3 backend/python/longcat-video/test.py (10 tests, 1 skipped).
  • Exact pinned-upstream patch apply check plus Python byte-compilation of both patched modules.
  • Targeted config-capability, importer, video-endpoint, and distributed-staging tests.
  • React production build, focused Playwright tests, Ruff, shellcheck, generated protobuf, Swagger, YAML/JSON, and requirements validation.

Full checkpoint inference and a complete backend container build were not run locally: the model set requires roughly 83 GiB of downloads and NVIDIA CUDA/DGX-class hardware. CI is responsible for the backend image builds.

Signed commits

  • Yes, I signed my commits.

The human submitter should review and add the required DCO sign-off before merge.

mudler added 2 commits July 12, 2026 17:50
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web]
Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook.

Assisted-by: Codex:GPT-5 [web]
@mudler mudler merged commit b00422e into master Jul 12, 2026
67 checks passed
@mudler mudler deleted the codex/avatar-video-backend branch July 12, 2026 21:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants