Skip to content

Promote llama.cpp Windows CUDA release in README #3250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
LauraGPT merged 2 commits into from
Jul 18, 2026
+25 −11
Merged

Promote llama.cpp Windows CUDA release in README #3250

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
97eff75
Promote llama.cpp Windows CUDA release
LauraGPT Jul 18, 2026
a1718b3
Preserve releases link in llama.cpp README snippets
LauraGPT Jul 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,9 +257,11 @@ bash download-funasr-model.sh sensevoice ./gguf # or: paraformer | nano
hf download FunAudioLLM/SenseVoiceSmall-GGUF sensevoice-small-q8.gguf --local-dir .\gguf
hf download FunAudioLLM/fsmn-vad-GGUF fsmn-vad.gguf --local-dir .\gguf
.\llama-funasr-sensevoice.exe -m .\gguf\sensevoice-small-q8.gguf --vad .\gguf\fsmn-vad.gguf -a audio.wav
# Use the windows-x64-cuda package on RTX 30-class GPUs:

@gemini-code-assist gemini-code-assist Bot Jul 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The prebuilt CUDA binary can also run on newer NVIDIA GPUs (such as RTX 40-class / Ada Lovelace) due to backward compatibility. Specifying "RTX 30-class (or newer)" would be more accurate and prevent users with newer GPUs from thinking they cannot use this package.

Suggested change
# Use the windows-x64-cuda package on RTX 30-class GPUs:
# Use the windows-x64-cuda package on RTX 30-class (or newer) GPUs:

.\llama-funasr-sensevoice.exe -m .\gguf\sensevoice-small-q8.gguf --vad .\gguf\fsmn-vad.gguf -a audio.wav --backend cuda
```

**Prebuilt binaries:** [Releases](https://github.com/modelscope/FunASR/releases) · **Download & quickstart:** [funasr.com/llama-cpp](https://www.funasr.com/llama-cpp.html) · **GGUF models:** [Hugging Face](https://huggingface.co/FunAudioLLM) · **Docs & benchmarks:** [runtime/llama.cpp/](./runtime/llama.cpp/)
**Prebuilt binaries:** [Releases](https://github.com/modelscope/FunASR/releases) · [v0.1.7](https://github.com/modelscope/FunASR/releases/tag/runtime-llamacpp-v0.1.7) · [Windows CUDA zip](https://github.com/modelscope/FunASR/releases/download/runtime-llamacpp-v0.1.7/funasr-llamacpp-windows-x64-cuda.zip) · **Download & quickstart:** [funasr.com/llama-cpp](https://www.funasr.com/llama-cpp.html) · **GGUF models:** [Hugging Face](https://huggingface.co/FunAudioLLM) · **Docs & benchmarks:** [runtime/llama.cpp/](./runtime/llama.cpp/)

[OpenAI API example →](./examples/openai_api/) · [Gradio demo →](./examples/openai_api/GRADIO.md) · [Client recipes →](./examples/openai_api/CLIENTS.md) · [JavaScript/TypeScript recipes →](./examples/openai_api/JAVASCRIPT.md) · [Kubernetes template →](./examples/openai_api/kubernetes/) · [Workflow recipes →](./examples/openai_api/WORKFLOWS.md) · [Postman collection →](./examples/openai_api/POSTMAN.md) · [OpenAPI spec →](./examples/openai_api/OPENAPI.md) · [Security guide →](./examples/openai_api/SECURITY.md) · [Deployment matrix →](./docs/deployment_matrix.md) · [Deployment docs →](./runtime/readme.md) · [Agent integration →](https://modelscope.github.io/FunASR/agent.html)

Expand All @@ -284,6 +286,7 @@ hf download FunAudioLLM/fsmn-vad-GGUF fsmn-vad.gguf --local-dir .\gguf
## What's new

- 2026/07/18: **v1.3.16 on PyPI** — client-driven realtime endpoints for Fun-ASR-Nano. Start one WebSocket session, stream PCM, and send `COMMIT` for each utterance without loading server-side VAD; short utterances finalize and timestamps remain monotonic across commits. Install with `pip install --upgrade funasr`, then run `funasr-realtime-server --endpoint-mode client`. [Guide →](examples/industrial_data_pretraining/fun_asr_nano/docs/realtime_demo.md)
- 2026/07/18: **llama.cpp runtime v0.1.7** — prebuilt Windows CUDA package for SenseVoiceSmall (`funasr-llamacpp-windows-x64-cuda.zip`) plus Linux / macOS / Windows CPU packages. Download the GGUF model, then run `llama-funasr-sensevoice ... --backend cuda` on supported NVIDIA GPUs. [Release →](https://github.com/modelscope/FunASR/releases/tag/runtime-llamacpp-v0.1.7)
- 2026/06/20: **llama.cpp / GGUF runtime** — run SenseVoice / Paraformer / Fun-ASR-Nano on CPU & edge as a single self-contained binary (a whisper.cpp-style alternative), built-in FSMN-VAD, no Python at runtime. Prebuilt binaries for Linux / macOS / Windows + **q8 quantized models (~half the size, same accuracy)**. [runtime/llama.cpp/](./runtime/llama.cpp/) · [Releases](../../releases)
- 2026/06/21: **v1.3.12** on PyPI — rolling fixes (qwen3-asr language codes, glm_asr, vLLM repetition_penalty). `pip install --upgrade funasr`
- 2026/05/24: **vLLM Inference Engine** — 2-3x faster LLM decoding for Fun-ASR-Nano. Streaming WebSocket service with VAD + Speaker Diarization. [Guide →](docs/vllm_guide.md) · [Realtime WS tuning →](docs/vllm_guide.md#67-production-concurrency-and-multi-process-deployment) · [API stability checklist →](docs/vllm_guide.md#production-api-stability-checklist)
Expand Down
5 changes: 4 additions & 1 deletion README_zh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ Whisper 是单个模型，**FunASR 是一个工具箱**——按场景挑模型
## 最新动态

- 2026/07/18：**v1.3.16 已发布到 PyPI** — Fun-ASR-Nano 实时服务新增客户端分句模式。一个 WebSocket 会话可连续发送 PCM，并用 `COMMIT` 提交每个句子；无需加载服务端 VAD，短句可正常结束，多轮时间戳保持递增。执行 `pip install --upgrade funasr` 后，可用 `funasr-realtime-server --endpoint-mode client` 启动。[使用文档 →](examples/industrial_data_pretraining/fun_asr_nano/docs/realtime_demo.md)
- 2026/07/18：**llama.cpp runtime v0.1.7** — 新增 SenseVoiceSmall 的 Windows CUDA 预编译包 `funasr-llamacpp-windows-x64-cuda.zip`，并保留 Linux / macOS / Windows CPU 包。下载 GGUF 模型后，可在支持的 NVIDIA GPU 上运行 `llama-funasr-sensevoice ... --backend cuda`。[发布页 →](https://github.com/modelscope/FunASR/releases/tag/runtime-llamacpp-v0.1.7)
- 2026/05/24：**vLLM 推理引擎** — Fun-ASR-Nano 解码加速 2-3 倍。支持流式 WebSocket 服务（VAD + 说话人分离 + 热词）。[文档 →](docs/vllm_guide_zh.md) · [实时 WS 调优 →](docs/vllm_guide_zh.md#67-生产并发与多进程部署) · [API 稳定性清单 →](docs/vllm_guide_zh.md#生产-api-稳定性清单)
- 2026/05/24：**动态 VAD** — 自适应静音阈值（默认开启），短句不切碎、长句自动切分。[详情 →](docs/vllm_guide_zh.md#7-动态-vad)
- 2026/05/24：**v1.3.3** — `funasr-server` 命令行工具、OpenAI 兼容 API、MCP 服务。`pip install --upgrade funasr`
Expand Down Expand Up @@ -291,9 +292,11 @@ bash download-funasr-model.sh sensevoice ./gguf # 也可使用 paraformer
hf download FunAudioLLM/SenseVoiceSmall-GGUF sensevoice-small-q8.gguf --local-dir .\gguf
hf download FunAudioLLM/fsmn-vad-GGUF fsmn-vad.gguf --local-dir .\gguf
.\llama-funasr-sensevoice.exe -m .\gguf\sensevoice-small-q8.gguf --vad .\gguf\fsmn-vad.gguf -a audio.wav
# RTX 30 系列等架构 86 GPU 可使用 windows-x64-cuda 包：

@gemini-code-assist gemini-code-assist Bot Jul 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The term "架构 86" (architecture 86) is non-standard in CUDA terminology. It is more accurate to refer to it as "Ampere 架构 (算力 8.6)" or "sm_86 架构". Also, mentioning "or newer" (及以上) is helpful since newer GPUs (like RTX 40-series) are backward-compatible.

Suggested change
# RTX 30 系列等架构 86 GPU 可使用 windows-x64-cuda 包：
# RTX 30 系列及以上（Ampere 架构，算力 8.6+）GPU 可使用 windows-x64-cuda 包：

.\llama-funasr-sensevoice.exe -m .\gguf\sensevoice-small-q8.gguf --vad .\gguf\fsmn-vad.gguf -a audio.wav --backend cuda
```

**预编译二进制：** [Releases](https://github.com/modelscope/FunASR/releases) · **下载与快速开始：** [funasr.com/llama-cpp](https://www.funasr.com/llama-cpp.html) · **GGUF 模型：** [Hugging Face](https://huggingface.co/FunAudioLLM) · **文档与评测：** [runtime/llama.cpp/](./runtime/llama.cpp/)
**预编译二进制：** [Releases](https://github.com/modelscope/FunASR/releases) · [v0.1.7](https://github.com/modelscope/FunASR/releases/tag/runtime-llamacpp-v0.1.7) · [Windows CUDA zip](https://github.com/modelscope/FunASR/releases/download/runtime-llamacpp-v0.1.7/funasr-llamacpp-windows-x64-cuda.zip) · **下载与快速开始：** [funasr.com/llama-cpp](https://www.funasr.com/llama-cpp.html) · **GGUF 模型：** [Hugging Face](https://huggingface.co/FunAudioLLM) · **文档与评测：** [runtime/llama.cpp/](./runtime/llama.cpp/)

[OpenAI API 示例 →](./examples/openai_api/README_zh.md) · [Gradio Demo →](./examples/openai_api/GRADIO_zh.md) · [客户端配方 →](./examples/openai_api/CLIENTS.md) · [JavaScript/TypeScript 配方 →](./examples/openai_api/JAVASCRIPT_zh.md) · [Kubernetes 模板 →](./examples/openai_api/kubernetes/README_zh.md) · [工作流配方 →](./examples/openai_api/WORKFLOWS_zh.md) · [Postman 集合 →](./examples/openai_api/POSTMAN_zh.md) · [OpenAPI 规范 →](./examples/openai_api/OPENAPI_zh.md) · [安全指南 →](./examples/openai_api/SECURITY_zh.md) · [部署选型 →](./docs/deployment_matrix_zh.md) · [部署文档 →](./runtime/readme_cn.md) · [Agent 集成 →](https://modelscope.github.io/FunASR/agent.html)

Expand Down
26 changes: 17 additions & 9 deletions runtime/llama.cpp/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,19 @@ cmake --build build -j # -> build/bin/llama-funasr-* (a

### Optional CUDA backend for SenseVoiceSmall

The prebuilt release ZIPs are CPU-first portable packages. To experiment with
SenseVoiceSmall graph execution on NVIDIA GPUs, build from source with ggml CUDA
enabled and select the backend at runtime:
The CPU release ZIPs are portable packages. Tagged releases also publish
`funasr-llamacpp-windows-x64-cuda.zip` for SenseVoiceSmall graph execution on
NVIDIA GPUs that match CUDA architecture 86. Download the CUDA ZIP from

@gemini-code-assist gemini-code-assist Bot Jul 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In CUDA terminology, "CUDA architecture 86" is more precisely referred to as "CUDA compute capability 8.6" or "sm_86 architecture" (which corresponds to the Ampere architecture). Clarifying this will make the technical requirements clearer to users.

Suggested change
NVIDIA GPUs that match CUDA architecture 86. Download the CUDA ZIP from
NVIDIA GPUs with CUDA compute capability 8.6 (Ampere architecture). Download the CUDA ZIP from

[runtime-llamacpp-v0.1.7](https://github.com/modelscope/FunASR/releases/tag/runtime-llamacpp-v0.1.7),
then select the backend at runtime:

```bash
# From the extracted windows-x64-cuda package:
./llama-funasr-sensevoice \
-m sensevoice-small-q8.gguf --vad fsmn-vad.gguf -a sample.wav --backend cuda
```

Build from source to target a different GPU architecture:

```bash
cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
Expand All @@ -73,12 +83,10 @@ cmake --build build-cuda -j --target llama-funasr-sensevoice
```

`--backend cpu` remains the default and is what the portable cross-platform
prebuilt binaries use. Tagged runtime releases also publish a
`funasr-llamacpp-windows-x64-cuda.zip` package when the Windows CUDA release job
passes; it requires an NVIDIA driver compatible with the CUDA Toolkit version
configured by that release job and targets CUDA architecture 86. Build from source
for other GPU architectures. A binary built without `-DGGML_CUDA=ON` exits with a
clear message if `--backend cuda` is requested.
prebuilt binaries use. The CUDA package requires an NVIDIA driver compatible
with the CUDA Toolkit version configured by the release workflow. A binary built
without `-DGGML_CUDA=ON` exits with a clear message if `--backend cuda` is
requested.

## Build (shared)
```bash
Expand Down
Loading