Skip to content

let ai hear and talk through the game#149

Merged
xingjianll merged 1 commit into
mainfrom
feat/audio
Apr 6, 2026
Merged

let ai hear and talk through the game#149
xingjianll merged 1 commit into
mainfrom
feat/audio

Conversation

@Accusamus2065

Copy link
Copy Markdown
Contributor

Summary

This PR adds game-audio routing support so OpenNeuro can hear in-game voice chat and speak back to players instead of being limited to the local microphone and speaker.

What was added:

  • AudioIn: same port interface as Mic, but with a dynamic input-device dropdown
  • AudioOut: same port interface as Speaker, but with a dynamic output-device dropdown
  • RemoteAudioIn: same role as AudioIn, but receives audio from a remote WebSocket bridge
  • RemoteAudioOut: same role as AudioOut, but sends audio to a remote WebSocket bridge
  • src.lib.audio.remote_audio_server: FastAPI WebSocket bridge that runs on the game machine
  • User docs for VB-Cable A+B and VRChat setup

Why

Before this change, OpenNeuro could only:

  • read from the local mic
  • write to the local speaker

That meant it could not:

  • hear VRChat voice chat
  • speak into the game's microphone path

This PR adds the missing local and remote routing pieces.

Local Same-Machine Setup

Use this when OpenNeuro and VRChat run on the same PC.

Recommended graph:
AudioIn -> VAD -> ASR -> LLM -> TTS -> AudioOut

OpenNeuro component settings:

  1. Replace Mic with AudioIn.
  2. Set AudioIn.device to CABLE-B Output.
  3. Set AudioIn.sample_rate to 48000.
  4. Set AudioIn.channels to 2.
  5. Set AudioIn.frame_ms to 20.
  6. Replace Speaker with AudioOut.
  7. Set AudioOut.device to CABLE-A Input.
  8. Set AudioOut.sample_rate to 48000.
  9. Set AudioOut.channels to 1.

VRChat / Windows settings:

  1. Install VB-Cable A+B.
  2. In VRChat, set Microphone to CABLE-A Output.
  3. VRChat does not expose a separate output-device selector here, so game output follows the Windows system playback device.
  4. In Windows, set the default playback device to CABLE-B Input.
  5. If you still want to hear the game locally, enable Listen to this device on CABLE-B Output and route it to your real headset/speakers.

Signal mapping:

  • AI speech path: AudioOut -> CABLE-A Input -> CABLE-A Output -> VRChat microphone
  • Game audio path: Windows default output / VRChat output -> CABLE-B Input -> CABLE-B Output -> AudioIn

Remote Two-Machine Setup

Use this when VRChat runs on one machine and OpenNeuro runs on another.

Game machine

  1. Install VB-Cable A+B.
  2. In VRChat, set Microphone to CABLE-A Output.
  3. In Windows, set the default playback device to CABLE-B Input.
  4. Start the bridge server:
cd backend
uv run python -m src.lib.audio.remote_audio_server

@github-actions

github-actions Bot commented Apr 5, 2026

Copy link
Copy Markdown

Test Coverage

Target Coverage Covered Lines
Backend 94.20% 10254 / 10885
Frontend 89.02% 1467 / 1648

@xingjianll xingjianll merged commit 41e4815 into main Apr 6, 2026
2 checks passed
@Accusamus2065 Accusamus2065 linked an issue Apr 6, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let AI Hear and Talk through the Game

2 participants