AudioVisual Dataset Processing

This project processes audio-visual datasets by replacing the audio track of raw videos with different types of processed audio (clean, noisy, enhanced, etc.).

Dataset Structure

dataset/
├── SP01/                    # Subject 01
│   ├── audio/
│   │   ├── clean/           # Preprocessed clean audio files
│   │   │   └── SP01_179.wav
│   │   ├── noisy/           # Noisy audio files
│   │   │   └── SP01_179.wav
│   │   ├── enh_mamba/       # Mamba enhanced audio files
│   │   └── enh_mmse/        # MMSE enhanced audio files
│   └── video/
│       ├── raw/             # Raw video files with original clean audio
│       │   └── SP01_179.mp4
│       ├── clean/           # Videos with replaced clean audio
│       ├── noisy/           # Videos with replaced noisy audio 
│       ├── enh_mamba/       # Videos with replaced enhanced audio
│       └── enh_mmse/        # Videos with replaced enhanced audio
└── SP04/                    # Subject 04
    ├── audio/
    │   ├── clean/
    │   │   └── SP04_169.wav
    │   ├── noisy/
    │   │   └── SP04_169.wav
    │   ├── enh_mamba/
    │   └── enh_mmse/
    └── video/
        ├── raw/
        │   └── SP04_169.mp4
        ├── clean/
        ├── noisy/
        ├── enh_mamba/
        └── enh_mmse/

Features

The program automatically:

1. Scans all video files in dataset/SP*/video/raw/
2. Checks if corresponding audio files exist in audio/clean/, audio/noisy/, audio/enh_mamba/, audio/enh_mmse/ folders
3. If audio files exist, replaces the raw video's audio track with the corresponding processed audio
4. Saves the synthesized videos in the corresponding folders under video/

Usage

Install Dependencies

pip install moviepy

Run the Program

python main.py

Notes

• The program automatically skips non-existent audio files
• If output videos already exist, processing is skipped to avoid duplication
• Supported audio format: .wav
• Supported video format: .mp4
• Audio length is automatically adjusted to match video length
• Raw videos contain the original clean audio track, while processed audio files are stored separately