Summary
Currently, all frame data flows through system memory using Arc<[u8]> byte slices. Frames are copied from GStreamer buffers to CPU memory via buffer.map_readable(). This introduces unnecessary memory copies and CPU overhead, especially for high-resolution or high-framerate video.
Current Implementation
- Frame extraction: GStreamer buffers are mapped to CPU memory and wrapped in
Arc<[u8]>
- No dma-buf support: Zero dma-buf references exist in the codebase
- Memory model: All frames pass through system memory regardless of source or destination
Key locations:
src/backends/camera/pipewire/pipeline.rs - Camera capture pipeline
src/backends/virtual_camera/pipeline.rs - Virtual camera output pipeline
src/backends/camera/types.rs - CameraFrame structure
Proposed Changes
-
Enable dma-buf in GStreamer pipeline
- Use GStreamer dma-buf elements for zero-copy buffer passing
- Expose file descriptors from dma-buf buffers
-
Update frame data structure
- Replace
Arc<[u8]> with dma-buf handles/file descriptors
- Track dma-buf lifecycle (acquire/release)
- Add proper synchronization between camera, GPU, and applications
-
GPU direct access
- Enable direct GPU→dma-buf mapping
- Eliminate CPU readback in filter pipeline where possible
- Keep data on GPU until final output
Benefits
- Reduced memory bandwidth: No unnecessary copies between GPU and CPU
- Lower latency: Direct buffer passing between components
- Lower CPU usage: Less memory management overhead
- Better efficiency: Especially important for 4K/high-framerate capture
Considerations
- Requires proper dma-buf lifecycle management
- Need to handle fallback for systems without dma-buf support
- May require Vulkan/NVIDIA extensions for GPU memory access
Summary
Currently, all frame data flows through system memory using
Arc<[u8]>byte slices. Frames are copied from GStreamer buffers to CPU memory viabuffer.map_readable(). This introduces unnecessary memory copies and CPU overhead, especially for high-resolution or high-framerate video.Current Implementation
Arc<[u8]>Key locations:
src/backends/camera/pipewire/pipeline.rs- Camera capture pipelinesrc/backends/virtual_camera/pipeline.rs- Virtual camera output pipelinesrc/backends/camera/types.rs- CameraFrame structureProposed Changes
Enable dma-buf in GStreamer pipeline
Update frame data structure
Arc<[u8]>with dma-buf handles/file descriptorsGPU direct access
Benefits
Considerations