AV-GS: Learning Material and Geometry Aware Priors for Novel View
  Acoustic Synthesis
Novel view acoustic synthesis (NVAS) aims to render binaural audio at any
target viewpoint, given a mono audio emitted by a sound source at a 3D scene.
Existing methods have proposed NeRF-based implicit models to exploit visual
cues as a condition for synthesizing binaural audio. However, in addition to
low efficiency originating from heavy NeRF rendering, these methods all have a
limited ability of characterizing the entire scene environment such as room
geometry, material properties, and the spatial relation between the listener
and sound source. To address these issues, we propose a novel Audio-Visual
Gaussian Splatting (AV-GS) model. To obtain a material-aware and geometry-aware
condition for audio synthesis, we learn an explicit point-based scene
representation with an audio-guidance parameter on locally initialized Gaussian
points, taking into account the space relation from the listener and sound
source. To make the visual scene model audio adaptive, we propose a point
densification and pruning strategy to optimally distribute the Gaussian points,
with the per-point contribution in sound propagation (e.g., more points needed
for texture-less wall surfaces as they affect sound path diversion). Extensive
experiments validate the superiority of our AV-GS over existing alternatives on
the real-world RWAS and simulation-based SoundSpaces datasets.
66705dfbb0cc5418431113a8
Swapnil Bhosale
66705dfbb0cc5418431113a9
Haosen Yang
claimed_verified
2024-09-06T11:15:16.552Z
64c281e30344289c376c0fa4
/avatars/dae01f7054da8c9517e3808a6f10eb1a.svg
HY
user
happy0612
66705dfbb0cc5418431113aa
Diptesh Kanojia
66705dfbb0cc5418431113ab
Jiankang Deng
66705dfbb0cc5418431113ac
Xiatian Zhu
claimed_verified
2024-09-09T06:57:42.267Z
647ef9e7aa8c04bbf9362a8a
/avatars/dc23e15acf7ee7f3b9f1a68c2716a66d.svg
Xiatian Zhu
user
Xiatian-Zhu
AV-GS: Learning Material and Geometry Aware Priors for Novel View
  Acoustic Synthesis
Novel view acoustic synthesis (NVAS) aims to render binaural audio at any
target viewpoint, given a mono audio emitted by a sound source at a 3D scene.
Existing methods have proposed NeRF-based implicit models to exploit visual
cues as a condition for synthesizing binaural audio. However, in addition to
low efficiency originating from heavy NeRF rendering, these methods all have a
limited ability of characterizing the entire scene environment such as room
geometry, material properties, and the spatial relation between the listener
and sound source. To address these issues, we propose a novel Audio-Visual
Gaussian Splatting (AV-GS) model. To obtain a material-aware and geometry-aware
condition for audio synthesis, we learn an explicit point-based scene
representation with an audio-guidance parameter on locally initialized Gaussian
points, taking into account the space relation from the listener and sound
source. To make the visual scene model audio adaptive, we propose a point
densification and pruning strategy to optimally distribute the Gaussian points,
with the per-point contribution in sound propagation (e.g., more points needed
for texture-less wall surfaces as they affect sound path diversion). Extensive
experiments validate the superiority of our AV-GS over existing alternatives on
the real-world RWAS and simulation-based SoundSpaces datasets.
A novel Audio-Visual Gaussian Splatting (AV-GS) model synthesizes binaural audio by learning an explicit point-based scene representation with audio guidance, improving upon existing NeRF-based methods.
NeRF-based implicit models binaural audio 3D scene material properties room geometry spatial relation explicit point-based representation audio-guidance parameter point densification pruning strategy sound propagation Gaussian points RWAS dataset SoundSpaces dataset