title: AV-GS: Learning Material and Geometry Aware Priors for Novel View
    Acoustic Synthesis
summary: Novel view acoustic synthesis (NVAS) aims to render binaural audio at any
  target viewpoint, given a mono audio emitted by a sound source at a 3D scene.
  Existing methods have proposed NeRF-based implicit models to exploit visual
  cues as a condition for synthesizing binaural audio. However, in addition to
  low efficiency originating from heavy NeRF rendering, these methods all have a
  limited ability of characterizing the entire scene environment such as room
  geometry, material properties, and the spatial relation between the listener
  and sound source. To address these issues, we propose a novel Audio-Visual
  Gaussian Splatting (AV-GS) model. To obtain a material-aware and geometry-aware
  condition for audio synthesis, we learn an explicit point-based scene
  representation with an audio-guidance parameter on locally initialized Gaussian
  points, taking into account the space relation from the listener and sound
  source. To make the visual scene model audio adaptive, we propose a point
  densification and pruning strategy to optimally distribute the Gaussian points,
  with the per-point contribution in sound propagation (e.g., more points needed
  for texture-less wall surfaces as they affect sound path diversion). Extensive
  experiments validate the superiority of our AV-GS over existing alternatives on
  the real-world RWAS and simulation-based SoundSpaces datasets.
paper_authors._id: 66705dfbb0cc5418431113a8
paper_authors.name: Swapnil Bhosale
paper_authors._id: 66705dfbb0cc5418431113a9
paper_authors.name: Haosen Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-06T11:15:16.552Z
paper_authors._id: 64c281e30344289c376c0fa4
paper_authors.avatarUrl: /avatars/dae01f7054da8c9517e3808a6f10eb1a.svg
paper_authors.fullname: HY
paper_authors.type: user
paper_authors.user: happy0612
paper_authors._id: 66705dfbb0cc5418431113aa
paper_authors.name: Diptesh Kanojia
paper_authors._id: 66705dfbb0cc5418431113ab
paper_authors.name: Jiankang Deng
paper_authors._id: 66705dfbb0cc5418431113ac
paper_authors.name: Xiatian Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T06:57:42.267Z
paper_authors._id: 647ef9e7aa8c04bbf9362a8a
paper_authors.avatarUrl: /avatars/dc23e15acf7ee7f3b9f1a68c2716a66d.svg
paper_authors.fullname: Xiatian Zhu
paper_authors.type: user
paper_authors.user: Xiatian-Zhu
paper_title: AV-GS: Learning Material and Geometry Aware Priors for Novel View
    Acoustic Synthesis
paper_summary: Novel view acoustic synthesis (NVAS) aims to render binaural audio at any
  target viewpoint, given a mono audio emitted by a sound source at a 3D scene.
  Existing methods have proposed NeRF-based implicit models to exploit visual
  cues as a condition for synthesizing binaural audio. However, in addition to
  low efficiency originating from heavy NeRF rendering, these methods all have a
  limited ability of characterizing the entire scene environment such as room
  geometry, material properties, and the spatial relation between the listener
  and sound source. To address these issues, we propose a novel Audio-Visual
  Gaussian Splatting (AV-GS) model. To obtain a material-aware and geometry-aware
  condition for audio synthesis, we learn an explicit point-based scene
  representation with an audio-guidance parameter on locally initialized Gaussian
  points, taking into account the space relation from the listener and sound
  source. To make the visual scene model audio adaptive, we propose a point
  densification and pruning strategy to optimally distribute the Gaussian points,
  with the per-point contribution in sound propagation (e.g., more points needed
  for texture-less wall surfaces as they affect sound path diversion). Extensive
  experiments validate the superiority of our AV-GS over existing alternatives on
  the real-world RWAS and simulation-based SoundSpaces datasets.
paper_ai_summary: A novel Audio-Visual Gaussian Splatting (AV-GS) model synthesizes binaural audio by learning an explicit point-based scene representation with audio guidance, improving upon existing NeRF-based methods.
paper_ai_keywords: NeRF-based implicit models binaural audio 3D scene material properties room geometry spatial relation explicit point-based representation audio-guidance parameter point densification pruning strategy sound propagation Gaussian points RWAS dataset SoundSpaces dataset