title: Self-Supervised Audio-Visual Soundscape Stylization
summary: Speech sounds convey a great deal of information about the scenes, resulting
  in a variety of effects ranging from reverberation to additional ambient
  sounds. In this paper, we manipulate input speech to sound as though it was
  recorded within a different scene, given an audio-visual conditional example
  recorded from that scene. Our model learns through self-supervision, taking
  advantage of the fact that natural video contains recurring sound events and
  textures. We extract an audio clip from a video and apply speech enhancement.
  We then train a latent diffusion model to recover the original speech, using
  another audio-visual clip taken from elsewhere in the video as a conditional
  hint. Through this process, the model learns to transfer the conditional
  example's sound properties to the input speech. We show that our model can be
  successfully trained using unlabeled, in-the-wild videos, and that an
  additional visual signal can improve its sound prediction abilities. Please see
  our project webpage for video results:
  https://tinglok.netlify.app/files/avsoundscape/
paper_authors._id: 66f23451df676bdfd954bed3
paper_authors.name: Tingle Li
paper_authors._id: 66f23451df676bdfd954bed4
paper_authors.name: Renhao Wang
paper_authors._id: 66f23451df676bdfd954bed5
paper_authors.name: Po-Yao Huang
paper_authors._id: 66f23451df676bdfd954bed6
paper_authors.name: Andrew Owens
paper_authors._id: 66f23451df676bdfd954bed7
paper_authors.name: Gopala Anumanchipalli
paper_title: Self-Supervised Audio-Visual Soundscape Stylization
paper_summary: Speech sounds convey a great deal of information about the scenes, resulting
  in a variety of effects ranging from reverberation to additional ambient
  sounds. In this paper, we manipulate input speech to sound as though it was
  recorded within a different scene, given an audio-visual conditional example
  recorded from that scene. Our model learns through self-supervision, taking
  advantage of the fact that natural video contains recurring sound events and
  textures. We extract an audio clip from a video and apply speech enhancement.
  We then train a latent diffusion model to recover the original speech, using
  another audio-visual clip taken from elsewhere in the video as a conditional
  hint. Through this process, the model learns to transfer the conditional
  example's sound properties to the input speech. We show that our model can be
  successfully trained using unlabeled, in-the-wild videos, and that an
  additional visual signal can improve its sound prediction abilities. Please see
  our project webpage for video results:
  https://tinglok.netlify.app/files/avsoundscape/
paper_ai_keywords: latent diffusion model audio-visual conditional example self-supervision recurring sounds speech enhancement audio clip conditional hint sound properties in-the-wild videos sound prediction abilities