Self-Supervised Audio-Visual Soundscape Stylization
Speech sounds convey a great deal of information about the scenes, resulting
in a variety of effects ranging from reverberation to additional ambient
sounds. In this paper, we manipulate input speech to sound as though it was
recorded within a different scene, given an audio-visual conditional example
recorded from that scene. Our model learns through self-supervision, taking
advantage of the fact that natural video contains recurring sound events and
textures. We extract an audio clip from a video and apply speech enhancement.
We then train a latent diffusion model to recover the original speech, using
another audio-visual clip taken from elsewhere in the video as a conditional
hint. Through this process, the model learns to transfer the conditional
example's sound properties to the input speech. We show that our model can be
successfully trained using unlabeled, in-the-wild videos, and that an
additional visual signal can improve its sound prediction abilities. Please see
our project webpage for video results:
https://tinglok.netlify.app/files/avsoundscape/
66f23451df676bdfd954bed3
Tingle Li
66f23451df676bdfd954bed4
Renhao Wang
66f23451df676bdfd954bed5
Po-Yao Huang
66f23451df676bdfd954bed6
Andrew Owens
66f23451df676bdfd954bed7
Gopala Anumanchipalli
Self-Supervised Audio-Visual Soundscape Stylization
Speech sounds convey a great deal of information about the scenes, resulting
in a variety of effects ranging from reverberation to additional ambient
sounds. In this paper, we manipulate input speech to sound as though it was
recorded within a different scene, given an audio-visual conditional example
recorded from that scene. Our model learns through self-supervision, taking
advantage of the fact that natural video contains recurring sound events and
textures. We extract an audio clip from a video and apply speech enhancement.
We then train a latent diffusion model to recover the original speech, using
another audio-visual clip taken from elsewhere in the video as a conditional
hint. Through this process, the model learns to transfer the conditional
example's sound properties to the input speech. We show that our model can be
successfully trained using unlabeled, in-the-wild videos, and that an
additional visual signal can improve its sound prediction abilities. Please see
our project webpage for video results:
https://tinglok.netlify.app/files/avsoundscape/
A latent diffusion model is trained via self-supervision to enhance and condition speech to sound as though it originated from a different audio-visual scene.
speech enhancement latent diffusion model self-supervision sound properties audio-visual conditional hint