CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained
  Language-Vision Models
Recent work has studied text-to-audio synthesis using large amounts of paired
text-audio data. However, audio recordings with high-quality text annotations
can be difficult to acquire. In this work, we approach text-to-audio synthesis
using unlabeled videos and pretrained language-vision models. We propose to
learn the desired text-audio correspondence by leveraging the visual modality
as a bridge. We train a conditional diffusion model to generate the audio track
of a video, given a video frame encoded by a pretrained contrastive
language-image pretraining (CLIP) model. At test time, we first explore
performing a zero-shot modality transfer and condition the diffusion model with
a CLIP-encoded text query. However, we observe a noticeable performance drop
with respect to image queries. To close this gap, we further adopt a pretrained
diffusion prior model to generate a CLIP image embedding given a CLIP text
embedding. Our results show the effectiveness of the proposed method, and that
the pretrained diffusion prior can reduce the modality transfer gap. While we
focus on text-to-audio synthesis, the proposed model can also generate audio
from image queries, and it shows competitive performance against a
state-of-the-art image-to-audio synthesis model in a subjective listening test.
This study offers a new direction of approaching text-to-audio synthesis that
leverages the naturally-occurring audio-visual correspondence in videos and the
power of pretrained language-vision models.
648fd142dd775153e51f0285
Hao-Wen Dong
648fd142dd775153e51f0286
Xiaoyu Liu
648fd142dd775153e51f0287
Jordi Pons
648fd142dd775153e51f0288
Gautam Bhattacharya
648fd142dd775153e51f0289
Santiago Pascual
648fd142dd775153e51f028a
Joan Serrà
648fd142dd775153e51f028b
Taylor Berg-Kirkpatrick
648fd142dd775153e51f028c
Julian McAuley
CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained
  Language-Vision Models
Recent work has studied text-to-audio synthesis using large amounts of paired
text-audio data. However, audio recordings with high-quality text annotations
can be difficult to acquire. In this work, we approach text-to-audio synthesis
using unlabeled videos and pretrained language-vision models. We propose to
learn the desired text-audio correspondence by leveraging the visual modality
as a bridge. We train a conditional diffusion model to generate the audio track
of a video, given a video frame encoded by a pretrained contrastive
language-image pretraining (CLIP) model. At test time, we first explore
performing a zero-shot modality transfer and condition the diffusion model with
a CLIP-encoded text query. However, we observe a noticeable performance drop
with respect to image queries. To close this gap, we further adopt a pretrained
diffusion prior model to generate a CLIP image embedding given a CLIP text
embedding. Our results show the effectiveness of the proposed method, and that
the pretrained diffusion prior can reduce the modality transfer gap. While we
focus on text-to-audio synthesis, the proposed model can also generate audio
from image queries, and it shows competitive performance against a
state-of-the-art image-to-audio synthesis model in a subjective listening test.
This study offers a new direction of approaching text-to-audio synthesis that
leverages the naturally-occurring audio-visual correspondence in videos and the
power of pretrained language-vision models.
A conditional diffusion model leverages pretrained CLIP and diffusion prior models to generate audio from text queries using unlabeled videos, offering competitive performance in text-to-audio and image-to-audio synthesis.
conditional diffusion model pretrained contrastive language-image pretraining (CLIP) CLIP image embedding CLIP text embedding pretrained diffusion prior model text-to-audio synthesis image-to-audio synthesis subject listening test audio-visual correspondence