title: CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained
    Language-Vision Models
summary: Recent work has studied text-to-audio synthesis using large amounts of paired
  text-audio data. However, audio recordings with high-quality text annotations
  can be difficult to acquire. In this work, we approach text-to-audio synthesis
  using unlabeled videos and pretrained language-vision models. We propose to
  learn the desired text-audio correspondence by leveraging the visual modality
  as a bridge. We train a conditional diffusion model to generate the audio track
  of a video, given a video frame encoded by a pretrained contrastive
  language-image pretraining (CLIP) model. At test time, we first explore
  performing a zero-shot modality transfer and condition the diffusion model with
  a CLIP-encoded text query. However, we observe a noticeable performance drop
  with respect to image queries. To close this gap, we further adopt a pretrained
  diffusion prior model to generate a CLIP image embedding given a CLIP text
  embedding. Our results show the effectiveness of the proposed method, and that
  the pretrained diffusion prior can reduce the modality transfer gap. While we
  focus on text-to-audio synthesis, the proposed model can also generate audio
  from image queries, and it shows competitive performance against a
  state-of-the-art image-to-audio synthesis model in a subjective listening test.
  This study offers a new direction of approaching text-to-audio synthesis that
  leverages the naturally-occurring audio-visual correspondence in videos and the
  power of pretrained language-vision models.
paper_authors._id: 648fd142dd775153e51f0285
paper_authors.name: Hao-Wen Dong
paper_authors._id: 648fd142dd775153e51f0286
paper_authors.name: Xiaoyu Liu
paper_authors._id: 648fd142dd775153e51f0287
paper_authors.name: Jordi Pons
paper_authors._id: 648fd142dd775153e51f0288
paper_authors.name: Gautam Bhattacharya
paper_authors._id: 648fd142dd775153e51f0289
paper_authors.name: Santiago Pascual
paper_authors._id: 648fd142dd775153e51f028a
paper_authors.name: Joan Serrà
paper_authors._id: 648fd142dd775153e51f028b
paper_authors.name: Taylor Berg-Kirkpatrick
paper_authors._id: 648fd142dd775153e51f028c
paper_authors.name: Julian McAuley
paper_title: CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained
    Language-Vision Models
paper_summary: Recent work has studied text-to-audio synthesis using large amounts of paired
  text-audio data. However, audio recordings with high-quality text annotations
  can be difficult to acquire. In this work, we approach text-to-audio synthesis
  using unlabeled videos and pretrained language-vision models. We propose to
  learn the desired text-audio correspondence by leveraging the visual modality
  as a bridge. We train a conditional diffusion model to generate the audio track
  of a video, given a video frame encoded by a pretrained contrastive
  language-image pretraining (CLIP) model. At test time, we first explore
  performing a zero-shot modality transfer and condition the diffusion model with
  a CLIP-encoded text query. However, we observe a noticeable performance drop
  with respect to image queries. To close this gap, we further adopt a pretrained
  diffusion prior model to generate a CLIP image embedding given a CLIP text
  embedding. Our results show the effectiveness of the proposed method, and that
  the pretrained diffusion prior can reduce the modality transfer gap. While we
  focus on text-to-audio synthesis, the proposed model can also generate audio
  from image queries, and it shows competitive performance against a
  state-of-the-art image-to-audio synthesis model in a subjective listening test.
  This study offers a new direction of approaching text-to-audio synthesis that
  leverages the naturally-occurring audio-visual correspondence in videos and the
  power of pretrained language-vision models.
paper_ai_keywords: conditional diffusion model CLIP model CLIP-encoded text query CLIP image embedding pretrained diffusion prior model zero-shot modality transfer image-to-audio synthesis