title: AudioToken: Adaptation of Text-Conditioned Diffusion Models for
    Audio-to-Image Generation
summary: In recent years, image generation has shown a great leap in performance,
  where diffusion models play a central role. Although generating high-quality
  images, such models are mainly conditioned on textual descriptions. This begs
  the question: "how can we adopt such models to be conditioned on other
  modalities?". In this paper, we propose a novel method utilizing latent
  diffusion models trained for text-to-image-generation to generate images
  conditioned on audio recordings. Using a pre-trained audio encoding model, the
  proposed method encodes audio into a new token, which can be considered as an
  adaptation layer between the audio and text representations. Such a modeling
  paradigm requires a small number of trainable parameters, making the proposed
  approach appealing for lightweight optimization. Results suggest the proposed
  method is superior to the evaluated baseline methods, considering objective and
  subjective metrics. Code and samples are available at:
  https://pages.cs.huji.ac.il/adiyoss-lab/AudioToken.
paper_authors._id: 646c12883e2a7b06594ad119
paper_authors.name: Guy Yariv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-24T15:30:50.778Z
paper_authors._id: 646d239f4220471ca0c6471c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d239f4220471ca0c6471c/sRwzko8XEUVCkeD7jXceH.jpeg
paper_authors.fullname: Guy Yariv
paper_authors.type: user
paper_authors.user: GuyYariv
paper_authors._id: 646c12883e2a7b06594ad11a
paper_authors.name: Itai Gat
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-23T08:11:32.894Z
paper_authors._id: 62a8fa984d933c74bf410c16
paper_authors.avatarUrl: /avatars/73519deba3176be9c23d49f749aee5da.svg
paper_authors.fullname: Itai Gat
paper_authors.type: user
paper_authors.user: itaigat
paper_authors._id: 646c12883e2a7b06594ad11b
paper_authors.name: Lior Wolf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-23T08:43:14.124Z
paper_authors._id: 62f9f170363251ee40a08046
paper_authors.avatarUrl: /avatars/67ad459db44b605be0c1f6bd33363d6e.svg
paper_authors.fullname: Lior Wolf
paper_authors.type: user
paper_authors.user: liorwolf
paper_authors._id: 646c12883e2a7b06594ad11c
paper_authors.name: Yossi Adi
paper_authors._id: 646c12883e2a7b06594ad11d
paper_authors.name: Idan Schwartz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-23T12:22:52.549Z
paper_authors._id: 617fb19badaa26f5e57e97cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1645002335184-617fb19badaa26f5e57e97cd.png
paper_authors.fullname: Schwartz
paper_authors.type: user
paper_authors.user: Idan
paper_title: AudioToken: Adaptation of Text-Conditioned Diffusion Models for
    Audio-to-Image Generation
paper_summary: In recent years, image generation has shown a great leap in performance,
  where diffusion models play a central role. Although generating high-quality
  images, such models are mainly conditioned on textual descriptions. This begs
  the question: "how can we adopt such models to be conditioned on other
  modalities?". In this paper, we propose a novel method utilizing latent
  diffusion models trained for text-to-image-generation to generate images
  conditioned on audio recordings. Using a pre-trained audio encoding model, the
  proposed method encodes audio into a new token, which can be considered as an
  adaptation layer between the audio and text representations. Such a modeling
  paradigm requires a small number of trainable parameters, making the proposed
  approach appealing for lightweight optimization. Results suggest the proposed
  method is superior to the evaluated baseline methods, considering objective and
  subjective metrics. Code and samples are available at:
  https://pages.cs.huji.ac.il/adiyoss-lab/AudioToken.
paper_ai_keywords: latent diffusion models text-to-image-generation audio recordings audio encoding model token adaptation layer trainable parameters lightweight optimization baseline methods