title: Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation
summary: In recent years, with the realistic generation results and a wide range of
  personalized applications, diffusion-based generative models gain huge
  attention in both visual and audio generation areas. Compared to the
  considerable advancements of text2image or text2audio generation, research in
  audio2visual or visual2audio generation has been relatively slow. The recent
  audio-visual generation methods usually resort to huge large language model or
  composable diffusion models. Instead of designing another giant model for
  audio-visual generation, in this paper we take a step back showing a simple and
  lightweight generative transformer, which is not fully investigated in
  multi-modal generation, can achieve excellent results on image2audio
  generation. The transformer operates in the discrete audio and visual
  Vector-Quantized GAN space, and is trained in the mask denoising manner. After
  training, the classifier-free guidance could be deployed off-the-shelf
  achieving better performance, without any extra training or modification. Since
  the transformer model is modality symmetrical, it could also be directly
  deployed for audio2image generation and co-generation. In the experiments, we
  show that our simple method surpasses recent image2audio generation methods.
  Generated audio samples can be found at
  https://docs.google.com/presentation/d/1ZtC0SeblKkut4XJcRaDsSTuCRIXB3ypxmSi7HTY3IyQ
paper_authors._id: 66501267780b951aa04ba8b3
paper_authors.name: Shiqi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-24T15:44:36.302Z
paper_authors._id: 6432a3bcfc29acb96be55640
paper_authors.avatarUrl: /avatars/a42670a8008ade62edc7f0fe5af4bbaf.svg
paper_authors.fullname: Shiqi
paper_authors.type: user
paper_authors.user: aquila147
paper_authors._id: 66501267780b951aa04ba8b4
paper_authors.name: Zhi Zhong
paper_authors._id: 66501267780b951aa04ba8b5
paper_authors.name: Mengjie Zhao
paper_authors._id: 66501267780b951aa04ba8b6
paper_authors.name: Shusuke Takahashi
paper_authors._id: 66501267780b951aa04ba8b7
paper_authors.name: Masato Ishii
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T10:48:16.278Z
paper_authors._id: 63119066bc152fa3e815d3b9
paper_authors.avatarUrl: /avatars/6521d2358f526d1895b63f1d0f79446e.svg
paper_authors.fullname: masato ishii
paper_authors.type: user
paper_authors.user: naotoron
paper_authors._id: 66501267780b951aa04ba8b8
paper_authors.name: Takashi Shibuya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-02T18:47:48.151Z
paper_authors._id: 6650773ca6acfdd2aba7d486
paper_authors.avatarUrl: /avatars/d297886ea60dbff98a043caf825820ed.svg
paper_authors.fullname: Takashi Shibuya
paper_authors.type: user
paper_authors.user: TakashiShibuyaSony
paper_authors._id: 66501267780b951aa04ba8b9
paper_authors.name: Yuki Mitsufuji
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-03T21:15:21.551Z
paper_authors._id: 665e32384ecc8a7181634f6d
paper_authors.avatarUrl: /avatars/8752f952010540d14f45eac849e91371.svg
paper_authors.fullname: Yuki Mitsufuji
paper_authors.type: user
paper_authors.user: mittu1204
paper_title: Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation
paper_summary: In recent years, with the realistic generation results and a wide range of
  personalized applications, diffusion-based generative models gain huge
  attention in both visual and audio generation areas. Compared to the
  considerable advancements of text2image or text2audio generation, research in
  audio2visual or visual2audio generation has been relatively slow. The recent
  audio-visual generation methods usually resort to huge large language model or
  composable diffusion models. Instead of designing another giant model for
  audio-visual generation, in this paper we take a step back showing a simple and
  lightweight generative transformer, which is not fully investigated in
  multi-modal generation, can achieve excellent results on image2audio
  generation. The transformer operates in the discrete audio and visual
  Vector-Quantized GAN space, and is trained in the mask denoising manner. After
  training, the classifier-free guidance could be deployed off-the-shelf
  achieving better performance, without any extra training or modification. Since
  the transformer model is modality symmetrical, it could also be directly
  deployed for audio2image generation and co-generation. In the experiments, we
  show that our simple method surpasses recent image2audio generation methods.
  Generated audio samples can be found at
  https://docs.google.com/presentation/d/1ZtC0SeblKkut4XJcRaDsSTuCRIXB3ypxmSi7HTY3IyQ
paper_ai_summary: A simple generative transformer in Vector-Quantized GAN space achieves excellent results for image2audio and audio2image generation without the need for large models or extensive training.
paper_ai_keywords: diffusion-based generative models text2image text2audio audio2visual visual2audio composable diffusion models generative transformer discrete audio Vector-Quantized GAN mask denoising classifier-free guidance multi-modal generation