Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation
In recent years, with the realistic generation results and a wide range of
personalized applications, diffusion-based generative models gain huge
attention in both visual and audio generation areas. Compared to the
considerable advancements of text2image or text2audio generation, research in
audio2visual or visual2audio generation has been relatively slow. The recent
audio-visual generation methods usually resort to huge large language model or
composable diffusion models. Instead of designing another giant model for
audio-visual generation, in this paper we take a step back showing a simple and
lightweight generative transformer, which is not fully investigated in
multi-modal generation, can achieve excellent results on image2audio
generation. The transformer operates in the discrete audio and visual
Vector-Quantized GAN space, and is trained in the mask denoising manner. After
training, the classifier-free guidance could be deployed off-the-shelf
achieving better performance, without any extra training or modification. Since
the transformer model is modality symmetrical, it could also be directly
deployed for audio2image generation and co-generation. In the experiments, we
show that our simple method surpasses recent image2audio generation methods.
Generated audio samples can be found at
https://docs.google.com/presentation/d/1ZtC0SeblKkut4XJcRaDsSTuCRIXB3ypxmSi7HTY3IyQ
66501267780b951aa04ba8b3
Shiqi Yang
claimed_verified
2024-05-24T15:44:36.302Z
6432a3bcfc29acb96be55640
/avatars/a42670a8008ade62edc7f0fe5af4bbaf.svg
Shiqi
user
aquila147
66501267780b951aa04ba8b4
Zhi Zhong
66501267780b951aa04ba8b5
Mengjie Zhao
66501267780b951aa04ba8b6
Shusuke Takahashi
66501267780b951aa04ba8b7
Masato Ishii
admin_assigned
2024-05-24T10:48:16.278Z
63119066bc152fa3e815d3b9
/avatars/6521d2358f526d1895b63f1d0f79446e.svg
masato ishii
user
naotoron
66501267780b951aa04ba8b8
Takashi Shibuya
claimed_verified
2024-06-02T18:47:48.151Z
6650773ca6acfdd2aba7d486
/avatars/d297886ea60dbff98a043caf825820ed.svg
Takashi Shibuya
user
TakashiShibuyaSony
66501267780b951aa04ba8b9
Yuki Mitsufuji
extracted_pending
2024-06-03T21:15:21.551Z
665e32384ecc8a7181634f6d
/avatars/8752f952010540d14f45eac849e91371.svg
Yuki Mitsufuji
user
mittu1204
Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation
In recent years, with the realistic generation results and a wide range of
personalized applications, diffusion-based generative models gain huge
attention in both visual and audio generation areas. Compared to the
considerable advancements of text2image or text2audio generation, research in
audio2visual or visual2audio generation has been relatively slow. The recent
audio-visual generation methods usually resort to huge large language model or
composable diffusion models. Instead of designing another giant model for
audio-visual generation, in this paper we take a step back showing a simple and
lightweight generative transformer, which is not fully investigated in
multi-modal generation, can achieve excellent results on image2audio
generation. The transformer operates in the discrete audio and visual
Vector-Quantized GAN space, and is trained in the mask denoising manner. After
training, the classifier-free guidance could be deployed off-the-shelf
achieving better performance, without any extra training or modification. Since
the transformer model is modality symmetrical, it could also be directly
deployed for audio2image generation and co-generation. In the experiments, we
show that our simple method surpasses recent image2audio generation methods.
Generated audio samples can be found at
https://docs.google.com/presentation/d/1ZtC0SeblKkut4XJcRaDsSTuCRIXB3ypxmSi7HTY3IyQ
A simple generative transformer in Vector-Quantized GAN space achieves excellent results for image2audio and audio2image generation without the need for large models or extensive training.
diffusion-based generative models text2image text2audio audio2visual visual2audio composable diffusion models generative transformer discrete audio Vector-Quantized GAN mask denoising classifier-free guidance multi-modal generation