AudioX: Diffusion Transformer for Anything-to-Audio Generation
Audio and music generation have emerged as crucial tasks in many
applications, yet existing approaches face significant limitations: they
operate in isolation without unified capabilities across modalities, suffer
from scarce high-quality, multi-modal training data, and struggle to
effectively integrate diverse inputs. In this work, we propose AudioX, a
unified Diffusion Transformer model for Anything-to-Audio and Music Generation.
Unlike previous domain-specific models, AudioX can generate both general audio
and music with high quality, while offering flexible natural language control
and seamless processing of various modalities including text, video, image,
music, and audio. Its key innovation is a multi-modal masked training strategy
that masks inputs across modalities and forces the model to learn from masked
inputs, yielding robust and unified cross-modal representations. To address
data scarcity, we curate two comprehensive datasets: vggsound-caps with 190K
audio captions based on the VGGSound dataset, and V2M-caps with 6 million music
captions derived from the V2M dataset. Extensive experiments demonstrate that
AudioX not only matches or outperforms state-of-the-art specialized models, but
also offers remarkable versatility in handling diverse input modalities and
generation tasks within a unified architecture. The code and datasets will be
available at https://zeyuet.github.io/AudioX/
67d6736ca8b6228408aa2a5c
Zeyue Tian
claimed_verified
2025-03-16T21:12:14.592Z
6504409b52ca06fef957266d
/avatars/ac96eab7318cd8d472601cf7e7c50c97.svg
Zeyue
user
Zeyue7
67d6736ca8b6228408aa2a5d
Yizhu Jin
67d6736ca8b6228408aa2a5e
Zhaoyang Liu
67d6736ca8b6228408aa2a5f
Ruibin Yuan
67d6736ca8b6228408aa2a60
Xu Tan
67d6736ca8b6228408aa2a61
Qifeng Chen
67d6736ca8b6228408aa2a62
Wei Xue
67d6736ca8b6228408aa2a63
Yike Guo
AudioX: Diffusion Transformer for Anything-to-Audio Generation
Audio and music generation have emerged as crucial tasks in many
applications, yet existing approaches face significant limitations: they
operate in isolation without unified capabilities across modalities, suffer
from scarce high-quality, multi-modal training data, and struggle to
effectively integrate diverse inputs. In this work, we propose AudioX, a
unified Diffusion Transformer model for Anything-to-Audio and Music Generation.
Unlike previous domain-specific models, AudioX can generate both general audio
and music with high quality, while offering flexible natural language control
and seamless processing of various modalities including text, video, image,
music, and audio. Its key innovation is a multi-modal masked training strategy
that masks inputs across modalities and forces the model to learn from masked
inputs, yielding robust and unified cross-modal representations. To address
data scarcity, we curate two comprehensive datasets: vggsound-caps with 190K
audio captions based on the VGGSound dataset, and V2M-caps with 6 million music
captions derived from the V2M dataset. Extensive experiments demonstrate that
AudioX not only matches or outperforms state-of-the-art specialized models, but
also offers remarkable versatility in handling diverse input modalities and
generation tasks within a unified architecture. The code and datasets will be
available at https://zeyuet.github.io/AudioX/
AudioX, a unified Diffusion Transformer, generates high-quality audio and music with natural language control and supports various modalities through masked multi-modal training.
Diffusion Transformer multi-modal masked training strategy audio captions music captions vggsound-caps V2M-caps natural language control