title: AudioX: Diffusion Transformer for Anything-to-Audio Generation
summary: Audio and music generation have emerged as crucial tasks in many
  applications, yet existing approaches face significant limitations: they
  operate in isolation without unified capabilities across modalities, suffer
  from scarce high-quality, multi-modal training data, and struggle to
  effectively integrate diverse inputs. In this work, we propose AudioX, a
  unified Diffusion Transformer model for Anything-to-Audio and Music Generation.
  Unlike previous domain-specific models, AudioX can generate both general audio
  and music with high quality, while offering flexible natural language control
  and seamless processing of various modalities including text, video, image,
  music, and audio. Its key innovation is a multi-modal masked training strategy
  that masks inputs across modalities and forces the model to learn from masked
  inputs, yielding robust and unified cross-modal representations. To address
  data scarcity, we curate two comprehensive datasets: vggsound-caps with 190K
  audio captions based on the VGGSound dataset, and V2M-caps with 6 million music
  captions derived from the V2M dataset. Extensive experiments demonstrate that
  AudioX not only matches or outperforms state-of-the-art specialized models, but
  also offers remarkable versatility in handling diverse input modalities and
  generation tasks within a unified architecture. The code and datasets will be
  available at https://zeyuet.github.io/AudioX/
paper_authors._id: 67d6736ca8b6228408aa2a5c
paper_authors.name: Zeyue Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:12:14.592Z
paper_authors._id: 6504409b52ca06fef957266d
paper_authors.avatarUrl: /avatars/ac96eab7318cd8d472601cf7e7c50c97.svg
paper_authors.fullname: Zeyue
paper_authors.type: user
paper_authors.user: Zeyue7
paper_authors._id: 67d6736ca8b6228408aa2a5d
paper_authors.name: Yizhu Jin
paper_authors._id: 67d6736ca8b6228408aa2a5e
paper_authors.name: Zhaoyang Liu
paper_authors._id: 67d6736ca8b6228408aa2a5f
paper_authors.name: Ruibin Yuan
paper_authors._id: 67d6736ca8b6228408aa2a60
paper_authors.name: Xu Tan
paper_authors._id: 67d6736ca8b6228408aa2a61
paper_authors.name: Qifeng Chen
paper_authors._id: 67d6736ca8b6228408aa2a62
paper_authors.name: Wei Xue
paper_authors._id: 67d6736ca8b6228408aa2a63
paper_authors.name: Yike Guo
paper_title: AudioX: Diffusion Transformer for Anything-to-Audio Generation
paper_summary: Audio and music generation have emerged as crucial tasks in many
  applications, yet existing approaches face significant limitations: they
  operate in isolation without unified capabilities across modalities, suffer
  from scarce high-quality, multi-modal training data, and struggle to
  effectively integrate diverse inputs. In this work, we propose AudioX, a
  unified Diffusion Transformer model for Anything-to-Audio and Music Generation.
  Unlike previous domain-specific models, AudioX can generate both general audio
  and music with high quality, while offering flexible natural language control
  and seamless processing of various modalities including text, video, image,
  music, and audio. Its key innovation is a multi-modal masked training strategy
  that masks inputs across modalities and forces the model to learn from masked
  inputs, yielding robust and unified cross-modal representations. To address
  data scarcity, we curate two comprehensive datasets: vggsound-caps with 190K
  audio captions based on the VGGSound dataset, and V2M-caps with 6 million music
  captions derived from the V2M dataset. Extensive experiments demonstrate that
  AudioX not only matches or outperforms state-of-the-art specialized models, but
  also offers remarkable versatility in handling diverse input modalities and
  generation tasks within a unified architecture. The code and datasets will be
  available at https://zeyuet.github.io/AudioX/
paper_ai_summary: AudioX, a unified Diffusion Transformer, generates high-quality audio and music with natural language control and supports various modalities through masked multi-modal training.
paper_ai_keywords: Diffusion Transformer multi-modal masked training strategy audio captions music captions vggsound-caps V2M-caps natural language control