title: UniAudio: An Audio Foundation Model Toward Universal Audio Generation
summary: Language models (LMs) have demonstrated the capability to handle a variety of
  generative tasks. This paper presents the UniAudio system, which, unlike prior
  task-specific approaches, leverages LMs techniques to generate multiple types
  of audio (including speech, sounds, music, and singing) with given input
  conditions. UniAudio 1) first tokenizes all types of target audio along with
  other condition modalities, 2) concatenates source-target pair as a single
  sequence, and 3) performs next-token prediction using LMs. Also, a multi-scale
  Transformer model is proposed to handle the overly long sequences caused by the
  residual vector quantization based neural codec in tokenization. Training of
  UniAudio is scaled up to 165K hours of audio and 1B parameters, based on all
  generative tasks, aiming to obtain sufficient prior knowledge not only in the
  intrinsic properties of audio but also the inter-relationship between audio and
  other modalities. Therefore, the trained UniAudio model has the potential to
  become a foundation model for universal audio generation: it shows strong
  capability in all trained tasks and can seamlessly support new audio generation
  tasks after simple fine-tuning. Experiments demonstrate that UniAudio achieves
  state-of-the-art or at least competitive results on most of the 11 tasks. Demo
  and code are released at https://github.com/yangdongchao/UniAudio
paper_authors._id: 651ba882e315e8a224a0a530
paper_authors.name: Dongchao Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-04T07:37:38.046Z
paper_authors._id: 63c7636b656e7822e23e6f6b
paper_authors.avatarUrl: /avatars/41bfd5e1ce6daab6058eacfd33c7a268.svg
paper_authors.fullname: Dongchao Yang
paper_authors.type: user
paper_authors.user: Dongchao
paper_authors._id: 651ba882e315e8a224a0a531
paper_authors.name: Jinchuan Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T09:02:25.455Z
paper_authors._id: 64770ff6d7cf39f2e937b882
paper_authors.avatarUrl: /avatars/7b729501856f5073cf9e484811012f2a.svg
paper_authors.fullname: Jinchuan Tian
paper_authors.type: user
paper_authors.user: JinchuanTian
paper_authors._id: 651ba882e315e8a224a0a532
paper_authors.name: Xu Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T09:02:40.924Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 651ba882e315e8a224a0a533
paper_authors.name: Rongjie Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T09:03:01.114Z
paper_authors._id: 62f6556e04e5e02f82ac8531
paper_authors.avatarUrl: /avatars/01024815aeaaf8c4721e937b3db68903.svg
paper_authors.fullname: Rongjiehuang
paper_authors.type: user
paper_authors.user: Rongjiehuang
paper_authors._id: 651ba882e315e8a224a0a534
paper_authors.name: Songxiang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T09:03:13.379Z
paper_authors._id: 626b4350822e3d85324c6054
paper_authors.avatarUrl: /avatars/3e27115ea73f33ba40e8599cdf2725af.svg
paper_authors.fullname: liusongxiang
paper_authors.type: user
paper_authors.user: liusongxiang
paper_authors._id: 651ba882e315e8a224a0a535
paper_authors.name: Xuankai Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T09:03:21.023Z
paper_authors._id: 6136b5707c6d9b8e41469bec
paper_authors.avatarUrl: /avatars/5b7f4ee0ba36bb6b87accda0ee27bb17.svg
paper_authors.fullname: Xuankai Chang
paper_authors.type: user
paper_authors.user: simpleoier
paper_authors._id: 651ba882e315e8a224a0a536
paper_authors.name: Jiatong Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T09:03:29.339Z
paper_authors._id: 6136ec7a40e43b8f748a0832
paper_authors.avatarUrl: /avatars/089afa82ecfe6432d2dbfe81048c6ffa.svg
paper_authors.fullname: Jiatong Shi
paper_authors.type: user
paper_authors.user: ftshijt
paper_authors._id: 651ba882e315e8a224a0a537
paper_authors.name: Sheng Zhao
paper_authors._id: 651ba882e315e8a224a0a538
paper_authors.name: Jiang Bian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T09:04:12.094Z
paper_authors._id: 63f253f8f4e30ffd2bd308fb
paper_authors.avatarUrl: /avatars/303f4c7ee588f638acf78a7966786e1e.svg
paper_authors.fullname: Jiang Bian
paper_authors.type: user
paper_authors.user: bianjiang
paper_authors._id: 651ba882e315e8a224a0a539
paper_authors.name: Xixin Wu
paper_authors._id: 651ba882e315e8a224a0a53a
paper_authors.name: Zhou Zhao
paper_authors._id: 651ba882e315e8a224a0a53b
paper_authors.name: Helen Meng
paper_title: UniAudio: An Audio Foundation Model Toward Universal Audio Generation
paper_summary: Language models (LMs) have demonstrated the capability to handle a variety of
  generative tasks. This paper presents the UniAudio system, which, unlike prior
  task-specific approaches, leverages LMs techniques to generate multiple types
  of audio (including speech, sounds, music, and singing) with given input
  conditions. UniAudio 1) first tokenizes all types of target audio along with
  other condition modalities, 2) concatenates source-target pair as a single
  sequence, and 3) performs next-token prediction using LMs. Also, a multi-scale
  Transformer model is proposed to handle the overly long sequences caused by the
  residual vector quantization based neural codec in tokenization. Training of
  UniAudio is scaled up to 165K hours of audio and 1B parameters, based on all
  generative tasks, aiming to obtain sufficient prior knowledge not only in the
  intrinsic properties of audio but also the inter-relationship between audio and
  other modalities. Therefore, the trained UniAudio model has the potential to
  become a foundation model for universal audio generation: it shows strong
  capability in all trained tasks and can seamlessly support new audio generation
  tasks after simple fine-tuning. Experiments demonstrate that UniAudio achieves
  state-of-the-art or at least competitive results on most of the 11 tasks. Demo
  and code are released at https://github.com/yangdongchao/UniAudio
paper_ai_keywords: LMs (Language models) tokenizes target audio condition modalities source-target pair next-token prediction multi-scale Transformer model residual vector quantization neural codec tokenization foundation model universal audio generation fine-tuning