title: Qwen-Audio: Advancing Universal Audio Understanding via Unified
    Large-Scale Audio-Language Models
summary: Recently, instruction-following audio-language models have received broad
  attention for audio interaction with humans. However, the absence of
  pre-trained audio models capable of handling diverse audio types and tasks has
  hindered progress in this field. Consequently, most existing works have only
  been able to support a limited range of interaction capabilities. In this
  paper, we develop the Qwen-Audio model and address this limitation by scaling
  up audio-language pre-training to cover over 30 tasks and various audio types,
  such as human speech, natural sounds, music, and songs, to facilitate universal
  audio understanding abilities. However, directly co-training all tasks and
  datasets can lead to interference issues, as the textual labels associated with
  different datasets exhibit considerable variations due to differences in task
  focus, language, granularity of annotation, and text structure. To overcome the
  one-to-many interference, we carefully design a multi-task training framework
  by conditioning on a sequence of hierarchical tags to the decoder for
  encouraging knowledge sharing and avoiding interference through shared and
  specified tags respectively. Remarkably, Qwen-Audio achieves impressive
  performance across diverse benchmark tasks without requiring any task-specific
  fine-tuning, surpassing its counterparts. Building upon the capabilities of
  Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from
  various audios and text inputs, enabling multi-turn dialogues and supporting
  various audio-central scenarios.
paper_authors._id: 655442a6e149dd8e9f30702a
paper_authors.name: Yunfei Chu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:15:38.617Z
paper_authors._id: 62c6a751a71b40cf26f359a8
paper_authors.avatarUrl: /avatars/49abd2e71946035452c316d703baaac6.svg
paper_authors.fullname: Yunfei Chu
paper_authors.type: user
paper_authors.user: faychu
paper_authors._id: 655442a6e149dd8e9f30702b
paper_authors.name: Jin Xu
paper_authors._id: 655442a6e149dd8e9f30702c
paper_authors.name: Xiaohuan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:54:18.487Z
paper_authors._id: 668f5875b5b3081d776e4094
paper_authors.avatarUrl: /avatars/8c763393f25afbe5fb8b132f775e746a.svg
paper_authors.fullname: Xiaohuan Zhou
paper_authors.type: user
paper_authors.user: XiaohuanZhou
paper_authors._id: 655442a6e149dd8e9f30702d
paper_authors.name: Qian Yang
paper_authors._id: 655442a6e149dd8e9f30702e
paper_authors.name: Shiliang Zhang
paper_authors._id: 655442a6e149dd8e9f30702f
paper_authors.name: Zhijie Yan
paper_authors._id: 655442a6e149dd8e9f307030
paper_authors.name: Chang Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:12:53.677Z
paper_authors._id: 622892774323cef93a956a4a
paper_authors.avatarUrl: /avatars/e57ef5c3b0c4289988ccd42f14e54336.svg
paper_authors.fullname: chang zhou
paper_authors.type: user
paper_authors.user: jiemizc
paper_authors._id: 655442a6e149dd8e9f307031
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:12:35.277Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_title: Qwen-Audio: Advancing Universal Audio Understanding via Unified
    Large-Scale Audio-Language Models
paper_summary: Recently, instruction-following audio-language models have received broad
  attention for audio interaction with humans. However, the absence of
  pre-trained audio models capable of handling diverse audio types and tasks has
  hindered progress in this field. Consequently, most existing works have only
  been able to support a limited range of interaction capabilities. In this
  paper, we develop the Qwen-Audio model and address this limitation by scaling
  up audio-language pre-training to cover over 30 tasks and various audio types,
  such as human speech, natural sounds, music, and songs, to facilitate universal
  audio understanding abilities. However, directly co-training all tasks and
  datasets can lead to interference issues, as the textual labels associated with
  different datasets exhibit considerable variations due to differences in task
  focus, language, granularity of annotation, and text structure. To overcome the
  one-to-many interference, we carefully design a multi-task training framework
  by conditioning on a sequence of hierarchical tags to the decoder for
  encouraging knowledge sharing and avoiding interference through shared and
  specified tags respectively. Remarkably, Qwen-Audio achieves impressive
  performance across diverse benchmark tasks without requiring any task-specific
  fine-tuning, surpassing its counterparts. Building upon the capabilities of
  Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from
  various audios and text inputs, enabling multi-turn dialogues and supporting
  various audio-central scenarios.
paper_ai_summary: Qwen-Audio addresses the limitation of diverse audio types and tasks in instruction-following models through a multi-task training framework with hierarchical tagging, achieving superior performance across benchmarks and enabling Qwen-Audio-Chat for versatile dialogue scenarios.
paper_ai_keywords: instruction-following audio-language models pre-trained audio models multi-task training framework hierarchical tags knowledge sharing Qwen-Audio Qwen-Audio-Chat multi-turn dialogues