title: Qwen2-Audio Technical Report
summary: We introduce the latest progress of Qwen-Audio, a large-scale audio-language
  model called Qwen2-Audio, which is capable of accepting various audio signal
  inputs and performing audio analysis or direct textual responses with regard to
  speech instructions. In contrast to complex hierarchical tags, we have
  simplified the pre-training process by utilizing natural language prompts for
  different data and tasks, and have further expanded the data volume. We have
  boosted the instruction-following capability of Qwen2-Audio and implemented two
  distinct audio interaction modes for voice chat and audio analysis. In the
  voice chat mode, users can freely engage in voice interactions with Qwen2-Audio
  without text input. In the audio analysis mode, users could provide audio and
  text instructions for analysis during the interaction. Note that we do not use
  any system prompts to switch between voice chat and audio analysis modes.
  Qwen2-Audio is capable of intelligently comprehending the content within audio
  and following voice commands to respond appropriately. For instance, in an
  audio segment that simultaneously contains sounds, multi-speaker conversations,
  and a voice command, Qwen2-Audio can directly understand the command and
  provide an interpretation and response to the audio. Additionally, DPO has
  optimized the model's performance in terms of factuality and adherence to
  desired behavior. According to the evaluation results from AIR-Bench,
  Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests
  focused on audio-centric instruction-following capabilities. Qwen2-Audio is
  open-sourced with the aim of fostering the advancement of the multi-modal
  language community.
paper_authors._id: 66973aa13db47c0771e24d99
paper_authors.name: Yunfei Chu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:19:50.377Z
paper_authors._id: 62c6a751a71b40cf26f359a8
paper_authors.avatarUrl: /avatars/49abd2e71946035452c316d703baaac6.svg
paper_authors.fullname: Yunfei Chu
paper_authors.type: user
paper_authors.user: faychu
paper_authors._id: 66973aa13db47c0771e24d9a
paper_authors.name: Jin Xu
paper_authors._id: 66973aa13db47c0771e24d9b
paper_authors.name: Qian Yang
paper_authors._id: 66973aa13db47c0771e24d9c
paper_authors.name: Haojie Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:21:40.644Z
paper_authors._id: 63bd0f800608fbc76b837e3f
paper_authors.avatarUrl: /avatars/b3089c6162e4db7ffbdd2700053c0e85.svg
paper_authors.fullname: Haojie Wei
paper_authors.type: user
paper_authors.user: Dream-High
paper_authors._id: 66973aa13db47c0771e24d9d
paper_authors.name: Xipin Wei
paper_authors._id: 66973aa13db47c0771e24d9e
paper_authors.name: Zhifang Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:21:21.580Z
paper_authors._id: 661e577cbac5d981f883b743
paper_authors.avatarUrl: /avatars/95e55e9707a6b55594c264081202d7f4.svg
paper_authors.fullname: GuoZhifang
paper_authors.type: user
paper_authors.user: ZhifangGuo
paper_authors._id: 66973aa13db47c0771e24d9f
paper_authors.name: Yichong Leng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:21:14.428Z
paper_authors._id: 64a0347b528a9bbe59d6e08c
paper_authors.avatarUrl: /avatars/6dd0bad84d711d1048a0a4169e621773.svg
paper_authors.fullname: Yichong Leng
paper_authors.type: user
paper_authors.user: ustcscallion
paper_authors._id: 66973aa13db47c0771e24da0
paper_authors.name: Yuanjun Lv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:06:42.599Z
paper_authors._id: 659e513ea9bc1f60189ac148
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/659e513ea9bc1f60189ac148/DBDDqjGTQ0SvjWyuYu7py.jpeg
paper_authors.fullname: YuanjunLv
paper_authors.type: user
paper_authors.user: Bakerbunker
paper_authors._id: 66973aa13db47c0771e24da1
paper_authors.name: Jinzheng He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:21:06.819Z
paper_authors._id: 6594f06ac04427eb38444bce
paper_authors.avatarUrl: /avatars/b13fbf589b25eff038deb3fa12d95871.svg
paper_authors.fullname: Jinzheng He
paper_authors.type: user
paper_authors.user: jinzheng-he
paper_authors._id: 66973aa13db47c0771e24da2
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:20:59.566Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_authors._id: 66973aa13db47c0771e24da3
paper_authors.name: Chang Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:20:51.996Z
paper_authors._id: 622892774323cef93a956a4a
paper_authors.avatarUrl: /avatars/e57ef5c3b0c4289988ccd42f14e54336.svg
paper_authors.fullname: chang zhou
paper_authors.type: user
paper_authors.user: jiemizc
paper_authors._id: 66973aa13db47c0771e24da4
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:20:36.040Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_title: Qwen2-Audio Technical Report
paper_summary: We introduce the latest progress of Qwen-Audio, a large-scale audio-language
  model called Qwen2-Audio, which is capable of accepting various audio signal
  inputs and performing audio analysis or direct textual responses with regard to
  speech instructions. In contrast to complex hierarchical tags, we have
  simplified the pre-training process by utilizing natural language prompts for
  different data and tasks, and have further expanded the data volume. We have
  boosted the instruction-following capability of Qwen2-Audio and implemented two
  distinct audio interaction modes for voice chat and audio analysis. In the
  voice chat mode, users can freely engage in voice interactions with Qwen2-Audio
  without text input. In the audio analysis mode, users could provide audio and
  text instructions for analysis during the interaction. Note that we do not use
  any system prompts to switch between voice chat and audio analysis modes.
  Qwen2-Audio is capable of intelligently comprehending the content within audio
  and following voice commands to respond appropriately. For instance, in an
  audio segment that simultaneously contains sounds, multi-speaker conversations,
  and a voice command, Qwen2-Audio can directly understand the command and
  provide an interpretation and response to the audio. Additionally, DPO has
  optimized the model's performance in terms of factuality and adherence to
  desired behavior. According to the evaluation results from AIR-Bench,
  Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests
  focused on audio-centric instruction-following capabilities. Qwen2-Audio is
  open-sourced with the aim of fostering the advancement of the multi-modal
  language community.
paper_ai_keywords: audio-language model Qwen2-Audio audio signal inputs natural language prompts instruction-following capability voice chat mode audio analysis mode voice interactions audio analysis content within audio multi-speaker conversations voice commands DPO factuality desired behavior AIR-Bench Gemini-1.5-pro multi-modal