title: Qwen2.5-Omni Technical Report
summary: In this report, we present Qwen2.5-Omni, an end-to-end multimodal model
  designed to perceive diverse modalities, including text, images, audio, and
  video, while simultaneously generating text and natural speech responses in a
  streaming manner. To enable the streaming of multimodal information inputs,
  both audio and visual encoders utilize a block-wise processing approach. To
  synchronize the timestamps of video inputs with audio, we organize the audio
  and video sequentially in an interleaved manner and propose a novel position
  embedding approach, named TMRoPE(Time-aligned Multimodal RoPE). To concurrently
  generate text and speech while avoiding interference between the two
  modalities, we propose Thinker-Talker architecture. In this framework,
  Thinker functions as a large language model tasked with text generation, while
  Talker is a dual-track autoregressive model that directly utilizes the hidden
  representations from the Thinker to produce audio tokens as output. Both the
  Thinker and Talker models are designed to be trained and inferred in an
  end-to-end manner. For decoding audio tokens in a streaming manner, we
  introduce a sliding-window DiT that restricts the receptive field, aiming to
  reduce the initial package delay. Qwen2.5-Omni is comparable with the similarly
  sized Qwen2.5-VL and outperforms Qwen2-Audio. Furthermore, Qwen2.5-Omni
  achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench.
  Notably, Qwen2.5-Omni's performance in end-to-end speech instruction following
  is comparable to its capabilities with text inputs, as evidenced by benchmarks
  such as MMLU and GSM8K. As for speech generation, Qwen2.5-Omni's streaming
  Talker outperforms most existing streaming and non-streaming alternatives in
  robustness and naturalness.
paper_authors._id: 67e4f2507e97884ba4205660
paper_authors.name: Jin Xu
paper_authors._id: 67e4f2507e97884ba4205661
paper_authors.name: Zhifang Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:52:32.600Z
paper_authors._id: 661e577cbac5d981f883b743
paper_authors.avatarUrl: /avatars/95e55e9707a6b55594c264081202d7f4.svg
paper_authors.fullname: GuoZhifang
paper_authors.type: user
paper_authors.user: ZhifangGuo
paper_authors._id: 67e4f2507e97884ba4205662
paper_authors.name: Jinzheng He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:52:48.707Z
paper_authors._id: 6594f06ac04427eb38444bce
paper_authors.avatarUrl: /avatars/b13fbf589b25eff038deb3fa12d95871.svg
paper_authors.fullname: Jinzheng He
paper_authors.type: user
paper_authors.user: jinzheng-he
paper_authors._id: 67e4f2507e97884ba4205663
paper_authors.name: Hangrui Hu
paper_authors._id: 67e4f2507e97884ba4205664
paper_authors.name: Ting He
paper_authors._id: 67e4f2507e97884ba4205665
paper_authors.name: Shuai Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:53:02.818Z
paper_authors._id: 63451cf0a05b51f7ded25505
paper_authors.avatarUrl: /avatars/dec4bbee4a82b773fc58dfc2dce9dbeb.svg
paper_authors.fullname: shuai bai
paper_authors.type: user
paper_authors.user: bluelike
paper_authors._id: 67e4f2507e97884ba4205666
paper_authors.name: Keqin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:53:19.770Z
paper_authors._id: 6461d675681b2e19b6acb5a5
paper_authors.avatarUrl: /avatars/0d95d65d30f6672ec09dc92155324d7f.svg
paper_authors.fullname: Keqin Chen
paper_authors.type: user
paper_authors.user: chenkq
paper_authors._id: 67e4f2507e97884ba4205667
paper_authors.name: Jialin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:53:26.467Z
paper_authors._id: 649a3ba9342f14148357c367
paper_authors.avatarUrl: /avatars/81a769fa38b7384f382ff3cc10d6d624.svg
paper_authors.fullname: Jialin Wang
paper_authors.type: user
paper_authors.user: JialinWang
paper_authors._id: 67e4f2507e97884ba4205668
paper_authors.name: Yang Fan
paper_authors._id: 67e4f2507e97884ba4205669
paper_authors.name: Kai Dang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:53:33.105Z
paper_authors._id: 6712930f0fac3235c56edf5b
paper_authors.avatarUrl: /avatars/cafe7cb56ce7c3b2572f5f2d0b89357a.svg
paper_authors.fullname: kai dang
paper_authors.type: user
paper_authors.user: 1vk5i
paper_authors._id: 67e4f2507e97884ba420566a
paper_authors.name: Bin Zhang
paper_authors._id: 67e4f2507e97884ba420566b
paper_authors.name: Xiong Wang
paper_authors._id: 67e4f2507e97884ba420566c
paper_authors.name: Yunfei Chu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:53:51.503Z
paper_authors._id: 62c6a751a71b40cf26f359a8
paper_authors.avatarUrl: /avatars/49abd2e71946035452c316d703baaac6.svg
paper_authors.fullname: Yunfei Chu
paper_authors.type: user
paper_authors.user: faychu
paper_authors._id: 67e4f2507e97884ba420566d
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:53:44.277Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_title: Qwen2.5-Omni Technical Report
paper_summary: In this report, we present Qwen2.5-Omni, an end-to-end multimodal model
  designed to perceive diverse modalities, including text, images, audio, and
  video, while simultaneously generating text and natural speech responses in a
  streaming manner. To enable the streaming of multimodal information inputs,
  both audio and visual encoders utilize a block-wise processing approach. To
  synchronize the timestamps of video inputs with audio, we organize the audio
  and video sequentially in an interleaved manner and propose a novel position
  embedding approach, named TMRoPE(Time-aligned Multimodal RoPE). To concurrently
  generate text and speech while avoiding interference between the two
  modalities, we propose Thinker-Talker architecture. In this framework,
  Thinker functions as a large language model tasked with text generation, while
  Talker is a dual-track autoregressive model that directly utilizes the hidden
  representations from the Thinker to produce audio tokens as output. Both the
  Thinker and Talker models are designed to be trained and inferred in an
  end-to-end manner. For decoding audio tokens in a streaming manner, we
  introduce a sliding-window DiT that restricts the receptive field, aiming to
  reduce the initial package delay. Qwen2.5-Omni is comparable with the similarly
  sized Qwen2.5-VL and outperforms Qwen2-Audio. Furthermore, Qwen2.5-Omni
  achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench.
  Notably, Qwen2.5-Omni's performance in end-to-end speech instruction following
  is comparable to its capabilities with text inputs, as evidenced by benchmarks
  such as MMLU and GSM8K. As for speech generation, Qwen2.5-Omni's streaming
  Talker outperforms most existing streaming and non-streaming alternatives in
  robustness and naturalness.
paper_ai_summary: Qwen2.5-Omni is a multimodal model that processes text, images, audio, and video in a streaming fashion and generates text and speech using a dual-track architecture, achieving state-of-the-art performance on multimodal benchmarks.
paper_ai_keywords: block-wise processing TMRoPE (Time-aligned Multimodal RoPE) Thinker-Talker architecture dual-track autoregressive model sliding-window DiT Omni-Bench MMLU GSM8K speech instruction following