Qwen2.5-Omni Technical Report
In this report, we present Qwen2.5-Omni, an end-to-end multimodal model
designed to perceive diverse modalities, including text, images, audio, and
video, while simultaneously generating text and natural speech responses in a
streaming manner. To enable the streaming of multimodal information inputs,
both audio and visual encoders utilize a block-wise processing approach. To
synchronize the timestamps of video inputs with audio, we organize the audio
and video sequentially in an interleaved manner and propose a novel position
embedding approach, named TMRoPE(Time-aligned Multimodal RoPE). To concurrently
generate text and speech while avoiding interference between the two
modalities, we propose Thinker-Talker architecture. In this framework,
Thinker functions as a large language model tasked with text generation, while
Talker is a dual-track autoregressive model that directly utilizes the hidden
representations from the Thinker to produce audio tokens as output. Both the
Thinker and Talker models are designed to be trained and inferred in an
end-to-end manner. For decoding audio tokens in a streaming manner, we
introduce a sliding-window DiT that restricts the receptive field, aiming to
reduce the initial package delay. Qwen2.5-Omni is comparable with the similarly
sized Qwen2.5-VL and outperforms Qwen2-Audio. Furthermore, Qwen2.5-Omni
achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench.
Notably, Qwen2.5-Omni's performance in end-to-end speech instruction following
is comparable to its capabilities with text inputs, as evidenced by benchmarks
such as MMLU and GSM8K. As for speech generation, Qwen2.5-Omni's streaming
Talker outperforms most existing streaming and non-streaming alternatives in
robustness and naturalness.
67e4f2507e97884ba4205660
Jin Xu
67e4f2507e97884ba4205661
Zhifang Guo
admin_assigned
2025-03-27T09:52:32.600Z
661e577cbac5d981f883b743
/avatars/95e55e9707a6b55594c264081202d7f4.svg
GuoZhifang
user
ZhifangGuo
67e4f2507e97884ba4205662
Jinzheng He
admin_assigned
2025-03-27T09:52:48.707Z
6594f06ac04427eb38444bce
/avatars/b13fbf589b25eff038deb3fa12d95871.svg
Jinzheng He
user
jinzheng-he
67e4f2507e97884ba4205663
Hangrui Hu
67e4f2507e97884ba4205664
Ting He
67e4f2507e97884ba4205665
Shuai Bai
admin_assigned
2025-03-27T09:53:02.818Z
63451cf0a05b51f7ded25505
/avatars/dec4bbee4a82b773fc58dfc2dce9dbeb.svg
shuai bai
user
bluelike
67e4f2507e97884ba4205666
Keqin Chen
admin_assigned
2025-03-27T09:53:19.770Z
6461d675681b2e19b6acb5a5
/avatars/0d95d65d30f6672ec09dc92155324d7f.svg
Keqin Chen
user
chenkq
67e4f2507e97884ba4205667
Jialin Wang
admin_assigned
2025-03-27T09:53:26.467Z
649a3ba9342f14148357c367
/avatars/81a769fa38b7384f382ff3cc10d6d624.svg
Jialin Wang
user
JialinWang
67e4f2507e97884ba4205668
Yang Fan
67e4f2507e97884ba4205669
Kai Dang
admin_assigned
2025-03-27T09:53:33.105Z
6712930f0fac3235c56edf5b
/avatars/cafe7cb56ce7c3b2572f5f2d0b89357a.svg
kai dang
user
1vk5i
67e4f2507e97884ba420566a
Bin Zhang
67e4f2507e97884ba420566b
Xiong Wang
67e4f2507e97884ba420566c
Yunfei Chu
admin_assigned
2025-03-27T09:53:51.503Z
62c6a751a71b40cf26f359a8
/avatars/49abd2e71946035452c316d703baaac6.svg
Yunfei Chu
user
faychu
67e4f2507e97884ba420566d
Junyang Lin
admin_assigned
2025-03-27T09:53:44.277Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
Qwen2.5-Omni Technical Report
In this report, we present Qwen2.5-Omni, an end-to-end multimodal model
designed to perceive diverse modalities, including text, images, audio, and
video, while simultaneously generating text and natural speech responses in a
streaming manner. To enable the streaming of multimodal information inputs,
both audio and visual encoders utilize a block-wise processing approach. To
synchronize the timestamps of video inputs with audio, we organize the audio
and video sequentially in an interleaved manner and propose a novel position
embedding approach, named TMRoPE(Time-aligned Multimodal RoPE). To concurrently
generate text and speech while avoiding interference between the two
modalities, we propose Thinker-Talker architecture. In this framework,
Thinker functions as a large language model tasked with text generation, while
Talker is a dual-track autoregressive model that directly utilizes the hidden
representations from the Thinker to produce audio tokens as output. Both the
Thinker and Talker models are designed to be trained and inferred in an
end-to-end manner. For decoding audio tokens in a streaming manner, we
introduce a sliding-window DiT that restricts the receptive field, aiming to
reduce the initial package delay. Qwen2.5-Omni is comparable with the similarly
sized Qwen2.5-VL and outperforms Qwen2-Audio. Furthermore, Qwen2.5-Omni
achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench.
Notably, Qwen2.5-Omni's performance in end-to-end speech instruction following
is comparable to its capabilities with text inputs, as evidenced by benchmarks
such as MMLU and GSM8K. As for speech generation, Qwen2.5-Omni's streaming
Talker outperforms most existing streaming and non-streaming alternatives in
robustness and naturalness.
Qwen2.5-Omni is a multimodal model that processes text, images, audio, and video in a streaming fashion and generates text and speech using a dual-track architecture, achieving state-of-the-art performance on multimodal benchmarks.
block-wise processing TMRoPE (Time-aligned Multimodal RoPE) Thinker-Talker architecture dual-track autoregressive model sliding-window DiT Omni-Bench MMLU GSM8K speech instruction following