OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment
  across Language with Real-time Self-Aware Emotional Speech Synthesis
Recent advancements in omnimodal learning have been achieved in understanding
and generation across images, text, and speech, though mainly within
proprietary models. Limited omnimodal datasets and the inherent challenges
associated with real-time emotional speech generation have hindered open-source
progress. To address these issues, we propose openomni, a two-stage training
method combining omnimodal alignment and speech generation to develop a
state-of-the-art omnimodal large language model. In the alignment phase, a
pre-trained speech model is further trained on text-image tasks to generalize
from vision to speech in a (near) zero-shot manner, outperforming models
trained on tri-modal datasets. In the speech generation phase, a lightweight
decoder facilitates real-time emotional speech through training on speech tasks
and preference learning. Experiments demonstrate that openomni consistently
improves across omnimodal, vision-language, and speech-language evaluations,
enabling natural, emotion-rich dialogues and real-time emotional speech
generation.
677f302c7bd81737ce7a9405
Run Luo
677f302c7bd81737ce7a9406
Ting-En Lin
677f302c7bd81737ce7a9407
Haonan Zhang
claimed_verified
2025-01-09T20:01:38.484Z
623437bd4f78e3acb8bd14bd
https://cdn-avatars.huggingface.co/v1/production/uploads/623437bd4f78e3acb8bd14bd/6ZIRpE9We4TiMtzDBtPFS.jpeg
Haonan Zhang
user
haonanzhang
677f302c7bd81737ce7a9408
Yuchuan Wu
677f302c7bd81737ce7a9409
Xiong Liu
677f302c7bd81737ce7a940a
Min Yang
677f302c7bd81737ce7a940b
Yongbin Li
677f302c7bd81737ce7a940c
Longze Chen
claimed_verified
2025-06-04T09:05:51.650Z
64c7b4d1c547ed5243c07b6c
https://cdn-avatars.huggingface.co/v1/production/uploads/64c7b4d1c547ed5243c07b6c/h96CLBj6dcm01soK2UJzr.jpeg
Longze Chen
user
lzchen2001
677f302c7bd81737ce7a940d
Jiaming Li
677f302c7bd81737ce7a940e
Lei Zhang
claimed_verified
2025-01-09T10:07:04.349Z
64c38871f9cd765462fa1a17
https://cdn-avatars.huggingface.co/v1/production/uploads/64c38871f9cd765462fa1a17/yuIlVcqeDlQVKsUF8uEl3.jpeg
Lei Zhang
user
Lemoncoke
677f302c7bd81737ce7a940f
Yangyi Chen
677f302c7bd81737ce7a9410
Hamid Alinejad-Rokny
677f302c7bd81737ce7a9411
Fei Huang
OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment
  across Language with Real-time Self-Aware Emotional Speech Synthesis
Recent advancements in omnimodal learning have been achieved in understanding
and generation across images, text, and speech, though mainly within
proprietary models. Limited omnimodal datasets and the inherent challenges
associated with real-time emotional speech generation have hindered open-source
progress. To address these issues, we propose openomni, a two-stage training
method combining omnimodal alignment and speech generation to develop a
state-of-the-art omnimodal large language model. In the alignment phase, a
pre-trained speech model is further trained on text-image tasks to generalize
from vision to speech in a (near) zero-shot manner, outperforming models
trained on tri-modal datasets. In the speech generation phase, a lightweight
decoder facilitates real-time emotional speech through training on speech tasks
and preference learning. Experiments demonstrate that openomni consistently
improves across omnimodal, vision-language, and speech-language evaluations,
enabling natural, emotion-rich dialogues and real-time emotional speech
generation.
Openomni, a two-stage training method combining omnimodal alignment and speech generation, achieves state-of-the-art performance in generating natural, emotion-rich dialogues and real-time emotional speech.
omnimodal learning pre-trained speech model text-image tasks zero-shot lightweight decoder preference learning omnimodal large language model