title: OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment
    across Language with Real-time Self-Aware Emotional Speech Synthesis
summary: Recent advancements in omnimodal learning have been achieved in understanding
  and generation across images, text, and speech, though mainly within
  proprietary models. Limited omnimodal datasets and the inherent challenges
  associated with real-time emotional speech generation have hindered open-source
  progress. To address these issues, we propose openomni, a two-stage training
  method combining omnimodal alignment and speech generation to develop a
  state-of-the-art omnimodal large language model. In the alignment phase, a
  pre-trained speech model is further trained on text-image tasks to generalize
  from vision to speech in a (near) zero-shot manner, outperforming models
  trained on tri-modal datasets. In the speech generation phase, a lightweight
  decoder facilitates real-time emotional speech through training on speech tasks
  and preference learning. Experiments demonstrate that openomni consistently
  improves across omnimodal, vision-language, and speech-language evaluations,
  enabling natural, emotion-rich dialogues and real-time emotional speech
  generation.
paper_authors._id: 677f302c7bd81737ce7a9405
paper_authors.name: Run Luo
paper_authors._id: 677f302c7bd81737ce7a9406
paper_authors.name: Ting-En Lin
paper_authors._id: 677f302c7bd81737ce7a9407
paper_authors.name: Haonan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T20:01:38.484Z
paper_authors._id: 623437bd4f78e3acb8bd14bd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623437bd4f78e3acb8bd14bd/6ZIRpE9We4TiMtzDBtPFS.jpeg
paper_authors.fullname: Haonan Zhang
paper_authors.type: user
paper_authors.user: haonanzhang
paper_authors._id: 677f302c7bd81737ce7a9408
paper_authors.name: Yuchuan Wu
paper_authors._id: 677f302c7bd81737ce7a9409
paper_authors.name: Xiong Liu
paper_authors._id: 677f302c7bd81737ce7a940a
paper_authors.name: Min Yang
paper_authors._id: 677f302c7bd81737ce7a940b
paper_authors.name: Yongbin Li
paper_authors._id: 677f302c7bd81737ce7a940c
paper_authors.name: Longze Chen
paper_authors._id: 677f302c7bd81737ce7a940d
paper_authors.name: Jiaming Li
paper_authors._id: 677f302c7bd81737ce7a940e
paper_authors.name: Lei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T10:07:04.349Z
paper_authors._id: 64c38871f9cd765462fa1a17
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c38871f9cd765462fa1a17/yuIlVcqeDlQVKsUF8uEl3.jpeg
paper_authors.fullname: Lei Zhang
paper_authors.type: user
paper_authors.user: Lemoncoke
paper_authors._id: 677f302c7bd81737ce7a940f
paper_authors.name: Yangyi Chen
paper_authors._id: 677f302c7bd81737ce7a9410
paper_authors.name: Hamid Alinejad-Rokny
paper_authors._id: 677f302c7bd81737ce7a9411
paper_authors.name: Fei Huang
paper_title: OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment
    across Language with Real-time Self-Aware Emotional Speech Synthesis
paper_summary: Recent advancements in omnimodal learning have been achieved in understanding
  and generation across images, text, and speech, though mainly within
  proprietary models. Limited omnimodal datasets and the inherent challenges
  associated with real-time emotional speech generation have hindered open-source
  progress. To address these issues, we propose openomni, a two-stage training
  method combining omnimodal alignment and speech generation to develop a
  state-of-the-art omnimodal large language model. In the alignment phase, a
  pre-trained speech model is further trained on text-image tasks to generalize
  from vision to speech in a (near) zero-shot manner, outperforming models
  trained on tri-modal datasets. In the speech generation phase, a lightweight
  decoder facilitates real-time emotional speech through training on speech tasks
  and preference learning. Experiments demonstrate that openomni consistently
  improves across omnimodal, vision-language, and speech-language evaluations,
  enabling natural, emotion-rich dialogues and real-time emotional speech
  generation.
paper_ai_keywords: omnimodal learning text-image tasks zero-shot manner omnimodal large language model speech generation preference learning omnimodal evaluations vision-language evaluations speech-language evaluations natural, emotion-rich dialogues real-time emotional speech generation