title: EMOVA: Empowering Language Models to See, Hear and Speak with Vivid
    Emotions
summary: GPT-4o, an omni-modal model that enables vocal conversations with diverse
  emotions and tones, marks a milestone for omni-modal foundation models.
  However, empowering Large Language Models to perceive and generate images,
  texts, and speeches end-to-end with publicly available data remains challenging
  in the open-source community. Existing vision-language models rely on external
  tools for the speech processing, while speech-language models still suffer from
  limited or even without vision-understanding abilities. To address this gap, we
  propose EMOVA (EMotionally Omni-present Voice Assistant), to enable Large
  Language Models with end-to-end speech capabilities while maintaining the
  leading vision-language performance. With a semantic-acoustic disentangled
  speech tokenizer, we notice surprisingly that omni-modal alignment can further
  enhance vision-language and speech abilities compared with the corresponding
  bi-modal aligned counterparts. Moreover, a lightweight style module is proposed
  for flexible speech style controls (e.g., emotions and pitches). For the first
  time, EMOVA achieves state-of-the-art performance on both the vision-language
  and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue
  with vivid emotions.
paper_authors._id: 66f61f0b5e1ef72590bbc638
paper_authors.name: Kai Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-27T06:58:09.820Z
paper_authors._id: 65bf3b304b5f8c270d50d6c8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65bf3b304b5f8c270d50d6c8/iSC_Em-JtfcsKFoGw4KFc.jpeg
paper_authors.fullname: Kai Chen
paper_authors.type: user
paper_authors.user: KaiChen1998
paper_authors._id: 66f61f0b5e1ef72590bbc639
paper_authors.name: Yunhao Gou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T09:32:01.795Z
paper_authors._id: 6181435169071e2bddc65578
paper_authors.avatarUrl: /avatars/c898f76c13d380f47b131bcb6289bb9a.svg
paper_authors.fullname: Yunhao Gou
paper_authors.type: user
paper_authors.user: gyhdog
paper_authors._id: 66f61f0b5e1ef72590bbc63a
paper_authors.name: Runhui Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:07:12.129Z
paper_authors._id: 630f0542cc8ed75decb03b68
paper_authors.avatarUrl: /avatars/f76c3603b2700591f33a8a931f7ca664.svg
paper_authors.fullname: huangrh9
paper_authors.type: user
paper_authors.user: huangrh9
paper_authors._id: 66f61f0b5e1ef72590bbc63b
paper_authors.name: Zhili Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T09:33:02.396Z
paper_authors._id: 64f395cef2eeda919ab980a5
paper_authors.avatarUrl: /avatars/f92116eaf94defd9873d9cdb4f98c598.svg
paper_authors.fullname: ZHILI LIU
paper_authors.type: user
paper_authors.user: zhili-liu
paper_authors._id: 66f61f0b5e1ef72590bbc63c
paper_authors.name: Daxin Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:07:09.619Z
paper_authors._id: 6458b6508aa54fb020f5c66c
paper_authors.avatarUrl: /avatars/66c8715fece349b579652989a2c79e41.svg
paper_authors.fullname: Tan
paper_authors.type: user
paper_authors.user: Darcy0123
paper_authors._id: 66f61f0b5e1ef72590bbc63d
paper_authors.name: Jing Xu
paper_authors._id: 66f61f0b5e1ef72590bbc63e
paper_authors.name: Chunwei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T09:33:32.389Z
paper_authors._id: 644131e5666947ad561349c1
paper_authors.avatarUrl: /avatars/8a2d9be81cef30cb3bc71a695c802d2f.svg
paper_authors.fullname: Wangchunwei
paper_authors.type: user
paper_authors.user: 17day
paper_authors._id: 66f61f0b5e1ef72590bbc63f
paper_authors.name: Yi Zhu
paper_authors._id: 66f61f0b5e1ef72590bbc640
paper_authors.name: Yihan Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T09:33:41.204Z
paper_authors._id: 65beb318358734fd090f742e
paper_authors.avatarUrl: /avatars/962420a21b2aa8689dd0e1d4531fbf35.svg
paper_authors.fullname: Yihan Zeng
paper_authors.type: user
paper_authors.user: vikyzeng2
paper_authors._id: 66f61f0b5e1ef72590bbc641
paper_authors.name: Kuo Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-27T09:33:47.438Z
paper_authors._id: 6366c38a95204b4649cc582a
paper_authors.avatarUrl: /avatars/ab97094994b6e82f047b43e436b9e852.svg
paper_authors.fullname: Kuo Yang
paper_authors.type: user
paper_authors.user: kuoyang1999
paper_authors._id: 66f61f0b5e1ef72590bbc642
paper_authors.name: Dingdong Wang
paper_authors._id: 66f61f0b5e1ef72590bbc643
paper_authors.name: Kun Xiang
paper_authors._id: 66f61f0b5e1ef72590bbc644
paper_authors.name: Haoyuan Li
paper_authors._id: 66f61f0b5e1ef72590bbc645
paper_authors.name: Haoli Bai
paper_authors._id: 66f61f0b5e1ef72590bbc646
paper_authors.name: Jianhua Han
paper_authors._id: 66f61f0b5e1ef72590bbc647
paper_authors.name: Xiaohui Li
paper_authors._id: 66f61f0b5e1ef72590bbc648
paper_authors.name: Weike Jin
paper_authors._id: 66f61f0b5e1ef72590bbc649
paper_authors.name: Nian Xie
paper_authors._id: 66f61f0b5e1ef72590bbc64a
paper_authors.name: Yu Zhang
paper_authors._id: 66f61f0b5e1ef72590bbc64b
paper_authors.name: James T. Kwok
paper_authors._id: 66f61f0b5e1ef72590bbc64c
paper_authors.name: Hengshuang Zhao
paper_authors._id: 66f61f0b5e1ef72590bbc64d
paper_authors.name: Xiaodan Liang
paper_authors._id: 66f61f0b5e1ef72590bbc64e
paper_authors.name: Dit-Yan Yeung
paper_authors._id: 66f61f0b5e1ef72590bbc64f
paper_authors.name: Xiao Chen
paper_authors._id: 66f61f0b5e1ef72590bbc650
paper_authors.name: Zhenguo Li
paper_authors._id: 66f61f0b5e1ef72590bbc651
paper_authors.name: Wei Zhang
paper_authors._id: 66f61f0b5e1ef72590bbc652
paper_authors.name: Qun Liu
paper_authors._id: 66f61f0b5e1ef72590bbc653
paper_authors.name: Lanqing Hong
paper_authors._id: 66f61f0b5e1ef72590bbc654
paper_authors.name: Lu Hou
paper_authors._id: 66f61f0b5e1ef72590bbc655
paper_authors.name: Hang Xu
paper_title: EMOVA: Empowering Language Models to See, Hear and Speak with Vivid
    Emotions
paper_summary: GPT-4o, an omni-modal model that enables vocal conversations with diverse
  emotions and tones, marks a milestone for omni-modal foundation models.
  However, empowering Large Language Models to perceive and generate images,
  texts, and speeches end-to-end with publicly available data remains challenging
  in the open-source community. Existing vision-language models rely on external
  tools for the speech processing, while speech-language models still suffer from
  limited or even without vision-understanding abilities. To address this gap, we
  propose EMOVA (EMotionally Omni-present Voice Assistant), to enable Large
  Language Models with end-to-end speech capabilities while maintaining the
  leading vision-language performance. With a semantic-acoustic disentangled
  speech tokenizer, we notice surprisingly that omni-modal alignment can further
  enhance vision-language and speech abilities compared with the corresponding
  bi-modal aligned counterparts. Moreover, a lightweight style module is proposed
  for flexible speech style controls (e.g., emotions and pitches). For the first
  time, EMOVA achieves state-of-the-art performance on both the vision-language
  and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue
  with vivid emotions.
paper_ai_summary: EMOVA integrates speech, vision, and language capabilities, achieving top performance in benchmarks and enabling emotive spoken dialogue using a disentangled speech tokenizer and style module.
paper_ai_keywords: omni-modal model Large Language Models end-to-end speech capabilities view-language models semantic-acoustic disentangled speech tokenizer lightweight style module vision-language benchmarks speech benchmarks omni-modal spoken dialogue