title: OmniResponse: Online Multimodal Conversational Response Generation in
    Dyadic Interactions
summary: In this paper, we introduce Online Multimodal Conversational Response
  Generation (OMCRG), a novel task that aims to online generate synchronized
  verbal and non-verbal listener feedback, conditioned on the speaker's
  multimodal input. OMCRG reflects natural dyadic interactions and poses new
  challenges in achieving synchronization between the generated audio and facial
  responses of the listener. To address these challenges, we innovatively
  introduce text as an intermediate modality to bridge the audio and facial
  responses. We hence propose OmniResponse, a Multimodal Large Language Model
  (MLLM) that autoregressively generates high-quality multi-modal listener
  responses. OmniResponse leverages a pretrained LLM enhanced with two novel
  components: Chrono-Text, which temporally anchors generated text tokens, and
  TempoVoice, a controllable online TTS module that produces speech synchronized
  with facial reactions. To support further OMCRG research, we present
  ResponseNet, a new dataset comprising 696 high-quality dyadic interactions
  featuring synchronized split-screen videos, multichannel audio, transcripts,
  and facial behavior annotations. Comprehensive evaluations conducted on
  ResponseNet demonstrate that OmniResponse significantly outperforms baseline
  models in terms of semantic speech content, audio-visual synchronization, and
  generation quality.
paper_authors._id: 683b44583f2842f6afcc5e6f
paper_authors.name: Cheng Luo
paper_authors._id: 683b44583f2842f6afcc5e70
paper_authors.name: Jianghui Wang
paper_authors._id: 683b44583f2842f6afcc5e71
paper_authors.name: Bing Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:45:31.879Z
paper_authors._id: 666ddb45c0f3d5afc27e85ba
paper_authors.avatarUrl: /avatars/dce98fc77bd8cf9e348f2d91bc3c0225.svg
paper_authors.fullname: Bing Li
paper_authors.type: user
paper_authors.user: bing-li-ai
paper_authors._id: 683b44583f2842f6afcc5e72
paper_authors.name: Siyang Song
paper_authors._id: 683b44583f2842f6afcc5e73
paper_authors.name: Bernard Ghanem
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:45:46.322Z
paper_authors._id: 6808bf97ffadd78ec71cb721
paper_authors.avatarUrl: /avatars/9adca3142c06b8f69889fcbe85fa374d.svg
paper_authors.fullname: Bernard Ghanem
paper_authors.type: user
paper_authors.user: bernardghanem
paper_title: OmniResponse: Online Multimodal Conversational Response Generation in
    Dyadic Interactions
paper_summary: In this paper, we introduce Online Multimodal Conversational Response
  Generation (OMCRG), a novel task that aims to online generate synchronized
  verbal and non-verbal listener feedback, conditioned on the speaker's
  multimodal input. OMCRG reflects natural dyadic interactions and poses new
  challenges in achieving synchronization between the generated audio and facial
  responses of the listener. To address these challenges, we innovatively
  introduce text as an intermediate modality to bridge the audio and facial
  responses. We hence propose OmniResponse, a Multimodal Large Language Model
  (MLLM) that autoregressively generates high-quality multi-modal listener
  responses. OmniResponse leverages a pretrained LLM enhanced with two novel
  components: Chrono-Text, which temporally anchors generated text tokens, and
  TempoVoice, a controllable online TTS module that produces speech synchronized
  with facial reactions. To support further OMCRG research, we present
  ResponseNet, a new dataset comprising 696 high-quality dyadic interactions
  featuring synchronized split-screen videos, multichannel audio, transcripts,
  and facial behavior annotations. Comprehensive evaluations conducted on
  ResponseNet demonstrate that OmniResponse significantly outperforms baseline
  models in terms of semantic speech content, audio-visual synchronization, and
  generation quality.
paper_ai_summary: OmniResponse, a Multimodal Large Language Model, generates high-quality synchronized verbal and non-verbal listener responses using text as an intermediate modality.
paper_ai_keywords: Online Multimodal Conversational Response Generation OmniResponse Multimodal Large Language Model Chrono-Text TempoVoice ResponseNet audio-visual synchronization