title: ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting
summary: Multimodal immersive spatial drama generation focuses on creating continuous
  multi-speaker binaural speech with dramatic prosody based on multimodal
  prompts, with potential applications in AR, VR, and others. This task requires
  simultaneous modeling of spatial information and dramatic prosody based on
  multimodal inputs, with high data collection costs. To the best of our
  knowledge, our work is the first attempt to address these challenges. We
  construct MRSDrama, the first multimodal recorded spatial drama dataset,
  containing binaural drama audios, scripts, videos, geometric poses, and textual
  prompts. Then, we propose ISDrama, the first immersive spatial drama generation
  model through multimodal prompting. ISDrama comprises these primary components:
  1) Multimodal Pose Encoder, based on contrastive learning, considering the
  Doppler effect caused by moving speakers to extract unified pose information
  from multimodal prompts. 2) Immersive Drama Transformer, a flow-based
  mamba-transformer model that generates high-quality drama, incorporating
  Drama-MOE to select proper experts for enhanced prosody and pose control. We
  also design a context-consistent classifier-free guidance strategy to
  coherently generate complete drama. Experimental results show that ISDrama
  outperforms baseline models on objective and subjective metrics. The demos and
  dataset are available at https://aaronz345.github.io/ISDramaDemo.
paper_authors._id: 6811ea47f8ca0d9acb45374b
paper_authors.name: Yu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T09:56:39.194Z
paper_authors._id: 66569729ea21cfae5f5797c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66569729ea21cfae5f5797c4/IguwJzljFN3QiEd1bn5BP.jpeg
paper_authors.fullname: Yu Zhang
paper_authors.type: user
paper_authors.user: AaronZ345
paper_authors._id: 6811ea47f8ca0d9acb45374c
paper_authors.name: Wenxiang Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:22:24.333Z
paper_authors._id: 66692faa83408bb0da40f8f5
paper_authors.avatarUrl: /avatars/09f81a8b0bb130c8fa72dbb6526ac4c4.svg
paper_authors.fullname: wenxiang guo
paper_authors.type: user
paper_authors.user: verstar
paper_authors._id: 6811ea47f8ca0d9acb45374d
paper_authors.name: Changhao Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:22:36.396Z
paper_authors._id: 66568060c6a8cb4e884be331
paper_authors.avatarUrl: /avatars/fe8de503648dda54307fec90cace7c07.svg
paper_authors.fullname: PanChanghao
paper_authors.type: user
paper_authors.user: DavidPigeon
paper_authors._id: 6811ea47f8ca0d9acb45374e
paper_authors.name: Zhiyuan Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:22:42.999Z
paper_authors._id: 66984c6a03421b737aa78f24
paper_authors.avatarUrl: /avatars/9c92b65469c0ef5f05a4c3c45a6dec97.svg
paper_authors.fullname: Zhiyuan Zhu
paper_authors.type: user
paper_authors.user: zhiyuanzhu2019
paper_authors._id: 6811ea47f8ca0d9acb45374f
paper_authors.name: Tao Jin
paper_authors._id: 6811ea47f8ca0d9acb453750
paper_authors.name: Zhou Zhao
paper_title: ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting
paper_summary: Multimodal immersive spatial drama generation focuses on creating continuous
  multi-speaker binaural speech with dramatic prosody based on multimodal
  prompts, with potential applications in AR, VR, and others. This task requires
  simultaneous modeling of spatial information and dramatic prosody based on
  multimodal inputs, with high data collection costs. To the best of our
  knowledge, our work is the first attempt to address these challenges. We
  construct MRSDrama, the first multimodal recorded spatial drama dataset,
  containing binaural drama audios, scripts, videos, geometric poses, and textual
  prompts. Then, we propose ISDrama, the first immersive spatial drama generation
  model through multimodal prompting. ISDrama comprises these primary components:
  1) Multimodal Pose Encoder, based on contrastive learning, considering the
  Doppler effect caused by moving speakers to extract unified pose information
  from multimodal prompts. 2) Immersive Drama Transformer, a flow-based
  mamba-transformer model that generates high-quality drama, incorporating
  Drama-MOE to select proper experts for enhanced prosody and pose control. We
  also design a context-consistent classifier-free guidance strategy to
  coherently generate complete drama. Experimental results show that ISDrama
  outperforms baseline models on objective and subjective metrics. The demos and
  dataset are available at https://aaronz345.github.io/ISDramaDemo.
paper_ai_summary: A multimodal spatial drama generation model, ISDrama, leverages contrastive learning and a flow-based transformer to produce high-quality binaural speech with dramatic prosody from multimodal inputs.
paper_ai_keywords: multipodal Pose Encoder contrastive learning Immersive Drama Transformer flow-based transformer Drama-MOE context-consistent classifier-free guidance