title: SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction
    with 3D Autonomous Characters
summary: Human beings are social animals. How to equip 3D autonomous characters with
  similar social intelligence that can perceive, understand and interact with
  humans remains an open yet foundamental problem. In this paper, we introduce
  SOLAMI, the first end-to-end Social vision-Language-Action (VLA) Modeling
  framework for Immersive interaction with 3D autonomous characters.
  Specifically, SOLAMI builds 3D autonomous characters from three aspects: (1)
  Social VLA Architecture: We propose a unified social VLA framework to generate
  multimodal response (speech and motion) based on the user's multimodal input to
  drive the character for social interaction. (2) Interactive Multimodal Data: We
  present SynMSI, a synthetic multimodal social interaction dataset generated by
  an automatic pipeline using only existing motion datasets to address the issue
  of data scarcity. (3) Immersive VR Interface: We develop a VR interface that
  enables users to immersively interact with these characters driven by various
  architectures. Extensive quantitative experiments and user studies demonstrate
  that our framework leads to more precise and natural character responses (in
  both speech and motion) that align with user expectations with lower latency.
paper_authors._id: 674eadabef38dd7bfb8863cf
paper_authors.name: Jianping Jiang
paper_authors._id: 674eadabef38dd7bfb8863d0
paper_authors.name: Weiye Xiao
paper_authors._id: 674eadabef38dd7bfb8863d1
paper_authors.name: Zhengyu Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:18:17.310Z
paper_authors._id: 6391fc80077fee9fbca29790
paper_authors.avatarUrl: /avatars/53f89c6de4e91cdcaaa52a295a5f4ea4.svg
paper_authors.fullname: Zhengyu Lin
paper_authors.type: user
paper_authors.user: hayashiLin
paper_authors._id: 674eadabef38dd7bfb8863d2
paper_authors.name: Huaizhong Zhang
paper_authors._id: 674eadabef38dd7bfb8863d3
paper_authors.name: Tianxiang Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:18:45.694Z
paper_authors._id: 66652269d22ee88501a25264
paper_authors.avatarUrl: /avatars/5680b90d20e179be4e56ceabb3aa145d.svg
paper_authors.fullname: Tianxiang Ren
paper_authors.type: user
paper_authors.user: rentxray
paper_authors._id: 674eadabef38dd7bfb8863d4
paper_authors.name: Yang Gao
paper_authors._id: 674eadabef38dd7bfb8863d5
paper_authors.name: Zhiqian Lin
paper_authors._id: 674eadabef38dd7bfb8863d6
paper_authors.name: Zhongang Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T09:56:37.337Z
paper_authors._id: 652d06833b5997ed71ce5c46
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/xZTXEcnEogEmBm_ledJQr.jpeg
paper_authors.fullname: Zhongang Cai
paper_authors.type: user
paper_authors.user: caizhongang
paper_authors._id: 674eadabef38dd7bfb8863d7
paper_authors.name: Lei Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T14:40:45.662Z
paper_authors._id: 6626a471430a124253f197c8
paper_authors.avatarUrl: /avatars/f5747fdbe495d1296fed9d16d8c95857.svg
paper_authors.fullname: yl-1993
paper_authors.type: user
paper_authors.user: yl-1993
paper_authors._id: 674eadabef38dd7bfb8863d8
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:20:46.734Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction
    with 3D Autonomous Characters
paper_summary: Human beings are social animals. How to equip 3D autonomous characters with
  similar social intelligence that can perceive, understand and interact with
  humans remains an open yet foundamental problem. In this paper, we introduce
  SOLAMI, the first end-to-end Social vision-Language-Action (VLA) Modeling
  framework for Immersive interaction with 3D autonomous characters.
  Specifically, SOLAMI builds 3D autonomous characters from three aspects: (1)
  Social VLA Architecture: We propose a unified social VLA framework to generate
  multimodal response (speech and motion) based on the user's multimodal input to
  drive the character for social interaction. (2) Interactive Multimodal Data: We
  present SynMSI, a synthetic multimodal social interaction dataset generated by
  an automatic pipeline using only existing motion datasets to address the issue
  of data scarcity. (3) Immersive VR Interface: We develop a VR interface that
  enables users to immersively interact with these characters driven by various
  architectures. Extensive quantitative experiments and user studies demonstrate
  that our framework leads to more precise and natural character responses (in
  both speech and motion) that align with user expectations with lower latency.
paper_ai_summary: SOLAMI is an end-to-end VLA framework for 3D autonomous characters that addresses social interaction by generating multimodal responses and enabling immersive VR interaction.
paper_ai_keywords: Social VLA Architecture SynMSI VR interface multimodal response multimodal input social interaction user expectations quantitative experiments user studies