title: Autonomous Character-Scene Interaction Synthesis from Text Instruction
summary: Synthesizing human motions in 3D environments, particularly those with
  complex activities such as locomotion, hand-reaching, and human-object
  interaction, presents substantial demands for user-defined waypoints and stage
  transitions. These requirements pose challenges for current models, leading to
  a notable gap in automating the animation of characters from simple human
  inputs. This paper addresses this challenge by introducing a comprehensive
  framework for synthesizing multi-stage scene-aware interaction motions directly
  from a single text instruction and goal location. Our approach employs an
  auto-regressive diffusion model to synthesize the next motion segment, along
  with an autonomous scheduler predicting the transition for each action stage.
  To ensure that the synthesized motions are seamlessly integrated within the
  environment, we propose a scene representation that considers the local
  perception both at the start and the goal location. We further enhance the
  coherence of the generated motion by integrating frame embeddings with language
  input. Additionally, to support model training, we present a comprehensive
  motion-captured dataset comprising 16 hours of motion sequences in 120 indoor
  scenes covering 40 types of motions, each annotated with precise language
  descriptions. Experimental results demonstrate the efficacy of our method in
  generating high-quality, multi-stage motions closely aligned with environmental
  and textual conditions.
paper_authors._id: 6704e701139773976ad519c2
paper_authors.name: Nan Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T10:07:18.456Z
paper_authors._id: 65f015dd6e38bf2ff9711f75
paper_authors.avatarUrl: /avatars/3a6afabfb94d78740ab93a6e3c803b7d.svg
paper_authors.fullname: Nan Jiang
paper_authors.type: user
paper_authors.user: jnnan
paper_authors._id: 6704e701139773976ad519c3
paper_authors.name: Zimo He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T10:06:37.400Z
paper_authors._id: 654a300f589f50f1b19bfcdc
paper_authors.avatarUrl: /avatars/a638c9cb783cd94e1f66e93c09675b69.svg
paper_authors.fullname: Zimo He
paper_authors.type: user
paper_authors.user: milleret
paper_authors._id: 6704e701139773976ad519c4
paper_authors.name: Zi Wang
paper_authors._id: 6704e701139773976ad519c5
paper_authors.name: Hongjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T10:07:46.269Z
paper_authors._id: 6565d682995cc49553fa9f14
paper_authors.avatarUrl: /avatars/0b85c4f9316e2e8c9fea37a0df10089d.svg
paper_authors.fullname: Hongjie Li
paper_authors.type: user
paper_authors.user: awfuact
paper_authors._id: 6704e701139773976ad519c6
paper_authors.name: Yixin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T10:08:10.510Z
paper_authors._id: 64c396def6fe448b1ad553d6
paper_authors.avatarUrl: /avatars/b2ce4739f42dc00ee974fff7ee1cb301.svg
paper_authors.fullname: Yixin Chen
paper_authors.type: user
paper_authors.user: YixinChen
paper_authors._id: 6704e701139773976ad519c7
paper_authors.name: Siyuan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T10:08:32.349Z
paper_authors._id: 63c7a33121bd95f80ed74652
paper_authors.avatarUrl: /avatars/7dd59afea785a2bff0ec2b757abd474e.svg
paper_authors.fullname: Siyuan Huang
paper_authors.type: user
paper_authors.user: thuhsy
paper_authors._id: 6704e701139773976ad519c8
paper_authors.name: Yixin Zhu
paper_title: Autonomous Character-Scene Interaction Synthesis from Text Instruction
paper_summary: Synthesizing human motions in 3D environments, particularly those with
  complex activities such as locomotion, hand-reaching, and human-object
  interaction, presents substantial demands for user-defined waypoints and stage
  transitions. These requirements pose challenges for current models, leading to
  a notable gap in automating the animation of characters from simple human
  inputs. This paper addresses this challenge by introducing a comprehensive
  framework for synthesizing multi-stage scene-aware interaction motions directly
  from a single text instruction and goal location. Our approach employs an
  auto-regressive diffusion model to synthesize the next motion segment, along
  with an autonomous scheduler predicting the transition for each action stage.
  To ensure that the synthesized motions are seamlessly integrated within the
  environment, we propose a scene representation that considers the local
  perception both at the start and the goal location. We further enhance the
  coherence of the generated motion by integrating frame embeddings with language
  input. Additionally, to support model training, we present a comprehensive
  motion-captured dataset comprising 16 hours of motion sequences in 120 indoor
  scenes covering 40 types of motions, each annotated with precise language
  descriptions. Experimental results demonstrate the efficacy of our method in
  generating high-quality, multi-stage motions closely aligned with environmental
  and textual conditions.
paper_ai_summary: A framework using auto-regressive diffusion models and autonomous scheduling synthesizes scene-aware human motions from text instructions and goal locations.
paper_ai_keywords: auto-regressive diffusion model autonomous scheduler scene representation frame embeddings motion-captured dataset multi-stage motions