Autonomous Character-Scene Interaction Synthesis from Text Instruction
Synthesizing human motions in 3D environments, particularly those with
complex activities such as locomotion, hand-reaching, and human-object
interaction, presents substantial demands for user-defined waypoints and stage
transitions. These requirements pose challenges for current models, leading to
a notable gap in automating the animation of characters from simple human
inputs. This paper addresses this challenge by introducing a comprehensive
framework for synthesizing multi-stage scene-aware interaction motions directly
from a single text instruction and goal location. Our approach employs an
auto-regressive diffusion model to synthesize the next motion segment, along
with an autonomous scheduler predicting the transition for each action stage.
To ensure that the synthesized motions are seamlessly integrated within the
environment, we propose a scene representation that considers the local
perception both at the start and the goal location. We further enhance the
coherence of the generated motion by integrating frame embeddings with language
input. Additionally, to support model training, we present a comprehensive
motion-captured dataset comprising 16 hours of motion sequences in 120 indoor
scenes covering 40 types of motions, each annotated with precise language
descriptions. Experimental results demonstrate the efficacy of our method in
generating high-quality, multi-stage motions closely aligned with environmental
and textual conditions.
6704e701139773976ad519c2
Nan Jiang
admin_assigned
2024-10-08T10:07:18.456Z
65f015dd6e38bf2ff9711f75
/avatars/3a6afabfb94d78740ab93a6e3c803b7d.svg
Nan Jiang
user
jnnan
6704e701139773976ad519c3
Zimo He
admin_assigned
2024-10-08T10:06:37.400Z
654a300f589f50f1b19bfcdc
/avatars/a638c9cb783cd94e1f66e93c09675b69.svg
Zimo He
user
milleret
6704e701139773976ad519c4
Zi Wang
6704e701139773976ad519c5
Hongjie Li
admin_assigned
2024-10-08T10:07:46.269Z
6565d682995cc49553fa9f14
/avatars/0b85c4f9316e2e8c9fea37a0df10089d.svg
Hongjie Li
user
awfuact
6704e701139773976ad519c6
Yixin Chen
admin_assigned
2024-10-08T10:08:10.510Z
64c396def6fe448b1ad553d6
/avatars/b2ce4739f42dc00ee974fff7ee1cb301.svg
Yixin Chen
user
YixinChen
6704e701139773976ad519c7
Siyuan Huang
admin_assigned
2024-10-08T10:08:32.349Z
63c7a33121bd95f80ed74652
/avatars/7dd59afea785a2bff0ec2b757abd474e.svg
Siyuan Huang
user
thuhsy
6704e701139773976ad519c8
Yixin Zhu
Autonomous Character-Scene Interaction Synthesis from Text Instruction
Synthesizing human motions in 3D environments, particularly those with
complex activities such as locomotion, hand-reaching, and human-object
interaction, presents substantial demands for user-defined waypoints and stage
transitions. These requirements pose challenges for current models, leading to
a notable gap in automating the animation of characters from simple human
inputs. This paper addresses this challenge by introducing a comprehensive
framework for synthesizing multi-stage scene-aware interaction motions directly
from a single text instruction and goal location. Our approach employs an
auto-regressive diffusion model to synthesize the next motion segment, along
with an autonomous scheduler predicting the transition for each action stage.
To ensure that the synthesized motions are seamlessly integrated within the
environment, we propose a scene representation that considers the local
perception both at the start and the goal location. We further enhance the
coherence of the generated motion by integrating frame embeddings with language
input. Additionally, to support model training, we present a comprehensive
motion-captured dataset comprising 16 hours of motion sequences in 120 indoor
scenes covering 40 types of motions, each annotated with precise language
descriptions. Experimental results demonstrate the efficacy of our method in
generating high-quality, multi-stage motions closely aligned with environmental
and textual conditions.
A framework using auto-regressive diffusion models and autonomous scheduling synthesizes scene-aware human motions from text instructions and goal locations.
auto-regressive diffusion model autonomous scheduler scene representation frame embeddings motion-captured dataset multi-stage motions