Physics-based Motion Retargeting from Sparse Inputs
Avatars are important to create interactive and immersive experiences in
virtual worlds. One challenge in animating these characters to mimic a user's
motion is that commercial AR/VR products consist only of a headset and
controllers, providing very limited sensor data of the user's pose. Another
challenge is that an avatar might have a different skeleton structure than a
human and the mapping between them is unclear. In this work we address both of
these challenges. We introduce a method to retarget motions in real-time from
sparse human sensor data to characters of various morphologies. Our method uses
reinforcement learning to train a policy to control characters in a physics
simulator. We only require human motion capture data for training, without
relying on artist-generated animations for each avatar. This allows us to use
large motion capture datasets to train general policies that can track unseen
users from real and sparse data in real-time. We demonstrate the feasibility of
our approach on three characters with different skeleton structure: a dinosaur,
a mouse-like creature and a human. We show that the avatar poses often match
the user surprisingly well, despite having no sensor information of the lower
body available. We discuss and ablate the important components in our
framework, specifically the kinematic retargeting step, the imitation, contact
and action reward as well as our asymmetric actor-critic observations. We
further explore the robustness of our method in a variety of settings including
unbalancing, dancing and sports motions.
64a64cc723caaba1d2e634bf
Daniele Reda
claimed_verified
2023-07-07T07:34:49.270Z
64a7289fc34b44e2ceb1022b
/avatars/cfe83f4fd4f7af71e95d7f4e9dc35963.svg
Daniele Reda
user
rdednl
64a64cc723caaba1d2e634c0
Jungdam Won
64a64cc723caaba1d2e634c1
Yuting Ye
64a64cc723caaba1d2e634c2
Michiel van de Panne
64a64cc723caaba1d2e634c3
Alexander Winkler
Physics-based Motion Retargeting from Sparse Inputs
Avatars are important to create interactive and immersive experiences in
virtual worlds. One challenge in animating these characters to mimic a user's
motion is that commercial AR/VR products consist only of a headset and
controllers, providing very limited sensor data of the user's pose. Another
challenge is that an avatar might have a different skeleton structure than a
human and the mapping between them is unclear. In this work we address both of
these challenges. We introduce a method to retarget motions in real-time from
sparse human sensor data to characters of various morphologies. Our method uses
reinforcement learning to train a policy to control characters in a physics
simulator. We only require human motion capture data for training, without
relying on artist-generated animations for each avatar. This allows us to use
large motion capture datasets to train general policies that can track unseen
users from real and sparse data in real-time. We demonstrate the feasibility of
our approach on three characters with different skeleton structure: a dinosaur,
a mouse-like creature and a human. We show that the avatar poses often match
the user surprisingly well, despite having no sensor information of the lower
body available. We discuss and ablate the important components in our
framework, specifically the kinematic retargeting step, the imitation, contact
and action reward as well as our asymmetric actor-critic observations. We
further explore the robustness of our method in a variety of settings including
unbalancing, dancing and sports motions.
Real-time motion retargeting from sparse human sensor data to diverse avatar morphologies is achieved using reinforcement learning in physics simulators.
motion retargeting reinforcement learning policy physics simulator motion capture data kinematic retargeting imitation reward contact reward action reward asymmetric actor-critic robustness