title: Physics-based Motion Retargeting from Sparse Inputs
summary: Avatars are important to create interactive and immersive experiences in
  virtual worlds. One challenge in animating these characters to mimic a user's
  motion is that commercial AR/VR products consist only of a headset and
  controllers, providing very limited sensor data of the user's pose. Another
  challenge is that an avatar might have a different skeleton structure than a
  human and the mapping between them is unclear. In this work we address both of
  these challenges. We introduce a method to retarget motions in real-time from
  sparse human sensor data to characters of various morphologies. Our method uses
  reinforcement learning to train a policy to control characters in a physics
  simulator. We only require human motion capture data for training, without
  relying on artist-generated animations for each avatar. This allows us to use
  large motion capture datasets to train general policies that can track unseen
  users from real and sparse data in real-time. We demonstrate the feasibility of
  our approach on three characters with different skeleton structure: a dinosaur,
  a mouse-like creature and a human. We show that the avatar poses often match
  the user surprisingly well, despite having no sensor information of the lower
  body available. We discuss and ablate the important components in our
  framework, specifically the kinematic retargeting step, the imitation, contact
  and action reward as well as our asymmetric actor-critic observations. We
  further explore the robustness of our method in a variety of settings including
  unbalancing, dancing and sports motions.
paper_authors._id: 64a64cc723caaba1d2e634bf
paper_authors.name: Daniele Reda
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-07T07:34:49.270Z
paper_authors._id: 64a7289fc34b44e2ceb1022b
paper_authors.avatarUrl: /avatars/cfe83f4fd4f7af71e95d7f4e9dc35963.svg
paper_authors.fullname: Daniele Reda
paper_authors.type: user
paper_authors.user: rdednl
paper_authors._id: 64a64cc723caaba1d2e634c0
paper_authors.name: Jungdam Won
paper_authors._id: 64a64cc723caaba1d2e634c1
paper_authors.name: Yuting Ye
paper_authors._id: 64a64cc723caaba1d2e634c2
paper_authors.name: Michiel van de Panne
paper_authors._id: 64a64cc723caaba1d2e634c3
paper_authors.name: Alexander Winkler
paper_title: Physics-based Motion Retargeting from Sparse Inputs
paper_summary: Avatars are important to create interactive and immersive experiences in
  virtual worlds. One challenge in animating these characters to mimic a user's
  motion is that commercial AR/VR products consist only of a headset and
  controllers, providing very limited sensor data of the user's pose. Another
  challenge is that an avatar might have a different skeleton structure than a
  human and the mapping between them is unclear. In this work we address both of
  these challenges. We introduce a method to retarget motions in real-time from
  sparse human sensor data to characters of various morphologies. Our method uses
  reinforcement learning to train a policy to control characters in a physics
  simulator. We only require human motion capture data for training, without
  relying on artist-generated animations for each avatar. This allows us to use
  large motion capture datasets to train general policies that can track unseen
  users from real and sparse data in real-time. We demonstrate the feasibility of
  our approach on three characters with different skeleton structure: a dinosaur,
  a mouse-like creature and a human. We show that the avatar poses often match
  the user surprisingly well, despite having no sensor information of the lower
  body available. We discuss and ablate the important components in our
  framework, specifically the kinematic retargeting step, the imitation, contact
  and action reward as well as our asymmetric actor-critic observations. We
  further explore the robustness of our method in a variety of settings including
  unbalancing, dancing and sports motions.
paper_ai_keywords: reinforcement learning policy physics simulator human motion capture data kinematic retargeting imitation reward contact reward action reward asymmetric actor-critic observations