title: Real-time Monocular Full-body Capture in World Space via Sequential
    Proxy-to-Motion Learning
summary: Learning-based approaches to monocular motion capture have recently shown
  promising results by learning to regress in a data-driven manner. However, due
  to the challenges in data collection and network designs, it remains
  challenging for existing solutions to achieve real-time full-body capture while
  being accurate in world space. In this work, we contribute a sequential
  proxy-to-motion learning scheme together with a proxy dataset of 2D skeleton
  sequences and 3D rotational motions in world space. Such proxy data enables us
  to build a learning-based network with accurate full-body supervision while
  also mitigating the generalization issues. For more accurate and physically
  plausible predictions, a contact-aware neural motion descent module is proposed
  in our network so that it can be aware of foot-ground contact and motion
  misalignment with the proxy observations. Additionally, we share the body-hand
  context information in our network for more compatible wrist poses recovery
  with the full-body model. With the proposed learning-based solution, we
  demonstrate the first real-time monocular full-body capture system with
  plausible foot-ground contact in world space. More video results can be found
  at our project page: https://liuyebin.com/proxycap.
paper_authors._id: 64a3a8417068337139e56456
paper_authors.name: Yuxiang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-04T19:16:22.065Z
paper_authors._id: 64a3e2da18317078589a9313
paper_authors.avatarUrl: /avatars/0dd2b80086871625ce8256cc23befafd.svg
paper_authors.fullname: Yuxiang Zhang
paper_authors.type: user
paper_authors.user: RainZhang
paper_authors._id: 64a3a8417068337139e56457
paper_authors.name: Hongwen Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-04T09:13:07.490Z
paper_authors._id: 638715a1aaee5cd69dc0c401
paper_authors.avatarUrl: /avatars/f5976b3b02fe86191cb7f59c3dae102b.svg
paper_authors.fullname: Hongwen Zhang
paper_authors.type: user
paper_authors.user: HongwenZhang
paper_authors._id: 64a3a8417068337139e56458
paper_authors.name: Liangxiao Hu
paper_authors._id: 64a3a8417068337139e56459
paper_authors.name: Hongwei Yi
paper_authors._id: 64a3a8417068337139e5645a
paper_authors.name: Shengping Zhang
paper_authors._id: 64a3a8417068337139e5645b
paper_authors.name: Yebin Liu
paper_title: Real-time Monocular Full-body Capture in World Space via Sequential
    Proxy-to-Motion Learning
paper_summary: Learning-based approaches to monocular motion capture have recently shown
  promising results by learning to regress in a data-driven manner. However, due
  to the challenges in data collection and network designs, it remains
  challenging for existing solutions to achieve real-time full-body capture while
  being accurate in world space. In this work, we contribute a sequential
  proxy-to-motion learning scheme together with a proxy dataset of 2D skeleton
  sequences and 3D rotational motions in world space. Such proxy data enables us
  to build a learning-based network with accurate full-body supervision while
  also mitigating the generalization issues. For more accurate and physically
  plausible predictions, a contact-aware neural motion descent module is proposed
  in our network so that it can be aware of foot-ground contact and motion
  misalignment with the proxy observations. Additionally, we share the body-hand
  context information in our network for more compatible wrist poses recovery
  with the full-body model. With the proposed learning-based solution, we
  demonstrate the first real-time monocular full-body capture system with
  plausible foot-ground contact in world space. More video results can be found
  at our project page: https://liuyebin.com/proxycap.
paper_ai_summary: A learning-based system using sequential proxy-to-motion learning achieves real-time monocular full-body capture with accurate foot-ground contact and world space representation.
paper_ai_keywords: monocular motion capture learning-based approaches 2D skeleton sequences 3D rotational motions proxy dataset neural motion descent contact-aware wrist poses recovery plausible foot-ground contact