Real-time Monocular Full-body Capture in World Space via Sequential
  Proxy-to-Motion Learning
Learning-based approaches to monocular motion capture have recently shown
promising results by learning to regress in a data-driven manner. However, due
to the challenges in data collection and network designs, it remains
challenging for existing solutions to achieve real-time full-body capture while
being accurate in world space. In this work, we contribute a sequential
proxy-to-motion learning scheme together with a proxy dataset of 2D skeleton
sequences and 3D rotational motions in world space. Such proxy data enables us
to build a learning-based network with accurate full-body supervision while
also mitigating the generalization issues. For more accurate and physically
plausible predictions, a contact-aware neural motion descent module is proposed
in our network so that it can be aware of foot-ground contact and motion
misalignment with the proxy observations. Additionally, we share the body-hand
context information in our network for more compatible wrist poses recovery
with the full-body model. With the proposed learning-based solution, we
demonstrate the first real-time monocular full-body capture system with
plausible foot-ground contact in world space. More video results can be found
at our project page: https://liuyebin.com/proxycap.
64a3a8417068337139e56456
Yuxiang Zhang
claimed_verified
2023-07-04T19:16:22.065Z
64a3e2da18317078589a9313
/avatars/0dd2b80086871625ce8256cc23befafd.svg
Yuxiang Zhang
user
RainZhang
64a3a8417068337139e56457
Hongwen Zhang
admin_assigned
2023-07-04T09:13:07.490Z
638715a1aaee5cd69dc0c401
/avatars/f5976b3b02fe86191cb7f59c3dae102b.svg
Hongwen Zhang
user
HongwenZhang
64a3a8417068337139e56458
Liangxiao Hu
64a3a8417068337139e56459
Hongwei Yi
64a3a8417068337139e5645a
Shengping Zhang
64a3a8417068337139e5645b
Yebin Liu
Real-time Monocular Full-body Capture in World Space via Sequential
  Proxy-to-Motion Learning
Learning-based approaches to monocular motion capture have recently shown
promising results by learning to regress in a data-driven manner. However, due
to the challenges in data collection and network designs, it remains
challenging for existing solutions to achieve real-time full-body capture while
being accurate in world space. In this work, we contribute a sequential
proxy-to-motion learning scheme together with a proxy dataset of 2D skeleton
sequences and 3D rotational motions in world space. Such proxy data enables us
to build a learning-based network with accurate full-body supervision while
also mitigating the generalization issues. For more accurate and physically
plausible predictions, a contact-aware neural motion descent module is proposed
in our network so that it can be aware of foot-ground contact and motion
misalignment with the proxy observations. Additionally, we share the body-hand
context information in our network for more compatible wrist poses recovery
with the full-body model. With the proposed learning-based solution, we
demonstrate the first real-time monocular full-body capture system with
plausible foot-ground contact in world space. More video results can be found
at our project page: https://liuyebin.com/proxycap.
A learning-based system using sequential proxy-to-motion learning achieves real-time monocular full-body capture with accurate foot-ground contact and world space representation.
monocular motion capture learning-based approaches 2D skeleton sequences 3D rotational motions proxy dataset neural motion descent contact-aware wrist poses recovery plausible foot-ground contact