HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations
<|reference_start|>HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations: It is especially challenging to achieve real-time human motion tracking on a standalone VR Head-Mounted Display (HMD) such as Meta Quest and PICO. In this paper, we propose HMD-Poser, the first unified approach to recover full-body motions using scalable sparse observations from HMD and body-worn IMUs. In particular, it can support a variety of input scenarios, such as HMD, HMD+2IMUs, HMD+3IMUs, etc. The scalability of inputs may accommodate users' choices for both high tracking accuracy and easy-to-wear. A lightweight temporal-spatial feature learning network is proposed in HMD-Poser to guarantee that the model runs in real-time on HMDs. Furthermore, HMD-Poser presents online body shape estimation to improve the position accuracy of body joints. Extensive experimental results on the challenging AMASS dataset show that HMD-Poser achieves new state-of-the-art results in both accuracy and real-time performance. We also build a new free-dancing motion dataset to evaluate HMD-Poser's on-device performance and investigate the performance gap between synthetic data and real-captured sensor data. Finally, we demonstrate our HMD-Poser with a real-time Avatar-driving application on a commercial HMD. Our code and free-dancing motion dataset are available https://pico-ai-team.github.io/hmd-poser<|reference_end|>
@article{dai2024hmd-poser:,
    title={HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable
  Sparse Observations},
    author={Peng Dai, Yang Zhang, Tao Liu, Zhen Fan, Tianyuan Du, Zhuo Su,
  Xiaozheng Zheng, Zeming Li},
    journal={arXiv preprint arXiv:2403.03561},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.03561},
    primaryClass={cs.CV}
}