HumanMM: Global Human Motion Recovery from Multi-shot Videos
In this paper, we present a novel framework designed to reconstruct
long-sequence 3D human motion in the world coordinates from in-the-wild videos
with multiple shot transitions. Such long-sequence in-the-wild motions are
highly valuable to applications such as motion generation and motion
understanding, but are of great challenge to be recovered due to abrupt shot
transitions, partial occlusions, and dynamic backgrounds presented in such
videos. Existing methods primarily focus on single-shot videos, where
continuity is maintained within a single camera view, or simplify multi-shot
alignment in camera space only. In this work, we tackle the challenges by
integrating an enhanced camera pose estimation with Human Motion Recovery (HMR)
by incorporating a shot transition detector and a robust alignment module for
accurate pose and orientation continuity across shots. By leveraging a custom
motion integrator, we effectively mitigate the problem of foot sliding and
ensure temporal consistency in human pose. Extensive evaluations on our created
multi-shot dataset from public 3D human datasets demonstrate the robustness of
our method in reconstructing realistic human motion in world coordinates.
67d029c7f8577b7e578cc5ff
Yuhong Zhang
admin_assigned
2025-03-11T14:24:04.517Z
65f8281209cf7381af189aa7
/avatars/fc7d570d1ff4df5b87b38a082b16f1ab.svg
Yuhong Zhang
user
YuhongZhang
67d029c7f8577b7e578cc600
Guanlin Wu
admin_assigned
2025-03-11T14:24:11.123Z
6523f611db79e539485521b1
/avatars/64aca195312084a98162b08152841429.svg
Guanlin WU
user
Frank0930
67d029c7f8577b7e578cc601
Ling-Hao Chen
admin_assigned
2025-03-11T14:24:17.437Z
63109a4d61cab0446e48c83b
https://cdn-avatars.huggingface.co/v1/production/uploads/63109a4d61cab0446e48c83b/JQlVkQp0ok586ND1GmB0w.png
Ling-Hao Chen
user
EvanTHU
67d029c7f8577b7e578cc602
Zhuokai Zhao
admin_assigned
2025-03-11T14:24:37.175Z
63af25605fe9db73f67a0fb7
https://cdn-avatars.huggingface.co/v1/production/uploads/63af25605fe9db73f67a0fb7/mcQHbTdJVxi2GdHzRB9ad.jpeg
Zhuokai Zhao
user
zhuokai
67d029c7f8577b7e578cc603
Jing Lin
67d029c7f8577b7e578cc604
Xiaoke Jiang
67d029c7f8577b7e578cc605
Jiamin Wu
admin_assigned
2025-03-11T14:24:52.948Z
61a48038cd3eb3043f38e402
/avatars/21f33c82fcbbf49434345d522c8865d9.svg
Jiamin Wu
user
Bunny
67d029c7f8577b7e578cc606
Zhuoheng Li
admin_assigned
2025-03-11T14:24:59.235Z
66d9e708dc8d21114944e4f8
/avatars/1f22d26eeb5b70d56a7031dca2d7b790.svg
Zhuoheng Li
user
Andy-LZH
67d029c7f8577b7e578cc607
Hao Frank Yang
67d029c7f8577b7e578cc608
Haoqian Wang
67d029c7f8577b7e578cc609
Lei Zhang
HumanMM: Global Human Motion Recovery from Multi-shot Videos
In this paper, we present a novel framework designed to reconstruct
long-sequence 3D human motion in the world coordinates from in-the-wild videos
with multiple shot transitions. Such long-sequence in-the-wild motions are
highly valuable to applications such as motion generation and motion
understanding, but are of great challenge to be recovered due to abrupt shot
transitions, partial occlusions, and dynamic backgrounds presented in such
videos. Existing methods primarily focus on single-shot videos, where
continuity is maintained within a single camera view, or simplify multi-shot
alignment in camera space only. In this work, we tackle the challenges by
integrating an enhanced camera pose estimation with Human Motion Recovery (HMR)
by incorporating a shot transition detector and a robust alignment module for
accurate pose and orientation continuity across shots. By leveraging a custom
motion integrator, we effectively mitigate the problem of foot sliding and
ensure temporal consistency in human pose. Extensive evaluations on our created
multi-shot dataset from public 3D human datasets demonstrate the robustness of
our method in reconstructing realistic human motion in world coordinates.
A framework for reconstructing long-sequence 3D human motion with multiple shot transitions by integrating enhanced camera pose estimation, Human Motion Recovery (HMR), shot transition detection, and a robust alignment module.
camera pose estimation Human Motion Recovery (HMR) shot transition detector robust alignment module multi-shot dataset foot sliding temporal consistency