Learning from Massive Human Videos for Universal Humanoid Pose Control
Scalable learning of humanoid robots is crucial for their deployment in
real-world applications. While traditional approaches primarily rely on
reinforcement learning or teleoperation to achieve whole-body control, they are
often limited by the diversity of simulated environments and the high costs of
demonstration collection. In contrast, human videos are ubiquitous and present
an untapped source of semantic and motion information that could significantly
enhance the generalization capabilities of humanoid robots. This paper
introduces Humanoid-X, a large-scale dataset of over 20 million humanoid robot
poses with corresponding text-based motion descriptions, designed to leverage
this abundant data. Humanoid-X is curated through a comprehensive pipeline:
data mining from the Internet, video caption generation, motion retargeting of
humans to humanoid robots, and policy learning for real-world deployment. With
Humanoid-X, we further train a large humanoid model, UH-1, which takes text
instructions as input and outputs corresponding actions to control a humanoid
robot. Extensive simulated and real-world experiments validate that our
scalable training approach leads to superior generalization in text-based
humanoid control, marking a significant step toward adaptable, real-world-ready
humanoid robots.
6763c8d3ad38702e6c9c1253
Jiageng Mao
6763c8d3ad38702e6c9c1254
Siheng Zhao
admin_assigned
2024-12-20T09:54:30.187Z
642e76a8113b200fd767b3c9
https://cdn-avatars.huggingface.co/v1/production/uploads/642e76a8113b200fd767b3c9/03_CAAr6gy7H_F_iUWx76.png
Siheng Zhao
user
OliverZhao
6763c8d3ad38702e6c9c1255
Siqi Song
6763c8d3ad38702e6c9c1256
Tianheng Shi
6763c8d3ad38702e6c9c1257
Junjie Ye
admin_assigned
2024-12-20T09:55:50.488Z
64b4e7890eb87fa990a29c0b
/avatars/ee49843e77a7671eb171d640944b5ac5.svg
Junjie Ye
user
Jay-Ye
6763c8d3ad38702e6c9c1258
Mingtong Zhang
admin_assigned
2024-12-20T09:56:11.489Z
671c6a3e255aa50ebb504fc5
/avatars/b549bb19990ae21690472799817f951e.svg
Mingtong Zhang
user
Mingtongz
6763c8d3ad38702e6c9c1259
Haoran Geng
admin_assigned
2024-12-20T09:56:18.462Z
651a752df7724d9aee9fb970
/avatars/956dc2a6fabf10bea15994152c04badc.svg
Haoran Geng
user
HaoranGeng
6763c8d3ad38702e6c9c125a
Jitendra Malik
admin_assigned
2024-12-20T09:56:24.513Z
65369a95605a07338de78ab0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
Jitendra Malik 
user
jitendra1995
6763c8d3ad38702e6c9c125b
Vitor Guizilini
6763c8d3ad38702e6c9c125c
Yue Wang
Learning from Massive Human Videos for Universal Humanoid Pose Control
Scalable learning of humanoid robots is crucial for their deployment in
real-world applications. While traditional approaches primarily rely on
reinforcement learning or teleoperation to achieve whole-body control, they are
often limited by the diversity of simulated environments and the high costs of
demonstration collection. In contrast, human videos are ubiquitous and present
an untapped source of semantic and motion information that could significantly
enhance the generalization capabilities of humanoid robots. This paper
introduces Humanoid-X, a large-scale dataset of over 20 million humanoid robot
poses with corresponding text-based motion descriptions, designed to leverage
this abundant data. Humanoid-X is curated through a comprehensive pipeline:
data mining from the Internet, video caption generation, motion retargeting of
humans to humanoid robots, and policy learning for real-world deployment. With
Humanoid-X, we further train a large humanoid model, UH-1, which takes text
instructions as input and outputs corresponding actions to control a humanoid
robot. Extensive simulated and real-world experiments validate that our
scalable training approach leads to superior generalization in text-based
humanoid control, marking a significant step toward adaptable, real-world-ready
humanoid robots.
Humanoid-X, a large dataset with text-based motion descriptions, is used to enhance a humanoid model's ability to generalize in text-based control tasks, improving real-world deployment.
humanoid robots reinforcement learning teleoperation simulated environments human videos semantic information motion information generalization capabilities large-scale dataset video caption generation motion retargeting policy learning real-world deployment text-based motion descriptions large humanoid model text instructions superior generalization text-based humanoid control