title: Learning from Massive Human Videos for Universal Humanoid Pose Control
summary: Scalable learning of humanoid robots is crucial for their deployment in
  real-world applications. While traditional approaches primarily rely on
  reinforcement learning or teleoperation to achieve whole-body control, they are
  often limited by the diversity of simulated environments and the high costs of
  demonstration collection. In contrast, human videos are ubiquitous and present
  an untapped source of semantic and motion information that could significantly
  enhance the generalization capabilities of humanoid robots. This paper
  introduces Humanoid-X, a large-scale dataset of over 20 million humanoid robot
  poses with corresponding text-based motion descriptions, designed to leverage
  this abundant data. Humanoid-X is curated through a comprehensive pipeline:
  data mining from the Internet, video caption generation, motion retargeting of
  humans to humanoid robots, and policy learning for real-world deployment. With
  Humanoid-X, we further train a large humanoid model, UH-1, which takes text
  instructions as input and outputs corresponding actions to control a humanoid
  robot. Extensive simulated and real-world experiments validate that our
  scalable training approach leads to superior generalization in text-based
  humanoid control, marking a significant step toward adaptable, real-world-ready
  humanoid robots.
paper_authors._id: 6763c8d3ad38702e6c9c1253
paper_authors.name: Jiageng Mao
paper_authors._id: 6763c8d3ad38702e6c9c1254
paper_authors.name: Siheng Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:54:30.187Z
paper_authors._id: 642e76a8113b200fd767b3c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642e76a8113b200fd767b3c9/03_CAAr6gy7H_F_iUWx76.png
paper_authors.fullname: Siheng Zhao
paper_authors.type: user
paper_authors.user: OliverZhao
paper_authors._id: 6763c8d3ad38702e6c9c1255
paper_authors.name: Siqi Song
paper_authors._id: 6763c8d3ad38702e6c9c1256
paper_authors.name: Tianheng Shi
paper_authors._id: 6763c8d3ad38702e6c9c1257
paper_authors.name: Junjie Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:55:50.488Z
paper_authors._id: 64b4e7890eb87fa990a29c0b
paper_authors.avatarUrl: /avatars/ee49843e77a7671eb171d640944b5ac5.svg
paper_authors.fullname: Junjie Ye
paper_authors.type: user
paper_authors.user: Jay-Ye
paper_authors._id: 6763c8d3ad38702e6c9c1258
paper_authors.name: Mingtong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:56:11.489Z
paper_authors._id: 671c6a3e255aa50ebb504fc5
paper_authors.avatarUrl: /avatars/b549bb19990ae21690472799817f951e.svg
paper_authors.fullname: Mingtong Zhang
paper_authors.type: user
paper_authors.user: Mingtongz
paper_authors._id: 6763c8d3ad38702e6c9c1259
paper_authors.name: Haoran Geng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:56:18.462Z
paper_authors._id: 651a752df7724d9aee9fb970
paper_authors.avatarUrl: /avatars/956dc2a6fabf10bea15994152c04badc.svg
paper_authors.fullname: Haoran Geng
paper_authors.type: user
paper_authors.user: HaoranGeng
paper_authors._id: 6763c8d3ad38702e6c9c125a
paper_authors.name: Jitendra Malik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:56:24.513Z
paper_authors._id: 65369a95605a07338de78ab0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
paper_authors.fullname: Jitendra Malik 
paper_authors.type: user
paper_authors.user: jitendra1995
paper_authors._id: 6763c8d3ad38702e6c9c125b
paper_authors.name: Vitor Guizilini
paper_authors._id: 6763c8d3ad38702e6c9c125c
paper_authors.name: Yue Wang
paper_title: Learning from Massive Human Videos for Universal Humanoid Pose Control
paper_summary: Scalable learning of humanoid robots is crucial for their deployment in
  real-world applications. While traditional approaches primarily rely on
  reinforcement learning or teleoperation to achieve whole-body control, they are
  often limited by the diversity of simulated environments and the high costs of
  demonstration collection. In contrast, human videos are ubiquitous and present
  an untapped source of semantic and motion information that could significantly
  enhance the generalization capabilities of humanoid robots. This paper
  introduces Humanoid-X, a large-scale dataset of over 20 million humanoid robot
  poses with corresponding text-based motion descriptions, designed to leverage
  this abundant data. Humanoid-X is curated through a comprehensive pipeline:
  data mining from the Internet, video caption generation, motion retargeting of
  humans to humanoid robots, and policy learning for real-world deployment. With
  Humanoid-X, we further train a large humanoid model, UH-1, which takes text
  instructions as input and outputs corresponding actions to control a humanoid
  robot. Extensive simulated and real-world experiments validate that our
  scalable training approach leads to superior generalization in text-based
  humanoid control, marking a significant step toward adaptable, real-world-ready
  humanoid robots.
paper_ai_keywords: reinforcement learning teleoperation simulated environments demonstration collection human videos semantic information motion information generalization capabilities Humanoid-X large-scale dataset humanoid robot poses text-based motion descriptions data mining video caption generation motion retargeting policy learning scalable training approach text-based humanoid control adaptable humanoid robots