Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation
<|reference_start|>Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation: We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time whole-body teleoperation of a full-sized humanoid robot with only an RGB camera. To create a large-scale retargeted motion dataset of human movements for humanoid robots, we propose a scalable "sim-to-data" process to filter and pick feasible motions using a privileged motion imitator. Afterwards, we train a robust real-time humanoid motion imitator in simulation using these refined motions and transfer it to the real humanoid robot in a zero-shot manner. We successfully achieve teleoperation of dynamic whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc. To the best of our knowledge, this is the first demonstration to achieve learning-based real-time whole-body humanoid teleoperation.<|reference_end|>
@article{he2024learning,
    title={Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation},
    author={Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu
  Liu, Guanya Shi},
    journal={arXiv preprint arXiv:2403.04436},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.04436},
    primaryClass={cs.RO cs.AI cs.LG cs.SY eess.SY}
}