TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach
<|reference_start|>TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach: The tie-knotting task is highly challenging due to the tie's high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie's meshes from the demonstration video. With these estimated meshes used as subgoals, we first learn a teacher policy using privileged information. Then, we learn a student policy with point cloud observation by imitating teacher policy. Lastly, our pipeline learns a residual policy when the learned policy is applied to real-world execution, mitigating the Sim2Real gap. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, a dual-arm robot successfully knots a tie, achieving 50% success rate among 10 trials. Videos can be found https://tiebots.github.io/.<|reference_end|>
@article{peng2024tiebot:,
    title={TieBot: Learning to Knot a Tie from Visual Demonstration through a
  Real-to-Sim-to-Real Approach},
    author={Weikun Peng, Jun Lv, Yuwei Zeng, Haonan Chen, Siheng Zhao, Jichen Sun,
  Cewu Lu, Lin Shao},
    journal={arXiv preprint arXiv:2407.03245},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.03245},
    primaryClass={cs.RO cs.AI cs.SY eess.SY}
}