title: DexTrack: Towards Generalizable Neural Tracking Control for Dexterous
    Manipulation from Human References
summary: We address the challenge of developing a generalizable neural tracking
  controller for dexterous manipulation from human references. This controller
  aims to manage a dexterous robot hand to manipulate diverse objects for various
  purposes defined by kinematic human-object interactions. Developing such a
  controller is complicated by the intricate contact dynamics of dexterous
  manipulation and the need for adaptivity, generalizability, and robustness.
  Current reinforcement learning and trajectory optimization methods often fall
  short due to their dependence on task-specific rewards or precise system
  models. We introduce an approach that curates large-scale successful robot
  tracking demonstrations, comprising pairs of human references and robot
  actions, to train a neural controller. Utilizing a data flywheel, we
  iteratively enhance the controller's performance, as well as the number and
  quality of successful tracking demonstrations. We exploit available tracking
  demonstrations and carefully integrate reinforcement learning and imitation
  learning to boost the controller's performance in dynamic environments. At the
  same time, to obtain high-quality tracking demonstrations, we individually
  optimize per-trajectory tracking by leveraging the learned tracking controller
  in a homotopy optimization method. The homotopy optimization, mimicking
  chain-of-thought, aids in solving challenging trajectory tracking problems to
  increase demonstration diversity. We showcase our success by training a
  generalizable neural controller and evaluating it in both simulation and real
  world. Our method achieves over a 10% improvement in success rates compared to
  leading baselines. The project website with animated results is available at
  https://meowuu7.github.io/DexTrack/.
paper_authors._id: 67af107d6bd28b8bd4e13c38
paper_authors.name: Xueyi Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:55:06.626Z
paper_authors._id: 65b8070ad49f4330ab0ca5f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/t4fI-3djMfgXCchU_xpjL.png
paper_authors.fullname: Xueyi Liu
paper_authors.type: user
paper_authors.user: xymeow7
paper_authors._id: 67af107d6bd28b8bd4e13c39
paper_authors.name: Jianibieke Adalibieke
paper_authors._id: 67af107d6bd28b8bd4e13c3a
paper_authors.name: Qianwei Han
paper_authors._id: 67af107d6bd28b8bd4e13c3b
paper_authors.name: Yuzhe Qin
paper_authors._id: 67af107d6bd28b8bd4e13c3c
paper_authors.name: Li Yi
paper_title: DexTrack: Towards Generalizable Neural Tracking Control for Dexterous
    Manipulation from Human References
paper_summary: We address the challenge of developing a generalizable neural tracking
  controller for dexterous manipulation from human references. This controller
  aims to manage a dexterous robot hand to manipulate diverse objects for various
  purposes defined by kinematic human-object interactions. Developing such a
  controller is complicated by the intricate contact dynamics of dexterous
  manipulation and the need for adaptivity, generalizability, and robustness.
  Current reinforcement learning and trajectory optimization methods often fall
  short due to their dependence on task-specific rewards or precise system
  models. We introduce an approach that curates large-scale successful robot
  tracking demonstrations, comprising pairs of human references and robot
  actions, to train a neural controller. Utilizing a data flywheel, we
  iteratively enhance the controller's performance, as well as the number and
  quality of successful tracking demonstrations. We exploit available tracking
  demonstrations and carefully integrate reinforcement learning and imitation
  learning to boost the controller's performance in dynamic environments. At the
  same time, to obtain high-quality tracking demonstrations, we individually
  optimize per-trajectory tracking by leveraging the learned tracking controller
  in a homotopy optimization method. The homotopy optimization, mimicking
  chain-of-thought, aids in solving challenging trajectory tracking problems to
  increase demonstration diversity. We showcase our success by training a
  generalizable neural controller and evaluating it in both simulation and real
  world. Our method achieves over a 10% improvement in success rates compared to
  leading baselines. The project website with animated results is available at
  https://meowuu7.github.io/DexTrack/.
paper_ai_summary: A neural tracking controller is developed using a combination of reinforcement learning and imitation learning, enhanced by homotopy optimization, to improve performance in dexterous manipulation tasks.
paper_ai_keywords: neural tracking controller dexterous manipulation kinematic human-object interactions reinforcement learning trajectory optimization imitation learning homotopy optimization