DexTrack: Towards Generalizable Neural Tracking Control for Dexterous
  Manipulation from Human References
We address the challenge of developing a generalizable neural tracking
controller for dexterous manipulation from human references. This controller
aims to manage a dexterous robot hand to manipulate diverse objects for various
purposes defined by kinematic human-object interactions. Developing such a
controller is complicated by the intricate contact dynamics of dexterous
manipulation and the need for adaptivity, generalizability, and robustness.
Current reinforcement learning and trajectory optimization methods often fall
short due to their dependence on task-specific rewards or precise system
models. We introduce an approach that curates large-scale successful robot
tracking demonstrations, comprising pairs of human references and robot
actions, to train a neural controller. Utilizing a data flywheel, we
iteratively enhance the controller's performance, as well as the number and
quality of successful tracking demonstrations. We exploit available tracking
demonstrations and carefully integrate reinforcement learning and imitation
learning to boost the controller's performance in dynamic environments. At the
same time, to obtain high-quality tracking demonstrations, we individually
optimize per-trajectory tracking by leveraging the learned tracking controller
in a homotopy optimization method. The homotopy optimization, mimicking
chain-of-thought, aids in solving challenging trajectory tracking problems to
increase demonstration diversity. We showcase our success by training a
generalizable neural controller and evaluating it in both simulation and real
world. Our method achieves over a 10% improvement in success rates compared to
leading baselines. The project website with animated results is available at
https://meowuu7.github.io/DexTrack/.
67af107d6bd28b8bd4e13c38
Xueyi Liu
admin_assigned
2025-02-14T12:55:06.626Z
65b8070ad49f4330ab0ca5f7
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/t4fI-3djMfgXCchU_xpjL.png
Xueyi Liu
user
xymeow7
67af107d6bd28b8bd4e13c39
Jianibieke Adalibieke
67af107d6bd28b8bd4e13c3a
Qianwei Han
67af107d6bd28b8bd4e13c3b
Yuzhe Qin
67af107d6bd28b8bd4e13c3c
Li Yi
DexTrack: Towards Generalizable Neural Tracking Control for Dexterous
  Manipulation from Human References
We address the challenge of developing a generalizable neural tracking
controller for dexterous manipulation from human references. This controller
aims to manage a dexterous robot hand to manipulate diverse objects for various
purposes defined by kinematic human-object interactions. Developing such a
controller is complicated by the intricate contact dynamics of dexterous
manipulation and the need for adaptivity, generalizability, and robustness.
Current reinforcement learning and trajectory optimization methods often fall
short due to their dependence on task-specific rewards or precise system
models. We introduce an approach that curates large-scale successful robot
tracking demonstrations, comprising pairs of human references and robot
actions, to train a neural controller. Utilizing a data flywheel, we
iteratively enhance the controller's performance, as well as the number and
quality of successful tracking demonstrations. We exploit available tracking
demonstrations and carefully integrate reinforcement learning and imitation
learning to boost the controller's performance in dynamic environments. At the
same time, to obtain high-quality tracking demonstrations, we individually
optimize per-trajectory tracking by leveraging the learned tracking controller
in a homotopy optimization method. The homotopy optimization, mimicking
chain-of-thought, aids in solving challenging trajectory tracking problems to
increase demonstration diversity. We showcase our success by training a
generalizable neural controller and evaluating it in both simulation and real
world. Our method achieves over a 10% improvement in success rates compared to
leading baselines. The project website with animated results is available at
https://meowuu7.github.io/DexTrack/.
A neural tracking controller is developed using a combination of reinforcement learning and imitation learning, enhanced by homotopy optimization, to improve performance in dexterous manipulation tasks.
neural tracking controller dexterous manipulation kinematic human-object interactions reinforcement learning trajectory optimization imitation learning homotopy optimization