HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers
<|reference_start|>HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers: Large Vision-Language-Action (VLA) models, leveraging powerful pre trained Vision-Language Models (VLMs) backends, have shown promise in robotic control due to their impressive generalization ability. However, the success comes at a cost. Their reliance on VLM backends with billions of parameters leads to high computational costs and inference latency, limiting the testing scenarios to mainly quasi-static tasks and hindering performance in dynamic tasks requiring rapid interactions. To address these limitations, this paper proposes HiRT, a Hierarchical Robot Transformer framework that enables flexible frequency and performance trade-off. HiRT keeps VLMs running at low frequencies to capture temporarily invariant features while enabling real-time interaction through a high-frequency vision-based policy guided by the slowly updated features. Experiment results in both simulation and real-world settings demonstrate significant improvements over baseline methods. Empirically, in static tasks, we double the control frequency and achieve comparable success rates. Additionally, on novel real-world dynamic ma nipulation tasks which are challenging for previous VLA models, HiRT improves the success rate from 48% to 75%.<|reference_end|>
@article{zhang2024hirt:,
    title={HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers},
    author={Jianke Zhang, Yanjiang Guo, Xiaoyu Chen, Yen-Jen Wang, Yucheng Hu,
  Chengming Shi, Jianyu Chen},
    journal={CoRL2024},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.05273},
    primaryClass={cs.CV cs.AI cs.RO}
}