CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive
  Prediction
In robotic visuomotor policy learning, diffusion-based models have achieved
significant success in improving the accuracy of action trajectory generation
compared to traditional autoregressive models. However, they suffer from
inefficiency due to multiple denoising steps and limited flexibility from
complex constraints. In this paper, we introduce Coarse-to-Fine AutoRegressive
Policy (CARP), a novel paradigm for visuomotor policy learning that redefines
the autoregressive action generation process as a coarse-to-fine, next-scale
approach. CARP decouples action generation into two stages: first, an action
autoencoder learns multi-scale representations of the entire action sequence;
then, a GPT-style transformer refines the sequence prediction through a
coarse-to-fine autoregressive process. This straightforward and intuitive
approach produces highly accurate and smooth actions, matching or even
surpassing the performance of diffusion-based policies while maintaining
efficiency on par with autoregressive policies. We conduct extensive
evaluations across diverse settings, including single-task and multi-task
scenarios on state-based and image-based simulation benchmarks, as well as
real-world tasks. CARP achieves competitive success rates, with up to a 10%
improvement, and delivers 10x faster inference compared to state-of-the-art
policies, establishing a high-performance, efficient, and flexible paradigm for
action generation in robotic tasks.
6757e0d412d7d9e5dcffaf95
Zhefei Gong
admin_assigned
2024-12-10T13:08:37.754Z
64cc96cb275c76304617ba4e
https://cdn-avatars.huggingface.co/v1/production/uploads/64cc96cb275c76304617ba4e/6-z5s0mBrbu3TYcebXou5.png
Jeffrey
user
zhefeigong
6757e0d412d7d9e5dcffaf96
Pengxiang Ding
6757e0d412d7d9e5dcffaf97
Shangke Lyu
6757e0d412d7d9e5dcffaf98
Siteng Huang
admin_assigned
2024-12-10T13:09:32.865Z
65fd82762bf2cd20ddaa193f
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yBYbWp_mT7UusYdkqtAvw.png
Siteng Huang
user
huangsiteng
6757e0d412d7d9e5dcffaf99
Mingyang Sun
6757e0d412d7d9e5dcffaf9a
Wei Zhao
6757e0d412d7d9e5dcffaf9b
Zhaoxin Fan
admin_assigned
2024-12-10T13:10:01.836Z
6462eab02538819c72a0b2cc
/avatars/b4ece21ce35da7506db9d8bef4abcf68.svg
Zhaoxin Fan
user
Jason007x
6757e0d412d7d9e5dcffaf9c
Donglin Wang
CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive
  Prediction
In robotic visuomotor policy learning, diffusion-based models have achieved
significant success in improving the accuracy of action trajectory generation
compared to traditional autoregressive models. However, they suffer from
inefficiency due to multiple denoising steps and limited flexibility from
complex constraints. In this paper, we introduce Coarse-to-Fine AutoRegressive
Policy (CARP), a novel paradigm for visuomotor policy learning that redefines
the autoregressive action generation process as a coarse-to-fine, next-scale
approach. CARP decouples action generation into two stages: first, an action
autoencoder learns multi-scale representations of the entire action sequence;
then, a GPT-style transformer refines the sequence prediction through a
coarse-to-fine autoregressive process. This straightforward and intuitive
approach produces highly accurate and smooth actions, matching or even
surpassing the performance of diffusion-based policies while maintaining
efficiency on par with autoregressive policies. We conduct extensive
evaluations across diverse settings, including single-task and multi-task
scenarios on state-based and image-based simulation benchmarks, as well as
real-world tasks. CARP achieves competitive success rates, with up to a 10%
improvement, and delivers 10x faster inference compared to state-of-the-art
policies, establishing a high-performance, efficient, and flexible paradigm for
action generation in robotic tasks.
CARP, a coarse-to-fine autoregressive policy, improves action generation efficiency and accuracy in robotic tasks by using an action autoencoder and GPT-style transformer, outperforming diffusion-based and traditional policies.
diffusion-based models autoregressive models action trajectory generation action autoencoder GPT-style transformer coarse-to-fine multi-scale representations single-task multi-task state-based image-based simulation benchmarks real-world tasks