GRAPE: Generalizing Robot Policy via Preference Alignment
Despite the recent advancements of vision-language-action (VLA) models on a
variety of robotics tasks, they suffer from critical issues such as poor
generalizability to unseen tasks, due to their reliance on behavior cloning
exclusively from successful rollouts. Furthermore, they are typically
fine-tuned to replicate demonstrations collected by experts under different
settings, thus introducing distribution bias and limiting their adaptability to
diverse manipulation objectives, such as efficiency, safety, and task
completion. To bridge this gap, we introduce GRAPE: Generalizing Robot Policy
via Preference Alignment. Specifically, GRAPE aligns VLAs on a trajectory level
and implicitly models reward from both successful and failure trials to boost
generalizability to diverse tasks. Moreover, GRAPE breaks down complex
manipulation tasks to independent stages and automatically guides preference
modeling through customized spatiotemporal constraints with keypoints proposed
by a large vision-language model. Notably, these constraints are flexible and
can be customized to align the model with varying objectives, such as safety,
efficiency, or task success. We evaluate GRAPE across a diverse array of tasks
in both real-world and simulated environments. Experimental results demonstrate
that GRAPE enhances the performance of state-of-the-art VLA models, increasing
success rates on in-domain and unseen manipulation tasks by 51.79% and 60.36%,
respectively. Additionally, GRAPE can be aligned with various objectives, such
as safety and efficiency, reducing collision rates by 44.31% and rollout
step-length by 11.15%, respectively. All code, models, and data are available
at https://grape-vla.github.io/
674e322b6cc0873e3af7d1d9
Zijian Zhang
claimed_verified
2024-12-03T11:16:02.101Z
666fb5c44fa667819e2292e9
/avatars/272445a55c7576c0dad532985768f6a4.svg
Zijian Zhang
user
ZijianZhang
674e322b6cc0873e3af7d1da
Kaiyuan Zheng
claimed_verified
2024-12-04T09:46:34.175Z
6657a3d5ddc1b637c122208a
/avatars/55d768e035d8f3d9af0a94ba777d555d.svg
Kaiyuan Zheng
user
KyleZ0906
674e322b6cc0873e3af7d1db
Zhaorun Chen
claimed_verified
2024-12-03T11:15:58.265Z
64d448d3d095077728f7c740
https://cdn-avatars.huggingface.co/v1/production/uploads/64d448d3d095077728f7c740/4CFkiVoXXDsu7uQPTU0n3.jpeg
Zhaorun Chen
user
Zhaorun
674e322b6cc0873e3af7d1dc
Joel Jang
674e322b6cc0873e3af7d1dd
Yi Li
674e322b6cc0873e3af7d1de
Chaoqi Wang
674e322b6cc0873e3af7d1df
Mingyu Ding
674e322b6cc0873e3af7d1e0
Dieter Fox
674e322b6cc0873e3af7d1e1
Huaxiu Yao
GRAPE: Generalizing Robot Policy via Preference Alignment
Despite the recent advancements of vision-language-action (VLA) models on a
variety of robotics tasks, they suffer from critical issues such as poor
generalizability to unseen tasks, due to their reliance on behavior cloning
exclusively from successful rollouts. Furthermore, they are typically
fine-tuned to replicate demonstrations collected by experts under different
settings, thus introducing distribution bias and limiting their adaptability to
diverse manipulation objectives, such as efficiency, safety, and task
completion. To bridge this gap, we introduce GRAPE: Generalizing Robot Policy
via Preference Alignment. Specifically, GRAPE aligns VLAs on a trajectory level
and implicitly models reward from both successful and failure trials to boost
generalizability to diverse tasks. Moreover, GRAPE breaks down complex
manipulation tasks to independent stages and automatically guides preference
modeling through customized spatiotemporal constraints with keypoints proposed
by a large vision-language model. Notably, these constraints are flexible and
can be customized to align the model with varying objectives, such as safety,
efficiency, or task success. We evaluate GRAPE across a diverse array of tasks
in both real-world and simulated environments. Experimental results demonstrate
that GRAPE enhances the performance of state-of-the-art VLA models, increasing
success rates on in-domain and unseen manipulation tasks by 51.79% and 60.36%,
respectively. Additionally, GRAPE can be aligned with various objectives, such
as safety and efficiency, reducing collision rates by 44.31% and rollout
step-length by 11.15%, respectively. All code, models, and data are available
at https://grape-vla.github.io/
GRAPE improves vision-language-action models' performance by aligning policies via preference modeling, enhancing generalizability and allowing customization of objectives like safety and efficiency.
vision-language-action models GRAPE behavior cloning preference alignment trajectory level reward modeling spatiotemporal constraints keypoints constraint customization safety efficiency task success success rates collision rates rollout step-length