title: GRAPE: Generalizing Robot Policy via Preference Alignment
summary: Despite the recent advancements of vision-language-action (VLA) models on a
  variety of robotics tasks, they suffer from critical issues such as poor
  generalizability to unseen tasks, due to their reliance on behavior cloning
  exclusively from successful rollouts. Furthermore, they are typically
  fine-tuned to replicate demonstrations collected by experts under different
  settings, thus introducing distribution bias and limiting their adaptability to
  diverse manipulation objectives, such as efficiency, safety, and task
  completion. To bridge this gap, we introduce GRAPE: Generalizing Robot Policy
  via Preference Alignment. Specifically, GRAPE aligns VLAs on a trajectory level
  and implicitly models reward from both successful and failure trials to boost
  generalizability to diverse tasks. Moreover, GRAPE breaks down complex
  manipulation tasks to independent stages and automatically guides preference
  modeling through customized spatiotemporal constraints with keypoints proposed
  by a large vision-language model. Notably, these constraints are flexible and
  can be customized to align the model with varying objectives, such as safety,
  efficiency, or task success. We evaluate GRAPE across a diverse array of tasks
  in both real-world and simulated environments. Experimental results demonstrate
  that GRAPE enhances the performance of state-of-the-art VLA models, increasing
  success rates on in-domain and unseen manipulation tasks by 51.79% and 60.36%,
  respectively. Additionally, GRAPE can be aligned with various objectives, such
  as safety and efficiency, reducing collision rates by 44.31% and rollout
  step-length by 11.15%, respectively. All code, models, and data are available
  at https://grape-vla.github.io/
paper_authors._id: 674e322b6cc0873e3af7d1d9
paper_authors.name: Zijian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:16:02.101Z
paper_authors._id: 666fb5c44fa667819e2292e9
paper_authors.avatarUrl: /avatars/272445a55c7576c0dad532985768f6a4.svg
paper_authors.fullname: Zijian Zhang
paper_authors.type: user
paper_authors.user: ZijianZhang
paper_authors._id: 674e322b6cc0873e3af7d1da
paper_authors.name: Kaiyuan Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:46:34.175Z
paper_authors._id: 6657a3d5ddc1b637c122208a
paper_authors.avatarUrl: /avatars/55d768e035d8f3d9af0a94ba777d555d.svg
paper_authors.fullname: Kaiyuan Zheng
paper_authors.type: user
paper_authors.user: KyleZ0906
paper_authors._id: 674e322b6cc0873e3af7d1db
paper_authors.name: Zhaorun Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:15:58.265Z
paper_authors._id: 64d448d3d095077728f7c740
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d448d3d095077728f7c740/4CFkiVoXXDsu7uQPTU0n3.jpeg
paper_authors.fullname: Zhaorun Chen
paper_authors.type: user
paper_authors.user: Zhaorun
paper_authors._id: 674e322b6cc0873e3af7d1dc
paper_authors.name: Joel Jang
paper_authors._id: 674e322b6cc0873e3af7d1dd
paper_authors.name: Yi Li
paper_authors._id: 674e322b6cc0873e3af7d1de
paper_authors.name: Chaoqi Wang
paper_authors._id: 674e322b6cc0873e3af7d1df
paper_authors.name: Mingyu Ding
paper_authors._id: 674e322b6cc0873e3af7d1e0
paper_authors.name: Dieter Fox
paper_authors._id: 674e322b6cc0873e3af7d1e1
paper_authors.name: Huaxiu Yao
paper_title: GRAPE: Generalizing Robot Policy via Preference Alignment
paper_summary: Despite the recent advancements of vision-language-action (VLA) models on a
  variety of robotics tasks, they suffer from critical issues such as poor
  generalizability to unseen tasks, due to their reliance on behavior cloning
  exclusively from successful rollouts. Furthermore, they are typically
  fine-tuned to replicate demonstrations collected by experts under different
  settings, thus introducing distribution bias and limiting their adaptability to
  diverse manipulation objectives, such as efficiency, safety, and task
  completion. To bridge this gap, we introduce GRAPE: Generalizing Robot Policy
  via Preference Alignment. Specifically, GRAPE aligns VLAs on a trajectory level
  and implicitly models reward from both successful and failure trials to boost
  generalizability to diverse tasks. Moreover, GRAPE breaks down complex
  manipulation tasks to independent stages and automatically guides preference
  modeling through customized spatiotemporal constraints with keypoints proposed
  by a large vision-language model. Notably, these constraints are flexible and
  can be customized to align the model with varying objectives, such as safety,
  efficiency, or task success. We evaluate GRAPE across a diverse array of tasks
  in both real-world and simulated environments. Experimental results demonstrate
  that GRAPE enhances the performance of state-of-the-art VLA models, increasing
  success rates on in-domain and unseen manipulation tasks by 51.79% and 60.36%,
  respectively. Additionally, GRAPE can be aligned with various objectives, such
  as safety and efficiency, reducing collision rates by 44.31% and rollout
  step-length by 11.15%, respectively. All code, models, and data are available
  at https://grape-vla.github.io/
paper_ai_summary: GRAPE improves vision-language-action models' performance by aligning policies via preference modeling, enhancing generalizability and allowing customization of objectives like safety and efficiency.
paper_ai_keywords: vision-language-action models GRAPE behavior cloning preference alignment trajectory level reward modeling spatiotemporal constraints keypoints constraint customization safety efficiency task success success rates collision rates rollout step-length