title: Affordance-based Robot Manipulation with Flow Matching
summary: We present a framework for assistive robot manipulation, which focuses on two
  fundamental challenges: first, efficiently adapting large-scale models to
  downstream scene affordance understanding tasks, especially in daily living
  scenarios where gathering multi-task data involving humans requires strenuous
  effort; second, effectively learning robot trajectories by grounding the visual
  affordance model. We tackle the first challenge by employing a
  parameter-efficient prompt tuning method that prepends learnable text prompts
  to the frozen vision model to predict manipulation affordances in multi-task
  scenarios. Then we propose to learn robot trajectories guided by affordances in
  a supervised Flow Matching method. Flow matching represents a robot visuomotor
  policy as a conditional process of flowing random waypoints to desired robot
  trajectories. Finally, we introduce a real-world dataset with 10 tasks across
  Activities of Daily Living to test our framework. Our extensive evaluation
  highlights that the proposed prompt tuning method for learning manipulation
  affordance with language prompter achieves competitive performance and even
  outperforms other finetuning protocols across data scales, while satisfying
  parameter efficiency. Learning multi-task robot trajectories with a single flow
  matching policy also leads to consistently better performance than alternative
  behavior cloning methods, especially given multimodal robot action
  distributions. Our framework seamlessly unifies affordance model learning and
  trajectory generation with flow matching for robot manipulation.
paper_authors._id: 66d802ec0d88366001bafcce
paper_authors.name: Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T07:14:55.358Z
paper_authors._id: 64e43a3f8b5060e515b490f9
paper_authors.avatarUrl: /avatars/ff890611ee8f7870dc4ad487b6d7b85d.svg
paper_authors.fullname: Fan Zhang
paper_authors.type: user
paper_authors.user: Fanzhri
paper_authors._id: 66d802ec0d88366001bafccf
paper_authors.name: Michael Gienger
paper_title: Affordance-based Robot Manipulation with Flow Matching
paper_summary: We present a framework for assistive robot manipulation, which focuses on two
  fundamental challenges: first, efficiently adapting large-scale models to
  downstream scene affordance understanding tasks, especially in daily living
  scenarios where gathering multi-task data involving humans requires strenuous
  effort; second, effectively learning robot trajectories by grounding the visual
  affordance model. We tackle the first challenge by employing a
  parameter-efficient prompt tuning method that prepends learnable text prompts
  to the frozen vision model to predict manipulation affordances in multi-task
  scenarios. Then we propose to learn robot trajectories guided by affordances in
  a supervised Flow Matching method. Flow matching represents a robot visuomotor
  policy as a conditional process of flowing random waypoints to desired robot
  trajectories. Finally, we introduce a real-world dataset with 10 tasks across
  Activities of Daily Living to test our framework. Our extensive evaluation
  highlights that the proposed prompt tuning method for learning manipulation
  affordance with language prompter achieves competitive performance and even
  outperforms other finetuning protocols across data scales, while satisfying
  parameter efficiency. Learning multi-task robot trajectories with a single flow
  matching policy also leads to consistently better performance than alternative
  behavior cloning methods, especially given multimodal robot action
  distributions. Our framework seamlessly unifies affordance model learning and
  trajectory generation with flow matching for robot manipulation.
paper_ai_keywords: parameter-efficient prompt tuning prompt tuning method frozen vision model manipulation affordances multi-task scenarios supervised Flow Matching flow matching visuomotor policy random waypoints desired robot trajectories Activities of Daily Living behavior cloning multimodal robot action distributions