Affordance-based Robot Manipulation with Flow Matching
We present a framework for assistive robot manipulation, which focuses on two
fundamental challenges: first, efficiently adapting large-scale models to
downstream scene affordance understanding tasks, especially in daily living
scenarios where gathering multi-task data involving humans requires strenuous
effort; second, effectively learning robot trajectories by grounding the visual
affordance model. We tackle the first challenge by employing a
parameter-efficient prompt tuning method that prepends learnable text prompts
to the frozen vision model to predict manipulation affordances in multi-task
scenarios. Then we propose to learn robot trajectories guided by affordances in
a supervised Flow Matching method. Flow matching represents a robot visuomotor
policy as a conditional process of flowing random waypoints to desired robot
trajectories. Finally, we introduce a real-world dataset with 10 tasks across
Activities of Daily Living to test our framework. Our extensive evaluation
highlights that the proposed prompt tuning method for learning manipulation
affordance with language prompter achieves competitive performance and even
outperforms other finetuning protocols across data scales, while satisfying
parameter efficiency. Learning multi-task robot trajectories with a single flow
matching policy also leads to consistently better performance than alternative
behavior cloning methods, especially given multimodal robot action
distributions. Our framework seamlessly unifies affordance model learning and
trajectory generation with flow matching for robot manipulation.
66d802ec0d88366001bafcce
Fan Zhang
claimed_verified
2024-09-04T07:14:55.358Z
64e43a3f8b5060e515b490f9
/avatars/ff890611ee8f7870dc4ad487b6d7b85d.svg
Fan Zhang
user
Fanzhri
66d802ec0d88366001bafccf
Michael Gienger
Affordance-based Robot Manipulation with Flow Matching
We present a framework for assistive robot manipulation, which focuses on two
fundamental challenges: first, efficiently adapting large-scale models to
downstream scene affordance understanding tasks, especially in daily living
scenarios where gathering multi-task data involving humans requires strenuous
effort; second, effectively learning robot trajectories by grounding the visual
affordance model. We tackle the first challenge by employing a
parameter-efficient prompt tuning method that prepends learnable text prompts
to the frozen vision model to predict manipulation affordances in multi-task
scenarios. Then we propose to learn robot trajectories guided by affordances in
a supervised Flow Matching method. Flow matching represents a robot visuomotor
policy as a conditional process of flowing random waypoints to desired robot
trajectories. Finally, we introduce a real-world dataset with 10 tasks across
Activities of Daily Living to test our framework. Our extensive evaluation
highlights that the proposed prompt tuning method for learning manipulation
affordance with language prompter achieves competitive performance and even
outperforms other finetuning protocols across data scales, while satisfying
parameter efficiency. Learning multi-task robot trajectories with a single flow
matching policy also leads to consistently better performance than alternative
behavior cloning methods, especially given multimodal robot action
distributions. Our framework seamlessly unifies affordance model learning and
trajectory generation with flow matching for robot manipulation.
A framework for assistive robot manipulation uses parameter-efficient prompt tuning and supervised Flow Matching to learn manipulation affordances and robot trajectories in daily living scenarios.
parameter-efficient prompt tuning prompt tuning Flow Matching visuomotor policy manipulation affordances trajectory generation behavior cloning multimodal robot action distributions