UniAff: A Unified Representation of Affordances for Tool Usage and
  Articulation with Vision-Language Models
Previous studies on robotic manipulation are based on a limited understanding
of the underlying 3D motion constraints and affordances. To address these
challenges, we propose a comprehensive paradigm, termed UniAff, that integrates
3D object-centric manipulation and task understanding in a unified formulation.
Specifically, we constructed a dataset labeled with manipulation-related key
attributes, comprising 900 articulated objects from 19 categories and 600 tools
from 12 categories. Furthermore, we leverage MLLMs to infer object-centric
representations for manipulation tasks, including affordance recognition and
reasoning about 3D motion constraints. Comprehensive experiments in both
simulation and real-world settings indicate that UniAff significantly improves
the generalization of robotic manipulation for tools and articulated objects.
We hope that UniAff will serve as a general baseline for unified robotic
manipulation tasks in the future. Images, videos, dataset, and code are
published on the project website at:https://sites.google.com/view/uni-aff/home
66fb9261a9312392f229f490
Qiaojun Yu
claimed_verified
2024-10-01T13:10:32.486Z
66211d1b679757f73639e4ed
/avatars/f084583c1f1ea207cd0185a89497736d.svg
 qiaojun yu 
user
qiaojunyu
66fb9261a9312392f229f491
Siyuan Huang
claimed_verified
2024-10-01T07:16:31.351Z
634e4120038b5879133552f5
/avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
Siyuan
user
SiyuanH
66fb9261a9312392f229f492
Xibin Yuan
66fb9261a9312392f229f493
Zhengkai Jiang
claimed_verified
2024-10-01T13:10:30.233Z
6371dda2ffc0489ed7d712c2
/avatars/dcb80cae00535c22af99baf86cb70d93.svg
zkjiang
user
justin-zk
66fb9261a9312392f229f494
Ce Hao
66fb9261a9312392f229f495
Xin Li
66fb9261a9312392f229f496
Haonan Chang
66fb9261a9312392f229f497
Junbo Wang
66fb9261a9312392f229f498
Liu Liu
66fb9261a9312392f229f499
Hongsheng Li
66fb9261a9312392f229f49a
Peng Gao
66fb9261a9312392f229f49b
Cewu Lu
UniAff: A Unified Representation of Affordances for Tool Usage and
  Articulation with Vision-Language Models
Previous studies on robotic manipulation are based on a limited understanding
of the underlying 3D motion constraints and affordances. To address these
challenges, we propose a comprehensive paradigm, termed UniAff, that integrates
3D object-centric manipulation and task understanding in a unified formulation.
Specifically, we constructed a dataset labeled with manipulation-related key
attributes, comprising 900 articulated objects from 19 categories and 600 tools
from 12 categories. Furthermore, we leverage MLLMs to infer object-centric
representations for manipulation tasks, including affordance recognition and
reasoning about 3D motion constraints. Comprehensive experiments in both
simulation and real-world settings indicate that UniAff significantly improves
the generalization of robotic manipulation for tools and articulated objects.
We hope that UniAff will serve as a general baseline for unified robotic
manipulation tasks in the future. Images, videos, dataset, and code are
published on the project website at:https://sites.google.com/view/uni-aff/home
UniAff integrates 3D object-centric manipulation and task understanding using MLLMs to improve robotic manipulation generalization for tools and articulated objects.
UniAff MLLMs affordance recognition 3D motion constraints