title: UniAff: A Unified Representation of Affordances for Tool Usage and
    Articulation with Vision-Language Models
summary: Previous studies on robotic manipulation are based on a limited understanding
  of the underlying 3D motion constraints and affordances. To address these
  challenges, we propose a comprehensive paradigm, termed UniAff, that integrates
  3D object-centric manipulation and task understanding in a unified formulation.
  Specifically, we constructed a dataset labeled with manipulation-related key
  attributes, comprising 900 articulated objects from 19 categories and 600 tools
  from 12 categories. Furthermore, we leverage MLLMs to infer object-centric
  representations for manipulation tasks, including affordance recognition and
  reasoning about 3D motion constraints. Comprehensive experiments in both
  simulation and real-world settings indicate that UniAff significantly improves
  the generalization of robotic manipulation for tools and articulated objects.
  We hope that UniAff will serve as a general baseline for unified robotic
  manipulation tasks in the future. Images, videos, dataset, and code are
  published on the project website at:https://sites.google.com/view/uni-aff/home
paper_authors._id: 66fb9261a9312392f229f490
paper_authors.name: Qiaojun Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-01T13:10:32.486Z
paper_authors._id: 66211d1b679757f73639e4ed
paper_authors.avatarUrl: /avatars/f084583c1f1ea207cd0185a89497736d.svg
paper_authors.fullname:  qiaojun yu 
paper_authors.type: user
paper_authors.user: qiaojunyu
paper_authors._id: 66fb9261a9312392f229f491
paper_authors.name: Siyuan Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-01T07:16:31.351Z
paper_authors._id: 634e4120038b5879133552f5
paper_authors.avatarUrl: /avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
paper_authors.fullname: Siyuan
paper_authors.type: user
paper_authors.user: SiyuanH
paper_authors._id: 66fb9261a9312392f229f492
paper_authors.name: Xibin Yuan
paper_authors._id: 66fb9261a9312392f229f493
paper_authors.name: Zhengkai Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-01T13:10:30.233Z
paper_authors._id: 6371dda2ffc0489ed7d712c2
paper_authors.avatarUrl: /avatars/dcb80cae00535c22af99baf86cb70d93.svg
paper_authors.fullname: zkjiang
paper_authors.type: user
paper_authors.user: justin-zk
paper_authors._id: 66fb9261a9312392f229f494
paper_authors.name: Ce Hao
paper_authors._id: 66fb9261a9312392f229f495
paper_authors.name: Xin Li
paper_authors._id: 66fb9261a9312392f229f496
paper_authors.name: Haonan Chang
paper_authors._id: 66fb9261a9312392f229f497
paper_authors.name: Junbo Wang
paper_authors._id: 66fb9261a9312392f229f498
paper_authors.name: Liu Liu
paper_authors._id: 66fb9261a9312392f229f499
paper_authors.name: Hongsheng Li
paper_authors._id: 66fb9261a9312392f229f49a
paper_authors.name: Peng Gao
paper_authors._id: 66fb9261a9312392f229f49b
paper_authors.name: Cewu Lu
paper_title: UniAff: A Unified Representation of Affordances for Tool Usage and
    Articulation with Vision-Language Models
paper_summary: Previous studies on robotic manipulation are based on a limited understanding
  of the underlying 3D motion constraints and affordances. To address these
  challenges, we propose a comprehensive paradigm, termed UniAff, that integrates
  3D object-centric manipulation and task understanding in a unified formulation.
  Specifically, we constructed a dataset labeled with manipulation-related key
  attributes, comprising 900 articulated objects from 19 categories and 600 tools
  from 12 categories. Furthermore, we leverage MLLMs to infer object-centric
  representations for manipulation tasks, including affordance recognition and
  reasoning about 3D motion constraints. Comprehensive experiments in both
  simulation and real-world settings indicate that UniAff significantly improves
  the generalization of robotic manipulation for tools and articulated objects.
  We hope that UniAff will serve as a general baseline for unified robotic
  manipulation tasks in the future. Images, videos, dataset, and code are
  published on the project website at:https://sites.google.com/view/uni-aff/home
paper_ai_keywords: MLLMs (Multimodal Large Language Models) affordance recognition 3D motion constraints