title: InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning
summary: Large multimodal foundation models, particularly in the domains of language
  and vision, have significantly advanced various tasks, including robotics,
  autonomous driving, information retrieval, and grounding. However, many of
  these models perceive objects as indivisible, overlooking the components that
  constitute them. Understanding these components and their associated
  affordances provides valuable insights into an object's functionality, which is
  fundamental for performing a wide range of tasks. In this work, we introduce a
  novel real-world benchmark, InstructPart, comprising hand-labeled part
  segmentation annotations and task-oriented instructions to evaluate the
  performance of current models in understanding and executing part-level tasks
  within everyday contexts. Through our experiments, we demonstrate that
  task-oriented part segmentation remains a challenging problem, even for
  state-of-the-art Vision-Language Models (VLMs). In addition to our benchmark,
  we introduce a simple baseline that achieves a twofold performance improvement
  through fine-tuning with our dataset. With our dataset and benchmark, we aim to
  facilitate research on task-oriented part segmentation and enhance the
  applicability of VLMs across various domains, including robotics, virtual
  reality, information retrieval, and other related fields. Project website:
  https://zifuwan.github.io/InstructPart/.
paper_authors._id: 6835e8411900f053ee40f208
paper_authors.name: Zifu Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:58:58.035Z
paper_authors._id: 65493c5b8fde27109badf365
paper_authors.avatarUrl: /avatars/8ff7e84f7fe3cefd80c344b17986c5b3.svg
paper_authors.fullname: Zifu Wan
paper_authors.type: user
paper_authors.user: zifuwan
paper_authors._id: 6835e8411900f053ee40f209
paper_authors.name: Yaqi Xie
paper_authors._id: 6835e8411900f053ee40f20a
paper_authors.name: Ce Zhang
paper_authors._id: 6835e8411900f053ee40f20b
paper_authors.name: Zhiqiu Lin
paper_authors._id: 6835e8411900f053ee40f20c
paper_authors.name: Zihan Wang
paper_authors._id: 6835e8411900f053ee40f20d
paper_authors.name: Simon Stepputtis
paper_authors._id: 6835e8411900f053ee40f20e
paper_authors.name: Deva Ramanan
paper_authors._id: 6835e8411900f053ee40f20f
paper_authors.name: Katia Sycara
paper_title: InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning
paper_summary: Large multimodal foundation models, particularly in the domains of language
  and vision, have significantly advanced various tasks, including robotics,
  autonomous driving, information retrieval, and grounding. However, many of
  these models perceive objects as indivisible, overlooking the components that
  constitute them. Understanding these components and their associated
  affordances provides valuable insights into an object's functionality, which is
  fundamental for performing a wide range of tasks. In this work, we introduce a
  novel real-world benchmark, InstructPart, comprising hand-labeled part
  segmentation annotations and task-oriented instructions to evaluate the
  performance of current models in understanding and executing part-level tasks
  within everyday contexts. Through our experiments, we demonstrate that
  task-oriented part segmentation remains a challenging problem, even for
  state-of-the-art Vision-Language Models (VLMs). In addition to our benchmark,
  we introduce a simple baseline that achieves a twofold performance improvement
  through fine-tuning with our dataset. With our dataset and benchmark, we aim to
  facilitate research on task-oriented part segmentation and enhance the
  applicability of VLMs across various domains, including robotics, virtual
  reality, information retrieval, and other related fields. Project website:
  https://zifuwan.github.io/InstructPart/.
paper_ai_summary: A new benchmark, InstructPart, and a task-oriented part segmentation dataset are introduced to evaluate and improve the performance of Vision-Language Models in real-world contexts.
paper_ai_keywords: Vision-Language Models part segmentation part-level tasks task-oriented instructions fine-tuning