InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning
Large multimodal foundation models, particularly in the domains of language
and vision, have significantly advanced various tasks, including robotics,
autonomous driving, information retrieval, and grounding. However, many of
these models perceive objects as indivisible, overlooking the components that
constitute them. Understanding these components and their associated
affordances provides valuable insights into an object's functionality, which is
fundamental for performing a wide range of tasks. In this work, we introduce a
novel real-world benchmark, InstructPart, comprising hand-labeled part
segmentation annotations and task-oriented instructions to evaluate the
performance of current models in understanding and executing part-level tasks
within everyday contexts. Through our experiments, we demonstrate that
task-oriented part segmentation remains a challenging problem, even for
state-of-the-art Vision-Language Models (VLMs). In addition to our benchmark,
we introduce a simple baseline that achieves a twofold performance improvement
through fine-tuning with our dataset. With our dataset and benchmark, we aim to
facilitate research on task-oriented part segmentation and enhance the
applicability of VLMs across various domains, including robotics, virtual
reality, information retrieval, and other related fields. Project website:
https://zifuwan.github.io/InstructPart/.
6835e8411900f053ee40f208
Zifu Wan
claimed_verified
2025-05-28T08:58:58.035Z
65493c5b8fde27109badf365
/avatars/8ff7e84f7fe3cefd80c344b17986c5b3.svg
Zifu Wan
user
zifuwan
6835e8411900f053ee40f209
Yaqi Xie
6835e8411900f053ee40f20a
Ce Zhang
6835e8411900f053ee40f20b
Zhiqiu Lin
6835e8411900f053ee40f20c
Zihan Wang
6835e8411900f053ee40f20d
Simon Stepputtis
6835e8411900f053ee40f20e
Deva Ramanan
6835e8411900f053ee40f20f
Katia Sycara
InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning
Large multimodal foundation models, particularly in the domains of language
and vision, have significantly advanced various tasks, including robotics,
autonomous driving, information retrieval, and grounding. However, many of
these models perceive objects as indivisible, overlooking the components that
constitute them. Understanding these components and their associated
affordances provides valuable insights into an object's functionality, which is
fundamental for performing a wide range of tasks. In this work, we introduce a
novel real-world benchmark, InstructPart, comprising hand-labeled part
segmentation annotations and task-oriented instructions to evaluate the
performance of current models in understanding and executing part-level tasks
within everyday contexts. Through our experiments, we demonstrate that
task-oriented part segmentation remains a challenging problem, even for
state-of-the-art Vision-Language Models (VLMs). In addition to our benchmark,
we introduce a simple baseline that achieves a twofold performance improvement
through fine-tuning with our dataset. With our dataset and benchmark, we aim to
facilitate research on task-oriented part segmentation and enhance the
applicability of VLMs across various domains, including robotics, virtual
reality, information retrieval, and other related fields. Project website:
https://zifuwan.github.io/InstructPart/.
A new benchmark, InstructPart, and a task-oriented part segmentation dataset are introduced to evaluate and improve the performance of Vision-Language Models in real-world contexts.
Vision-Language Models part segmentation part-level tasks task-oriented instructions fine-tuning