SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by
  Imitating Human Annotator Trajectories
While MLLMs have demonstrated adequate image understanding capabilities, they
still struggle with pixel-level comprehension, limiting their practical
applications. Current evaluation tasks like VQA and visual grounding remain too
coarse to assess fine-grained pixel comprehension accurately. Though
segmentation is foundational for pixel-level understanding, existing methods
often require MLLMs to generate implicit tokens, decoded through external pixel
decoders. This approach disrupts the MLLM's text output space, potentially
compromising language capabilities and reducing flexibility and extensibility,
while failing to reflect the model's intrinsic pixel-level understanding.
  Thus, we introduce the Human-Like Mask Annotation Task (HLMAT), a new
paradigm where MLLMs mimic human annotators using interactive segmentation
tools. Modeling segmentation as a multi-step Markov Decision Process, HLMAT
enables MLLMs to iteratively generate text-based click points, achieving
high-quality masks without architectural changes or implicit tokens. Through
this setup, we develop SegAgent, a model fine-tuned on human-like annotation
trajectories, which achieves performance comparable to state-of-the-art (SOTA)
methods and supports additional tasks like mask refinement and annotation
filtering.
  HLMAT provides a protocol for assessing fine-grained pixel understanding in
MLLMs and introduces a vision-centric, multi-step decision-making task that
facilitates exploration of MLLMs' visual reasoning abilities. Our adaptations
of policy improvement method StaR and PRM-guided tree search further enhance
model robustness in complex segmentation tasks, laying a foundation for future
advancements in fine-grained visual perception and multi-step decision-making
for MLLMs.
67d0fd74f8595b656f921a48
Muzhi Zhu
claimed_verified
2025-03-12T08:36:42.495Z
632179745fc60c44fd91fc33
/avatars/37d4fefbcc19f091dccffefec9706de2.svg
zhumuzhi
user
Z-MU-Z
67d0fd74f8595b656f921a49
Yuzhuo Tian
67d0fd74f8595b656f921a4a
Hao Chen
67d0fd74f8595b656f921a4b
Chunluan Zhou
67d0fd74f8595b656f921a4c
Qingpei Guo
67d0fd74f8595b656f921a4d
Yang Liu
67d0fd74f8595b656f921a4e
Ming Yang
67d0fd74f8595b656f921a4f
Chunhua Shen
SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by
  Imitating Human Annotator Trajectories
While MLLMs have demonstrated adequate image understanding capabilities, they
still struggle with pixel-level comprehension, limiting their practical
applications. Current evaluation tasks like VQA and visual grounding remain too
coarse to assess fine-grained pixel comprehension accurately. Though
segmentation is foundational for pixel-level understanding, existing methods
often require MLLMs to generate implicit tokens, decoded through external pixel
decoders. This approach disrupts the MLLM's text output space, potentially
compromising language capabilities and reducing flexibility and extensibility,
while failing to reflect the model's intrinsic pixel-level understanding.
  Thus, we introduce the Human-Like Mask Annotation Task (HLMAT), a new
paradigm where MLLMs mimic human annotators using interactive segmentation
tools. Modeling segmentation as a multi-step Markov Decision Process, HLMAT
enables MLLMs to iteratively generate text-based click points, achieving
high-quality masks without architectural changes or implicit tokens. Through
this setup, we develop SegAgent, a model fine-tuned on human-like annotation
trajectories, which achieves performance comparable to state-of-the-art (SOTA)
methods and supports additional tasks like mask refinement and annotation
filtering.
  HLMAT provides a protocol for assessing fine-grained pixel understanding in
MLLMs and introduces a vision-centric, multi-step decision-making task that
facilitates exploration of MLLMs' visual reasoning abilities. Our adaptations
of policy improvement method StaR and PRM-guided tree search further enhance
model robustness in complex segmentation tasks, laying a foundation for future
advancements in fine-grained visual perception and multi-step decision-making
for MLLMs.
HLMAT introduces a new paradigm for assessing fine-grained pixel understanding in MLLMs through interactive segmentation tasks, enabling high-quality mask generation and supporting additional tasks without architectural changes.
MLLMs pixel-level comprehension VQA visual grounding segmentation implicit tokens pixel decoders Human-Like Mask Annotation Task (HLMAT) multi-step Markov Decision Process SegAgent StaR PRM-guided tree search vision-centric visual reasoning fine-grained visual perception