title: SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by
    Imitating Human Annotator Trajectories
summary: While MLLMs have demonstrated adequate image understanding capabilities, they
  still struggle with pixel-level comprehension, limiting their practical
  applications. Current evaluation tasks like VQA and visual grounding remain too
  coarse to assess fine-grained pixel comprehension accurately. Though
  segmentation is foundational for pixel-level understanding, existing methods
  often require MLLMs to generate implicit tokens, decoded through external pixel
  decoders. This approach disrupts the MLLM's text output space, potentially
  compromising language capabilities and reducing flexibility and extensibility,
  while failing to reflect the model's intrinsic pixel-level understanding.
    Thus, we introduce the Human-Like Mask Annotation Task (HLMAT), a new
  paradigm where MLLMs mimic human annotators using interactive segmentation
  tools. Modeling segmentation as a multi-step Markov Decision Process, HLMAT
  enables MLLMs to iteratively generate text-based click points, achieving
  high-quality masks without architectural changes or implicit tokens. Through
  this setup, we develop SegAgent, a model fine-tuned on human-like annotation
  trajectories, which achieves performance comparable to state-of-the-art (SOTA)
  methods and supports additional tasks like mask refinement and annotation
  filtering.
    HLMAT provides a protocol for assessing fine-grained pixel understanding in
  MLLMs and introduces a vision-centric, multi-step decision-making task that
  facilitates exploration of MLLMs' visual reasoning abilities. Our adaptations
  of policy improvement method StaR and PRM-guided tree search further enhance
  model robustness in complex segmentation tasks, laying a foundation for future
  advancements in fine-grained visual perception and multi-step decision-making
  for MLLMs.
paper_authors._id: 67d0fd74f8595b656f921a48
paper_authors.name: Muzhi Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:36:42.495Z
paper_authors._id: 632179745fc60c44fd91fc33
paper_authors.avatarUrl: /avatars/37d4fefbcc19f091dccffefec9706de2.svg
paper_authors.fullname: zhumuzhi
paper_authors.type: user
paper_authors.user: Z-MU-Z
paper_authors._id: 67d0fd74f8595b656f921a49
paper_authors.name: Yuzhuo Tian
paper_authors._id: 67d0fd74f8595b656f921a4a
paper_authors.name: Hao Chen
paper_authors._id: 67d0fd74f8595b656f921a4b
paper_authors.name: Chunluan Zhou
paper_authors._id: 67d0fd74f8595b656f921a4c
paper_authors.name: Qingpei Guo
paper_authors._id: 67d0fd74f8595b656f921a4d
paper_authors.name: Yang Liu
paper_authors._id: 67d0fd74f8595b656f921a4e
paper_authors.name: Ming Yang
paper_authors._id: 67d0fd74f8595b656f921a4f
paper_authors.name: Chunhua Shen
paper_title: SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by
    Imitating Human Annotator Trajectories
paper_summary: While MLLMs have demonstrated adequate image understanding capabilities, they
  still struggle with pixel-level comprehension, limiting their practical
  applications. Current evaluation tasks like VQA and visual grounding remain too
  coarse to assess fine-grained pixel comprehension accurately. Though
  segmentation is foundational for pixel-level understanding, existing methods
  often require MLLMs to generate implicit tokens, decoded through external pixel
  decoders. This approach disrupts the MLLM's text output space, potentially
  compromising language capabilities and reducing flexibility and extensibility,
  while failing to reflect the model's intrinsic pixel-level understanding.
    Thus, we introduce the Human-Like Mask Annotation Task (HLMAT), a new
  paradigm where MLLMs mimic human annotators using interactive segmentation
  tools. Modeling segmentation as a multi-step Markov Decision Process, HLMAT
  enables MLLMs to iteratively generate text-based click points, achieving
  high-quality masks without architectural changes or implicit tokens. Through
  this setup, we develop SegAgent, a model fine-tuned on human-like annotation
  trajectories, which achieves performance comparable to state-of-the-art (SOTA)
  methods and supports additional tasks like mask refinement and annotation
  filtering.
    HLMAT provides a protocol for assessing fine-grained pixel understanding in
  MLLMs and introduces a vision-centric, multi-step decision-making task that
  facilitates exploration of MLLMs' visual reasoning abilities. Our adaptations
  of policy improvement method StaR and PRM-guided tree search further enhance
  model robustness in complex segmentation tasks, laying a foundation for future
  advancements in fine-grained visual perception and multi-step decision-making
  for MLLMs.
paper_ai_keywords: Human-Like Mask Annotation Task (HLMAT) Markov Decision Process multi-step decision-making click points masks policy improvement method StaR PRM-guided tree search mask refinement annotation filtering fine-grained pixel understanding vision-centric task visual reasoning abilities