INT: Instance-Specific Negative Mining for Task-Generic Promptable
  Segmentation
Task-generic promptable image segmentation aims to achieve segmentation of
diverse samples under a single task description by utilizing only one
task-generic prompt. Current methods leverage the generalization capabilities
of Vision-Language Models (VLMs) to infer instance-specific prompts from these
task-generic prompts in order to guide the segmentation process. However, when
VLMs struggle to generalise to some image instances, predicting
instance-specific prompts becomes poor. To solve this problem, we introduce
Instance-specific Negative Mining for Task-Generic
Promptable Segmentation (INT). The key idea of INT is to adaptively
reduce the influence of irrelevant (negative) prior knowledge whilst to
increase the use the most plausible prior knowledge, selected by negative
mining with higher contrast, in order to optimise instance-specific prompts
generation. Specifically, INT consists of two components: (1) instance-specific
prompt generation, which progressively fliters out incorrect information in
prompt generation; (2) semantic mask generation, which ensures each image
instance segmentation matches correctly the semantics of the instance-specific
prompts. INT is validated on six datasets, including camouflaged objects and
medical images, demonstrating its effectiveness, robustness and scalability.
67a0a09da2d6613d77a7d10e
Jian Hu
admin_assigned
2025-02-03T12:59:15.739Z
65e1b6e9501590df0173cbd3
/avatars/a73e2139700e23eff455734c99cef5ba.svg
Jian Hu
user
lwpyh
67a0a09da2d6613d77a7d10f
Zixu Cheng
admin_assigned
2025-02-03T12:59:22.704Z
667ee096b0fad0fdee319ed4
/avatars/d9df687e8522d47f7fcefe40fd9b575b.svg
Zixu Cheng
user
Cade921
67a0a09da2d6613d77a7d110
Shaogang Gong
INT: Instance-Specific Negative Mining for Task-Generic Promptable
  Segmentation
Task-generic promptable image segmentation aims to achieve segmentation of
diverse samples under a single task description by utilizing only one
task-generic prompt. Current methods leverage the generalization capabilities
of Vision-Language Models (VLMs) to infer instance-specific prompts from these
task-generic prompts in order to guide the segmentation process. However, when
VLMs struggle to generalise to some image instances, predicting
instance-specific prompts becomes poor. To solve this problem, we introduce
Instance-specific Negative Mining for Task-Generic
Promptable Segmentation (INT). The key idea of INT is to adaptively
reduce the influence of irrelevant (negative) prior knowledge whilst to
increase the use the most plausible prior knowledge, selected by negative
mining with higher contrast, in order to optimise instance-specific prompts
generation. Specifically, INT consists of two components: (1) instance-specific
prompt generation, which progressively fliters out incorrect information in
prompt generation; (2) semantic mask generation, which ensures each image
instance segmentation matches correctly the semantics of the instance-specific
prompts. INT is validated on six datasets, including camouflaged objects and
medical images, demonstrating its effectiveness, robustness and scalability.
The INT method enhances task-generic image segmentation by adaptively optimizing instance-specific prompts through negative mining to improve generalization and robustness across diverse datasets.
Vision-Language Models instance-specific prompts instance-specific negative mining prompt generation semantic mask generation camouflaged objects medical images