VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation
<|reference_start|>VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation: Vision Transformers (ViTs) have emerged as the backbone of many segmentation models, consistently achieving state-of-the-art (SOTA) performance. However, their success comes at a significant computational cost. Image token pruning is one of the most effective strategies to address this complexity. However, previous approaches fall short when applied to more complex task-oriented segmentation (TOS), where the class of each image patch is not predefined but dependent on the specific input task. This work introduces the Vision Language Guided Token Pruning (VLTP), a novel token pruning mechanism that can accelerate ViTbased segmentation models, particularly for TOS guided by multi-modal large language model (MLLM). We argue that ViT does not need to process every image token through all of its layers only the tokens related to reasoning tasks are necessary. We design a new pruning decoder to take both image tokens and vision-language guidance as input to predict the relevance of each image token to the task. Only image tokens with high relevance are passed to deeper layers of the ViT. Experiments show that the VLTP framework reduces the computational costs of ViT by approximately 25% without performance degradation and by around 40% with only a 1% performance drop.<|reference_end|>
@article{chen2024vltp:,
    title={VLTP: Vision-Language Guided Token Pruning for Task-Oriented
  Segmentation},
    author={Hanning Chen, Yang Ni, Wenjun Huang, Yezi Liu, SungHeon Jeong, Fei
  Wen, Nathaniel Bastian, Hugo Latapie, Mohsen Imani},
    journal={arXiv preprint arXiv:2409.08464},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.08464},
    primaryClass={cs.CV}
}