TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models
<|reference_start|>TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models: Vision-Language Models (VLMs) have shown impressive performance in vision tasks, but adapting them to new domains often requires expensive fine-tuning. Prompt tuning techniques, including textual, visual, and multimodal prompting, offer efficient alternatives by leveraging learnable prompts. However, their application to Vision-Language Segmentation Models (VLSMs) and evaluation under significant domain shifts remain unexplored. This work presents an open-source benchmarking framework, TuneVLSeg, to integrate various unimodal and multimodal prompt tuning techniques into VLSMs, making prompt tuning usable for downstream segmentation datasets with any number of classes. TuneVLSeg includes $6$ prompt tuning strategies on various prompt depths used in $2$ VLSMs totaling of $8$ different combinations. We test various prompt tuning on $8$ diverse medical datasets, including $3$ radiology datasets (breast tumor, echocardiograph, chest X-ray pathologies) and $5$ non-radiology datasets (polyp, ulcer, skin cancer), and two natural domain segmentation datasets. Our study found that textual prompt tuning struggles under significant domain shifts, from natural-domain images to medical data. Furthermore, visual prompt tuning, with fewer hyperparameters than multimodal prompt tuning, often achieves performance competitive to multimodal approaches, making it a valuable first attempt. Our work advances the understanding and applicability of different prompt-tuning techniques for robust domain-specific segmentation. The source code is available at https://github.com/naamiinepal/tunevlseg.<|reference_end|>
@article{adhikari2024tunevlseg:,
    title={TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation
  Models},
    author={Rabin Adhikari, Safal Thapaliya, Manish Dhakal, Bishesh Khanal},
    journal={arXiv preprint arXiv:2410.05239},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.05239},
    primaryClass={cs.CV cs.CL}
}