AdaViPro: Region-based Adaptive Visual Prompt for Large-Scale Models Adapting
<|reference_start|>AdaViPro: Region-based Adaptive Visual Prompt for Large-Scale Models Adapting: Recently, prompt-based methods have emerged as a new alternative `parameter-efficient fine-tuning' paradigm, which only fine-tunes a small number of additional parameters while keeping the original model frozen. However, despite achieving notable results, existing prompt methods mainly focus on `what to add', while overlooking the equally important aspect of `where to add', typically relying on the manually crafted placement. To this end, we propose a region-based Adaptive Visual Prompt, named AdaViPro, which integrates the `where to add' optimization of the prompt into the learning process. Specifically, we reconceptualize the `where to add' optimization as a problem of regional decision-making. During inference, AdaViPro generates a regionalized mask map for the whole image, which is composed of 0 and 1, to designate whether to apply or discard the prompt in each specific area. Therefore, we employ Gumbel-Softmax sampling to enable AdaViPro's end-to-end learning through standard back-propagation. Extensive experiments demonstrate that our AdaViPro yields new efficiency and accuracy trade-offs for adapting pre-trained models.<|reference_end|>
@article{yang2024adavipro:,
    title={AdaViPro: Region-based Adaptive Visual Prompt for Large-Scale Models
  Adapting},
    author={Mengyu Yang, Ye Tian, Lanshan Zhang, Xiao Liang, Xuming Ran, Wendong
  Wang},
    journal={arXiv preprint arXiv:2403.13282},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.13282},
    primaryClass={cs.CV}
}