LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Foundation Models
<|reference_start|>LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Foundation Models: Vision Language Models (VLMs) have shown impressive performances on numerous tasks but their zero-shot capabilities can be limited compared to dedicated or fine-tuned models. Yet, fine-tuning VLMs comes with limitations as it requires `white-box' access to the model's architecture and weights as well as expertise to design the fine-tuning objectives and optimize the hyper-parameters, which are specific to each VLM and downstream task. In this work, we propose LLM-wrapper, a novel approach to adapt VLMs in a `black-box' manner by leveraging large language models (LLMs) so as to reason on their outputs. We demonstrate the effectiveness of LLM-wrapper on Referring Expression Comprehension (REC), a challenging open-vocabulary task that requires spatial and semantic reasoning. Our approach significantly boosts the performance of off-the-shelf models, resulting in competitive results when compared with classic fine-tuning.<|reference_end|>
@article{cardiel2024llm-wrapper:,
    title={LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language
  Models for Referring Expression Comprehension},
    author={Amaia Cardiel, Eloi Zablocki, Elias Ramzi, Oriane Sim'eoni, Matthieu
  Cord},
    journal={arXiv preprint arXiv:2409.11919},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.11919},
    primaryClass={cs.CV}
}