LOVM: Language-Only Vision Model Selection
Pre-trained multi-modal vision-language models (VLMs) are becoming
increasingly popular due to their exceptional performance on downstream vision
applications, particularly in the few- and zero-shot settings. However,
selecting the best-performing VLM for some downstream applications is
non-trivial, as it is dataset and task-dependent. Meanwhile, the exhaustive
evaluation of all available VLMs on a novel application is not only time and
computationally demanding but also necessitates the collection of a labeled
dataset for evaluation. As the number of open-source VLM variants increases,
there is a need for an efficient model selection strategy that does not require
access to a curated evaluation dataset. This paper proposes a novel task and
benchmark for efficiently evaluating VLMs' zero-shot performance on downstream
applications without access to the downstream task dataset. Specifically, we
introduce a new task LOVM: Language-Only Vision Model Selection, where methods
are expected to perform both model selection and performance prediction based
solely on a text description of the desired downstream application. We then
introduced an extensive LOVM benchmark consisting of ground-truth evaluations
of 35 pre-trained VLMs and 23 datasets, where methods are expected to rank the
pre-trained VLMs and predict their zero-shot performance.
648bd85c33603b7ec89a4cab
Orr Zohar
claimed_verified
2023-06-16T17:05:33.437Z
648c9605565e3a44f3c9bb7b
https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg
Orr Zohar
user
orrzohar
648bd85c33603b7ec89a4cac
Shih-Cheng Huang
648bd85c33603b7ec89a4cad
Kuan-Chieh Wang
claimed_verified
2023-06-16T18:50:42.605Z
648ca58a39d2584ee47efef6
https://cdn-avatars.huggingface.co/v1/production/uploads/648ca58a39d2584ee47efef6/R7B72bnwc59mdK45rmzYS.png
Kuan-Chieh Wang
user
wangkua1
648bd85c33603b7ec89a4cae
Serena Yeung
extracted_pending
2025-01-07T02:05:14.400Z
677c8b2e92550a07fcad0f50
/avatars/2be26e8f25e98cfe5b1d227ee0409cd0.svg
Serena Yeung-Levy
user
yeunglevy
LOVM: Language-Only Vision Model Selection
Pre-trained multi-modal vision-language models (VLMs) are becoming
increasingly popular due to their exceptional performance on downstream vision
applications, particularly in the few- and zero-shot settings. However,
selecting the best-performing VLM for some downstream applications is
non-trivial, as it is dataset and task-dependent. Meanwhile, the exhaustive
evaluation of all available VLMs on a novel application is not only time and
computationally demanding but also necessitates the collection of a labeled
dataset for evaluation. As the number of open-source VLM variants increases,
there is a need for an efficient model selection strategy that does not require
access to a curated evaluation dataset. This paper proposes a novel task and
benchmark for efficiently evaluating VLMs' zero-shot performance on downstream
applications without access to the downstream task dataset. Specifically, we
introduce a new task LOVM: Language-Only Vision Model Selection, where methods
are expected to perform both model selection and performance prediction based
solely on a text description of the desired downstream application. We then
introduced an extensive LOVM benchmark consisting of ground-truth evaluations
of 35 pre-trained VLMs and 23 datasets, where methods are expected to rank the
pre-trained VLMs and predict their zero-shot performance.
A new benchmark is introduced for efficient zero-shot evaluation of pre-trained vision-language models without requiring a downstream dataset, focusing on model selection based on text descriptions.
multi-modal vision-language models VLMs zero-shot performance downstream applications model selection performance prediction LOVM Language-Only Vision Model Selection