title: LOVM: Language-Only Vision Model Selection
summary: Pre-trained multi-modal vision-language models (VLMs) are becoming
  increasingly popular due to their exceptional performance on downstream vision
  applications, particularly in the few- and zero-shot settings. However,
  selecting the best-performing VLM for some downstream applications is
  non-trivial, as it is dataset and task-dependent. Meanwhile, the exhaustive
  evaluation of all available VLMs on a novel application is not only time and
  computationally demanding but also necessitates the collection of a labeled
  dataset for evaluation. As the number of open-source VLM variants increases,
  there is a need for an efficient model selection strategy that does not require
  access to a curated evaluation dataset. This paper proposes a novel task and
  benchmark for efficiently evaluating VLMs' zero-shot performance on downstream
  applications without access to the downstream task dataset. Specifically, we
  introduce a new task LOVM: Language-Only Vision Model Selection, where methods
  are expected to perform both model selection and performance prediction based
  solely on a text description of the desired downstream application. We then
  introduced an extensive LOVM benchmark consisting of ground-truth evaluations
  of 35 pre-trained VLMs and 23 datasets, where methods are expected to rank the
  pre-trained VLMs and predict their zero-shot performance.
paper_authors._id: 648bd85c33603b7ec89a4cab
paper_authors.name: Orr Zohar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-16T17:05:33.437Z
paper_authors._id: 648c9605565e3a44f3c9bb7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg
paper_authors.fullname: Orr Zohar
paper_authors.type: user
paper_authors.user: orrzohar
paper_authors._id: 648bd85c33603b7ec89a4cac
paper_authors.name: Shih-Cheng Huang
paper_authors._id: 648bd85c33603b7ec89a4cad
paper_authors.name: Kuan-Chieh Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-16T18:50:42.605Z
paper_authors._id: 648ca58a39d2584ee47efef6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648ca58a39d2584ee47efef6/R7B72bnwc59mdK45rmzYS.png
paper_authors.fullname: Kuan-Chieh Wang
paper_authors.type: user
paper_authors.user: wangkua1
paper_authors._id: 648bd85c33603b7ec89a4cae
paper_authors.name: Serena Yeung
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-07T02:05:14.400Z
paper_authors._id: 677c8b2e92550a07fcad0f50
paper_authors.avatarUrl: /avatars/2be26e8f25e98cfe5b1d227ee0409cd0.svg
paper_authors.fullname: Serena Yeung-Levy
paper_authors.type: user
paper_authors.user: yeunglevy
paper_title: LOVM: Language-Only Vision Model Selection
paper_summary: Pre-trained multi-modal vision-language models (VLMs) are becoming
  increasingly popular due to their exceptional performance on downstream vision
  applications, particularly in the few- and zero-shot settings. However,
  selecting the best-performing VLM for some downstream applications is
  non-trivial, as it is dataset and task-dependent. Meanwhile, the exhaustive
  evaluation of all available VLMs on a novel application is not only time and
  computationally demanding but also necessitates the collection of a labeled
  dataset for evaluation. As the number of open-source VLM variants increases,
  there is a need for an efficient model selection strategy that does not require
  access to a curated evaluation dataset. This paper proposes a novel task and
  benchmark for efficiently evaluating VLMs' zero-shot performance on downstream
  applications without access to the downstream task dataset. Specifically, we
  introduce a new task LOVM: Language-Only Vision Model Selection, where methods
  are expected to perform both model selection and performance prediction based
  solely on a text description of the desired downstream application. We then
  introduced an extensive LOVM benchmark consisting of ground-truth evaluations
  of 35 pre-trained VLMs and 23 datasets, where methods are expected to rank the
  pre-trained VLMs and predict their zero-shot performance.
paper_ai_keywords: multi-modal vision-language models (VLMs) downstream vision applications few-shot zero-shot pre-trained VLM variants efficient model selection strategy LOVM: Language-Only Vision Model Selection model selection performance prediction text description ground-truth evaluations