title: AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
summary: Classifiers built upon vision-language models such as CLIP have shown
  remarkable zero-shot performance across a broad range of image classification
  tasks. Prior work has studied different ways of automatically creating
  descriptor sets for every class based on prompt templates, ranging from
  manually engineered templates over templates obtained from a large language
  model to templates built from random words and characters. In contrast,
  deriving zero-shot classifiers from the respective encoded class descriptors
  has remained nearly unchanged, that is: classify to the class that maximizes
  the cosine similarity between its averaged encoded class descriptors and the
  encoded image. However, weighting all class descriptors equally can be
  suboptimal when certain descriptors match visual clues on a given image better
  than others. In this work, we propose AutoCLIP, a method for auto-tuning
  zero-shot classifiers. AutoCLIP assigns to each prompt template per-image
  weights, which are derived from statistics of class descriptor-image
  similarities at inference time. AutoCLIP is fully unsupervised, has very low
  overhead, and can be easily implemented in few lines of code. We show that for
  a broad range of vision-language models, datasets, and prompt templates,
  AutoCLIP outperforms baselines consistently and by up to 3 percent point
  accuracy.
paper_authors._id: 65162d7399e8060f1e88170b
paper_authors.name: Jan Hendrik Metzen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:43:10.291Z
paper_authors._id: 6317016f7b0ee0136e5f567d
paper_authors.avatarUrl: /avatars/fe848d448318fdc17d42967b01101fea.svg
paper_authors.fullname: Jan Metzen
paper_authors.type: user
paper_authors.user: jmetzen
paper_authors._id: 65162d7399e8060f1e88170c
paper_authors.name: Piyapat Saranrittichai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-29T13:31:50.756Z
paper_authors._id: 6516c1e89f999b827aaf3a27
paper_authors.avatarUrl: /avatars/447225b5d99b8caa86772a34a73ac980.svg
paper_authors.fullname: Piyapat Saranrittichai
paper_authors.type: user
paper_authors.user: pathpot
paper_authors._id: 65162d7399e8060f1e88170d
paper_authors.name: Chaithanya Kumar Mummadi
paper_title: AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
paper_summary: Classifiers built upon vision-language models such as CLIP have shown
  remarkable zero-shot performance across a broad range of image classification
  tasks. Prior work has studied different ways of automatically creating
  descriptor sets for every class based on prompt templates, ranging from
  manually engineered templates over templates obtained from a large language
  model to templates built from random words and characters. In contrast,
  deriving zero-shot classifiers from the respective encoded class descriptors
  has remained nearly unchanged, that is: classify to the class that maximizes
  the cosine similarity between its averaged encoded class descriptors and the
  encoded image. However, weighting all class descriptors equally can be
  suboptimal when certain descriptors match visual clues on a given image better
  than others. In this work, we propose AutoCLIP, a method for auto-tuning
  zero-shot classifiers. AutoCLIP assigns to each prompt template per-image
  weights, which are derived from statistics of class descriptor-image
  similarities at inference time. AutoCLIP is fully unsupervised, has very low
  overhead, and can be easily implemented in few lines of code. We show that for
  a broad range of vision-language models, datasets, and prompt templates,
  AutoCLIP outperforms baselines consistently and by up to 3 percent point
  accuracy.
paper_ai_summary: AutoCLIP auto-tunes zero-shot classifiers by assigning per-image weights to prompt templates, improving performance across vision-language models and datasets.
paper_ai_keywords: CLIP zero-shot performance image classification prompt templates cosine similarity class descriptors AutoCLIP unsupervised per-image weights class descriptor-image similarities accuracy