AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
Classifiers built upon vision-language models such as CLIP have shown
remarkable zero-shot performance across a broad range of image classification
tasks. Prior work has studied different ways of automatically creating
descriptor sets for every class based on prompt templates, ranging from
manually engineered templates over templates obtained from a large language
model to templates built from random words and characters. In contrast,
deriving zero-shot classifiers from the respective encoded class descriptors
has remained nearly unchanged, that is: classify to the class that maximizes
the cosine similarity between its averaged encoded class descriptors and the
encoded image. However, weighting all class descriptors equally can be
suboptimal when certain descriptors match visual clues on a given image better
than others. In this work, we propose AutoCLIP, a method for auto-tuning
zero-shot classifiers. AutoCLIP assigns to each prompt template per-image
weights, which are derived from statistics of class descriptor-image
similarities at inference time. AutoCLIP is fully unsupervised, has very low
overhead, and can be easily implemented in few lines of code. We show that for
a broad range of vision-language models, datasets, and prompt templates,
AutoCLIP outperforms baselines consistently and by up to 3 percent point
accuracy.
65162d7399e8060f1e88170b
Jan Hendrik Metzen
admin_assigned
2023-09-29T11:43:10.291Z
6317016f7b0ee0136e5f567d
/avatars/fe848d448318fdc17d42967b01101fea.svg
Jan Metzen
user
jmetzen
65162d7399e8060f1e88170c
Piyapat Saranrittichai
claimed_verified
2023-09-29T13:31:50.756Z
6516c1e89f999b827aaf3a27
/avatars/6c9ea0f975e744e837dcd6d64f6e84b6.svg
Piyapat Saranrittichai
user
pathpot
65162d7399e8060f1e88170d
Chaithanya Kumar Mummadi
AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
Classifiers built upon vision-language models such as CLIP have shown
remarkable zero-shot performance across a broad range of image classification
tasks. Prior work has studied different ways of automatically creating
descriptor sets for every class based on prompt templates, ranging from
manually engineered templates over templates obtained from a large language
model to templates built from random words and characters. In contrast,
deriving zero-shot classifiers from the respective encoded class descriptors
has remained nearly unchanged, that is: classify to the class that maximizes
the cosine similarity between its averaged encoded class descriptors and the
encoded image. However, weighting all class descriptors equally can be
suboptimal when certain descriptors match visual clues on a given image better
than others. In this work, we propose AutoCLIP, a method for auto-tuning
zero-shot classifiers. AutoCLIP assigns to each prompt template per-image
weights, which are derived from statistics of class descriptor-image
similarities at inference time. AutoCLIP is fully unsupervised, has very low
overhead, and can be easily implemented in few lines of code. We show that for
a broad range of vision-language models, datasets, and prompt templates,
AutoCLIP outperforms baselines consistently and by up to 3 percent point
accuracy.
AutoCLIP auto-tunes zero-shot classifiers by assigning per-image weights to prompt templates, improving performance across vision-language models and datasets.
CLIP zero-shot performance image classification prompt templates cosine similarity class descriptors AutoCLIP unsupervised per-image weights class descriptor-image similarities accuracy