No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency
  Determines Multimodal Model Performance
Web-crawled pretraining datasets underlie the impressive "zero-shot"
evaluation performance of multimodal models, such as CLIP for
classification/retrieval and Stable-Diffusion for image generation. However, it
is unclear how meaningful the notion of "zero-shot" generalization is for such
multimodal models, as it is not known to what extent their pretraining datasets
encompass the downstream concepts targeted for during "zero-shot" evaluation.
In this work, we ask: How is the performance of multimodal models on downstream
concepts influenced by the frequency of these concepts in their pretraining
datasets? We comprehensively investigate this question across 34 models and
five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M,
LAION-Aesthetics), generating over 300GB of data artifacts. We consistently
find that, far from exhibiting "zero-shot" generalization, multimodal models
require exponentially more data to achieve linear improvements in downstream
"zero-shot" performance, following a sample inefficient log-linear scaling
trend. This trend persists even when controlling for sample-level similarity
between pretraining and downstream datasets, and testing on purely synthetic
data distributions. Furthermore, upon benchmarking models on long-tailed data
sampled based on our analysis, we demonstrate that multimodal models across the
board perform poorly. We contribute this long-tail test set as the "Let it
Wag!" benchmark to further research in this direction. Taken together, our
study reveals an exponential need for training data which implies that the key
to "zero-shot" generalization capabilities under large-scale training paradigms
remains to be found.
66136c284a55d90baf155cf4
Vishaal Udandarao
claimed_verified
2024-04-08T08:13:59.937Z
6304da46ce6b12280b1bd575
https://cdn-avatars.huggingface.co/v1/production/uploads/6304da46ce6b12280b1bd575/V96ocKW4HOoysAGxuAH1X.jpeg
Vishaal Udandarao
user
vishaal27
66136c284a55d90baf155cf5
Ameya Prabhu
claimed_verified
2024-04-09T19:23:04.701Z
6464a0d41683d3c81f51924a
https://cdn-avatars.huggingface.co/v1/production/uploads/6464a0d41683d3c81f51924a/s7yYVwfUB4WOhVFJS6A6T.jpeg
Ameya Prabhu
user
AmeyaPrabhu
66136c284a55d90baf155cf6
Adhiraj Ghosh
claimed_verified
2024-12-13T16:06:41.087Z
6321b4417bb41a713da9a1f4
/avatars/f2cde57366791eabee12d456c833baae.svg
Adhiraj Ghosh
user
adhiraj1998
66136c284a55d90baf155cf7
Yash Sharma
claimed_verified
2024-08-13T07:40:40.713Z
631d218471f8e7137df7a073
/avatars/fc8d4ba096cbe21d31930d530a1216a0.svg
Yash Sharma
user
ysharma1126
66136c284a55d90baf155cf8
Philip H. S. Torr
66136c284a55d90baf155cf9
Adel Bibi
66136c284a55d90baf155cfa
Samuel Albanie
66136c284a55d90baf155cfb
Matthias Bethge
No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency
  Determines Multimodal Model Performance
Web-crawled pretraining datasets underlie the impressive "zero-shot"
evaluation performance of multimodal models, such as CLIP for
classification/retrieval and Stable-Diffusion for image generation. However, it
is unclear how meaningful the notion of "zero-shot" generalization is for such
multimodal models, as it is not known to what extent their pretraining datasets
encompass the downstream concepts targeted for during "zero-shot" evaluation.
In this work, we ask: How is the performance of multimodal models on downstream
concepts influenced by the frequency of these concepts in their pretraining
datasets? We comprehensively investigate this question across 34 models and
five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M,
LAION-Aesthetics), generating over 300GB of data artifacts. We consistently
find that, far from exhibiting "zero-shot" generalization, multimodal models
require exponentially more data to achieve linear improvements in downstream
"zero-shot" performance, following a sample inefficient log-linear scaling
trend. This trend persists even when controlling for sample-level similarity
between pretraining and downstream datasets, and testing on purely synthetic
data distributions. Furthermore, upon benchmarking models on long-tailed data
sampled based on our analysis, we demonstrate that multimodal models across the
board perform poorly. We contribute this long-tail test set as the "Let it
Wag!" benchmark to further research in this direction. Taken together, our
study reveals an exponential need for training data which implies that the key
to "zero-shot" generalization capabilities under large-scale training paradigms
remains to be found.
Studies reveal that multimodal models require exponentially more pretraining data for linear improvements in zero-shot performance and perform poorly on long-tailed datasets.
zero-shot multimodal models CLIP Stable-Diffusion pretraining datasets CC-3M CC-12M YFCC-15M LAION-400M LAION-Aesthetics sample inefficient log-linear scaling long-tail data "Let it Wag!" benchmark