title: No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency
    Determines Multimodal Model Performance
summary: Web-crawled pretraining datasets underlie the impressive "zero-shot"
  evaluation performance of multimodal models, such as CLIP for
  classification/retrieval and Stable-Diffusion for image generation. However, it
  is unclear how meaningful the notion of "zero-shot" generalization is for such
  multimodal models, as it is not known to what extent their pretraining datasets
  encompass the downstream concepts targeted for during "zero-shot" evaluation.
  In this work, we ask: How is the performance of multimodal models on downstream
  concepts influenced by the frequency of these concepts in their pretraining
  datasets? We comprehensively investigate this question across 34 models and
  five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M,
  LAION-Aesthetics), generating over 300GB of data artifacts. We consistently
  find that, far from exhibiting "zero-shot" generalization, multimodal models
  require exponentially more data to achieve linear improvements in downstream
  "zero-shot" performance, following a sample inefficient log-linear scaling
  trend. This trend persists even when controlling for sample-level similarity
  between pretraining and downstream datasets, and testing on purely synthetic
  data distributions. Furthermore, upon benchmarking models on long-tailed data
  sampled based on our analysis, we demonstrate that multimodal models across the
  board perform poorly. We contribute this long-tail test set as the "Let it
  Wag!" benchmark to further research in this direction. Taken together, our
  study reveals an exponential need for training data which implies that the key
  to "zero-shot" generalization capabilities under large-scale training paradigms
  remains to be found.
paper_authors._id: 66136c284a55d90baf155cf4
paper_authors.name: Vishaal Udandarao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-08T08:13:59.937Z
paper_authors._id: 6304da46ce6b12280b1bd575
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6304da46ce6b12280b1bd575/V96ocKW4HOoysAGxuAH1X.jpeg
paper_authors.fullname: Vishaal Udandarao
paper_authors.type: user
paper_authors.user: vishaal27
paper_authors._id: 66136c284a55d90baf155cf5
paper_authors.name: Ameya Prabhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-09T19:23:04.701Z
paper_authors._id: 6464a0d41683d3c81f51924a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6464a0d41683d3c81f51924a/s7yYVwfUB4WOhVFJS6A6T.jpeg
paper_authors.fullname: Ameya Prabhu
paper_authors.type: user
paper_authors.user: AmeyaPrabhu
paper_authors._id: 66136c284a55d90baf155cf6
paper_authors.name: Adhiraj Ghosh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T16:06:41.087Z
paper_authors._id: 6321b4417bb41a713da9a1f4
paper_authors.avatarUrl: /avatars/f2cde57366791eabee12d456c833baae.svg
paper_authors.fullname: Adhiraj Ghosh
paper_authors.type: user
paper_authors.user: adhiraj1998
paper_authors._id: 66136c284a55d90baf155cf7
paper_authors.name: Yash Sharma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-13T07:40:40.713Z
paper_authors._id: 631d218471f8e7137df7a073
paper_authors.avatarUrl: /avatars/fc8d4ba096cbe21d31930d530a1216a0.svg
paper_authors.fullname: Yash Sharma
paper_authors.type: user
paper_authors.user: ysharma1126
paper_authors._id: 66136c284a55d90baf155cf8
paper_authors.name: Philip H. S. Torr
paper_authors._id: 66136c284a55d90baf155cf9
paper_authors.name: Adel Bibi
paper_authors._id: 66136c284a55d90baf155cfa
paper_authors.name: Samuel Albanie
paper_authors._id: 66136c284a55d90baf155cfb
paper_authors.name: Matthias Bethge
paper_title: No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency
    Determines Multimodal Model Performance
paper_summary: Web-crawled pretraining datasets underlie the impressive "zero-shot"
  evaluation performance of multimodal models, such as CLIP for
  classification/retrieval and Stable-Diffusion for image generation. However, it
  is unclear how meaningful the notion of "zero-shot" generalization is for such
  multimodal models, as it is not known to what extent their pretraining datasets
  encompass the downstream concepts targeted for during "zero-shot" evaluation.
  In this work, we ask: How is the performance of multimodal models on downstream
  concepts influenced by the frequency of these concepts in their pretraining
  datasets? We comprehensively investigate this question across 34 models and
  five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M,
  LAION-Aesthetics), generating over 300GB of data artifacts. We consistently
  find that, far from exhibiting "zero-shot" generalization, multimodal models
  require exponentially more data to achieve linear improvements in downstream
  "zero-shot" performance, following a sample inefficient log-linear scaling
  trend. This trend persists even when controlling for sample-level similarity
  between pretraining and downstream datasets, and testing on purely synthetic
  data distributions. Furthermore, upon benchmarking models on long-tailed data
  sampled based on our analysis, we demonstrate that multimodal models across the
  board perform poorly. We contribute this long-tail test set as the "Let it
  Wag!" benchmark to further research in this direction. Taken together, our
  study reveals an exponential need for training data which implies that the key
  to "zero-shot" generalization capabilities under large-scale training paradigms
  remains to be found.
paper_ai_summary: Studies reveal that multimodal models require exponentially more pretraining data for linear improvements in zero-shot performance and perform poorly on long-tailed datasets.
paper_ai_keywords: zero-shot multimodal models CLIP Stable-Diffusion pretraining datasets CC-3M CC-12M YFCC-15M LAION-400M LAION-Aesthetics sample inefficient log-linear scaling long-tail data "Let it Wag!" benchmark