title: Improving Multimodal Datasets with Image Captioning
summary: Massive web datasets play a key role in the success of large vision-language
  models like CLIP and Flamingo. However, the raw web data is noisy, and existing
  filtering methods to reduce noise often come at the expense of data diversity.
  Our work focuses on caption quality as one major source of noise, and studies
  how generated captions can increase the utility of web-scraped datapoints with
  nondescript text. Through exploring different mixing strategies for raw and
  generated captions, we outperform the best filtering method proposed by the
  DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given a
  candidate pool of 128M image-text pairs. Our best approach is also 2x better at
  Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an
  effective source of text supervision. In experimenting with different image
  captioning models, we also demonstrate that the performance of a model on
  standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable
  indicator of the utility of the captions it generates for multimodal training.
  Finally, our experiments with using generated captions at DataComp's large
  scale (1.28B image-text pairs) offer insights into the limitations of synthetic
  text, as well as the importance of image curation with increasing training data
  quantity.
paper_authors._id: 64b9f16e2cfce8f7b3d8e9f6
paper_authors.name: Thao Nguyen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T09:30:31.391Z
paper_authors._id: 64ba4923921dbcbc96006fb8
paper_authors.avatarUrl: /avatars/4eab3991e8502f72ddf8ca4825c2ad65.svg
paper_authors.fullname: Thao Nguyen
paper_authors.type: user
paper_authors.user: thaottn
paper_authors._id: 64b9f16e2cfce8f7b3d8e9f7
paper_authors.name: Samir Yitzhak Gadre
paper_authors._id: 64b9f16e2cfce8f7b3d8e9f8
paper_authors.name: Gabriel Ilharco
paper_authors._id: 64b9f16e2cfce8f7b3d8e9f9
paper_authors.name: Sewoong Oh
paper_authors._id: 64b9f16e2cfce8f7b3d8e9fa
paper_authors.name: Ludwig Schmidt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T07:57:33.903Z
paper_authors._id: 6252375c818a5dc29ab9d945
paper_authors.avatarUrl: /avatars/cd4f0b220ea36e13ff71157cffe9d9d1.svg
paper_authors.fullname: Ludwig Schmidt
paper_authors.type: user
paper_authors.user: ludwigschmidt
paper_title: Improving Multimodal Datasets with Image Captioning
paper_summary: Massive web datasets play a key role in the success of large vision-language
  models like CLIP and Flamingo. However, the raw web data is noisy, and existing
  filtering methods to reduce noise often come at the expense of data diversity.
  Our work focuses on caption quality as one major source of noise, and studies
  how generated captions can increase the utility of web-scraped datapoints with
  nondescript text. Through exploring different mixing strategies for raw and
  generated captions, we outperform the best filtering method proposed by the
  DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given a
  candidate pool of 128M image-text pairs. Our best approach is also 2x better at
  Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an
  effective source of text supervision. In experimenting with different image
  captioning models, we also demonstrate that the performance of a model on
  standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable
  indicator of the utility of the captions it generates for multimodal training.
  Finally, our experiments with using generated captions at DataComp's large
  scale (1.28B image-text pairs) offer insights into the limitations of synthetic
  text, as well as the importance of image curation with increasing training data
  quantity.
paper_ai_keywords: vision-language models CLIP Flamingo caption quality generated captions DataComp benchmark ImageNet Flickr MS-COCO retrieval NoCaps CIDEr multimodal training image curation