Improving Multimodal Datasets with Image Captioning
Massive web datasets play a key role in the success of large vision-language
models like CLIP and Flamingo. However, the raw web data is noisy, and existing
filtering methods to reduce noise often come at the expense of data diversity.
Our work focuses on caption quality as one major source of noise, and studies
how generated captions can increase the utility of web-scraped datapoints with
nondescript text. Through exploring different mixing strategies for raw and
generated captions, we outperform the best filtering method proposed by the
DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given a
candidate pool of 128M image-text pairs. Our best approach is also 2x better at
Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an
effective source of text supervision. In experimenting with different image
captioning models, we also demonstrate that the performance of a model on
standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable
indicator of the utility of the captions it generates for multimodal training.
Finally, our experiments with using generated captions at DataComp's large
scale (1.28B image-text pairs) offer insights into the limitations of synthetic
text, as well as the importance of image curation with increasing training data
quantity.
64b9f16e2cfce8f7b3d8e9f6
Thao Nguyen
admin_assigned
2023-07-21T09:30:31.391Z
64ba4923921dbcbc96006fb8
/avatars/4eab3991e8502f72ddf8ca4825c2ad65.svg
Thao Nguyen
user
thaottn
64b9f16e2cfce8f7b3d8e9f7
Samir Yitzhak Gadre
64b9f16e2cfce8f7b3d8e9f8
Gabriel Ilharco
64b9f16e2cfce8f7b3d8e9f9
Sewoong Oh
64b9f16e2cfce8f7b3d8e9fa
Ludwig Schmidt
admin_assigned
2023-07-21T07:57:33.903Z
6252375c818a5dc29ab9d945
/avatars/cd4f0b220ea36e13ff71157cffe9d9d1.svg
Ludwig Schmidt
user
ludwigschmidt
Improving Multimodal Datasets with Image Captioning
Massive web datasets play a key role in the success of large vision-language
models like CLIP and Flamingo. However, the raw web data is noisy, and existing
filtering methods to reduce noise often come at the expense of data diversity.
Our work focuses on caption quality as one major source of noise, and studies
how generated captions can increase the utility of web-scraped datapoints with
nondescript text. Through exploring different mixing strategies for raw and
generated captions, we outperform the best filtering method proposed by the
DataComp benchmark by 2% on ImageNet and 4% on average across 38 tasks, given a
candidate pool of 128M image-text pairs. Our best approach is also 2x better at
Flickr and MS-COCO retrieval. We then analyze what makes synthetic captions an
effective source of text supervision. In experimenting with different image
captioning models, we also demonstrate that the performance of a model on
standard image captioning benchmarks (e.g., NoCaps CIDEr) is not a reliable
indicator of the utility of the captions it generates for multimodal training.
Finally, our experiments with using generated captions at DataComp's large
scale (1.28B image-text pairs) offer insights into the limitations of synthetic
text, as well as the importance of image curation with increasing training data
quantity.
Generated captions improve the utility of web-scraped image-text datasets by reducing noise without compromising diversity, outperforming existing filtering methods across multiple benchmarks and tasks.
vision-language models CLIP Flamingo caption quality web-scraped datapoints generated captions ImageNet Flickr MS-COCO retrieval DataComp benchmark NoCaps CIDEr multimodal training image curation