Scaling Pre-training to One Hundred Billion Data for Vision Language
  Models
We provide an empirical investigation of the potential of pre-training
vision-language models on an unprecedented scale: 100 billion examples. We find
that model performance tends to saturate at this scale on many common
Western-centric classification and retrieval benchmarks, such as COCO Captions.
Nevertheless, tasks of cultural diversity achieve more substantial gains from
the 100-billion scale web data, thanks to its coverage of long-tail concepts.
Furthermore, we analyze the model's multilinguality and show gains in
low-resource languages as well. In addition, we observe that reducing the size
of the pretraining dataset via quality filters like using CLIP, typically used
to enhance performance, may inadvertently reduce the cultural diversity
represented even in large-scale datasets. Our results highlight that while
traditional benchmarks may not benefit significantly from scaling noisy, raw
web data to 100 billion examples, this data scale is vital for building truly
inclusive multimodal systems.
67ac1d68c29356f92ed772c5
Xiao Wang
67ac1d68c29356f92ed772c6
Ibrahim Alabdulmohsin
67ac1d68c29356f92ed772c7
Daniel Salz
67ac1d68c29356f92ed772c8
Zhe Li
67ac1d68c29356f92ed772c9
Keran Rong
claimed_verified
2025-02-13T08:25:28.934Z
648a9083abcf427a9a498679
/avatars/e32d413ce0a48d83f95e29c11a8a8ae8.svg
Keran 
user
Keeera
67ac1d68c29356f92ed772ca
Xiaohua Zhai
Scaling Pre-training to One Hundred Billion Data for Vision Language
  Models
We provide an empirical investigation of the potential of pre-training
vision-language models on an unprecedented scale: 100 billion examples. We find
that model performance tends to saturate at this scale on many common
Western-centric classification and retrieval benchmarks, such as COCO Captions.
Nevertheless, tasks of cultural diversity achieve more substantial gains from
the 100-billion scale web data, thanks to its coverage of long-tail concepts.
Furthermore, we analyze the model's multilinguality and show gains in
low-resource languages as well. In addition, we observe that reducing the size
of the pretraining dataset via quality filters like using CLIP, typically used
to enhance performance, may inadvertently reduce the cultural diversity
represented even in large-scale datasets. Our results highlight that while
traditional benchmarks may not benefit significantly from scaling noisy, raw
web data to 100 billion examples, this data scale is vital for building truly
inclusive multimodal systems.
Pre-training vision-language models on a 100-billion example dataset benefits cultural diversity and multilingual performance more than traditional benchmarks.
vision-language models pre-training COCO Captions cultural diversity low-resource languages quality filters CLIP multimodal systems