title: Scaling Pre-training to One Hundred Billion Data for Vision Language
    Models
summary: We provide an empirical investigation of the potential of pre-training
  vision-language models on an unprecedented scale: 100 billion examples. We find
  that model performance tends to saturate at this scale on many common
  Western-centric classification and retrieval benchmarks, such as COCO Captions.
  Nevertheless, tasks of cultural diversity achieve more substantial gains from
  the 100-billion scale web data, thanks to its coverage of long-tail concepts.
  Furthermore, we analyze the model's multilinguality and show gains in
  low-resource languages as well. In addition, we observe that reducing the size
  of the pretraining dataset via quality filters like using CLIP, typically used
  to enhance performance, may inadvertently reduce the cultural diversity
  represented even in large-scale datasets. Our results highlight that while
  traditional benchmarks may not benefit significantly from scaling noisy, raw
  web data to 100 billion examples, this data scale is vital for building truly
  inclusive multimodal systems.
paper_authors._id: 67ac1d68c29356f92ed772c5
paper_authors.name: Xiao Wang
paper_authors._id: 67ac1d68c29356f92ed772c6
paper_authors.name: Ibrahim Alabdulmohsin
paper_authors._id: 67ac1d68c29356f92ed772c7
paper_authors.name: Daniel Salz
paper_authors._id: 67ac1d68c29356f92ed772c8
paper_authors.name: Zhe Li
paper_authors._id: 67ac1d68c29356f92ed772c9
paper_authors.name: Keran Rong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:25:28.934Z
paper_authors._id: 648a9083abcf427a9a498679
paper_authors.avatarUrl: /avatars/e32d413ce0a48d83f95e29c11a8a8ae8.svg
paper_authors.fullname: Keran 
paper_authors.type: user
paper_authors.user: Keeera
paper_authors._id: 67ac1d68c29356f92ed772ca
paper_authors.name: Xiaohua Zhai
paper_title: Scaling Pre-training to One Hundred Billion Data for Vision Language
    Models
paper_summary: We provide an empirical investigation of the potential of pre-training
  vision-language models on an unprecedented scale: 100 billion examples. We find
  that model performance tends to saturate at this scale on many common
  Western-centric classification and retrieval benchmarks, such as COCO Captions.
  Nevertheless, tasks of cultural diversity achieve more substantial gains from
  the 100-billion scale web data, thanks to its coverage of long-tail concepts.
  Furthermore, we analyze the model's multilinguality and show gains in
  low-resource languages as well. In addition, we observe that reducing the size
  of the pretraining dataset via quality filters like using CLIP, typically used
  to enhance performance, may inadvertently reduce the cultural diversity
  represented even in large-scale datasets. Our results highlight that while
  traditional benchmarks may not benefit significantly from scaling noisy, raw
  web data to 100 billion examples, this data scale is vital for building truly
  inclusive multimodal systems.
paper_ai_keywords: pre-training vision-language models COCO Captions cultural diversity long-tail concepts multilinguality low-resource languages CLIP