title: Scaling Laws for Robust Comparison of Open Foundation Language-Vision
    Models and Datasets
summary: In studies of transferable learning, scaling laws are obtained for various
  important foundation models to predict their properties and performance at
  larger scales. We show here how scaling law derivation can also be used for
  model and dataset comparison, allowing to decide which procedure is to be
  preferred for pre-training. For the first time, full scaling laws based on
  dense measurements across a wide span of model and samples seen scales are
  derived for two important language-vision learning procedures, CLIP and MaMMUT,
  that use either contrastive only or contrastive and captioning text generative
  loss. Ensuring sufficient prediction accuracy for held out points, we use
  derived scaling laws to compare both models, obtaining evidence for MaMMUT's
  stronger improvement with scale and better sample efficiency than standard
  CLIP. To strengthen validity of the comparison, we show scaling laws for
  various downstream tasks, classification, retrieval, and segmentation, and for
  different open datasets, DataComp, DFN and Re-LAION, observing consistently the
  same trends. We show that comparison can also be performed when deriving
  scaling laws with a constant learning rate schedule, reducing compute cost.
  Accurate derivation of scaling laws provides thus means to perform model and
  dataset comparison across scale spans, avoiding misleading conclusions based on
  measurements from single reference scales only, paving the road for systematic
  comparison and improvement of open foundation models and datasets for their
  creation. We release all the pre-trained models with their intermediate
  checkpoints, including openMaMMUT-L/14, which achieves 80.3% zero-shot
  ImageNet-1k accuracy, trained on 12.8B samples from DataComp-1.4B. Code for
  reproducing experiments in the paper and raw experiments data can be found at
  https://github.com/LAION-AI/scaling-laws-for-comparison.
paper_authors._id: 68423f5f0a25ba60aa257550
paper_authors.name: Marianna Nezhurina
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:49:11.910Z
paper_authors._id: 62500f361684e0335e527bc6
paper_authors.avatarUrl: /avatars/8d2bc4c9cfa8ea82049196431cc3ebea.svg
paper_authors.fullname: Marianna Nezhurina
paper_authors.type: user
paper_authors.user: marianna13
paper_authors._id: 68423f5f0a25ba60aa257551
paper_authors.name: Tomer Porian
paper_authors._id: 68423f5f0a25ba60aa257552
paper_authors.name: Giovanni Pucceti
paper_authors._id: 68423f5f0a25ba60aa257553
paper_authors.name: Tommie Kerssies
paper_authors._id: 68423f5f0a25ba60aa257554
paper_authors.name: Romain Beaumont
paper_authors._id: 68423f5f0a25ba60aa257555
paper_authors.name: Mehdi Cherti
paper_authors._id: 68423f5f0a25ba60aa257556
paper_authors.name: Jenia Jitsev
paper_title: Scaling Laws for Robust Comparison of Open Foundation Language-Vision
    Models and Datasets
paper_summary: In studies of transferable learning, scaling laws are obtained for various
  important foundation models to predict their properties and performance at
  larger scales. We show here how scaling law derivation can also be used for
  model and dataset comparison, allowing to decide which procedure is to be
  preferred for pre-training. For the first time, full scaling laws based on
  dense measurements across a wide span of model and samples seen scales are
  derived for two important language-vision learning procedures, CLIP and MaMMUT,
  that use either contrastive only or contrastive and captioning text generative
  loss. Ensuring sufficient prediction accuracy for held out points, we use
  derived scaling laws to compare both models, obtaining evidence for MaMMUT's
  stronger improvement with scale and better sample efficiency than standard
  CLIP. To strengthen validity of the comparison, we show scaling laws for
  various downstream tasks, classification, retrieval, and segmentation, and for
  different open datasets, DataComp, DFN and Re-LAION, observing consistently the
  same trends. We show that comparison can also be performed when deriving
  scaling laws with a constant learning rate schedule, reducing compute cost.
  Accurate derivation of scaling laws provides thus means to perform model and
  dataset comparison across scale spans, avoiding misleading conclusions based on
  measurements from single reference scales only, paving the road for systematic
  comparison and improvement of open foundation models and datasets for their
  creation. We release all the pre-trained models with their intermediate
  checkpoints, including openMaMMUT-L/14, which achieves 80.3% zero-shot
  ImageNet-1k accuracy, trained on 12.8B samples from DataComp-1.4B. Code for
  reproducing experiments in the paper and raw experiments data can be found at
  https://github.com/LAION-AI/scaling-laws-for-comparison.
paper_ai_summary: Scaling laws are derived for CLIP and MaMMUT to compare their performance and sample efficiency across different scales and datasets.
paper_ai_keywords: scaling laws transferable learning CLIP MaMMUT contrastive loss captioning text generative loss model comparison dataset comparison zero-shot accuracy DataComp DFN Re-LAION learning rate schedule