title: Scaling Laws of Synthetic Images for Model Training ... for Now
summary: Recent significant advances in text-to-image models unlock the possibility of
  training vision systems using synthetic images, potentially overcoming the
  difficulty of collecting curated data at scale. It is unclear, however, how
  these models behave at scale, as more synthetic data is added to the training
  set. In this paper we study the scaling laws of synthetic images generated by
  state of the art text-to-image models, for the training of supervised models:
  image classifiers with label supervision, and CLIP with language supervision.
  We identify several factors, including text prompts, classifier-free guidance
  scale, and types of text-to-image models, that significantly affect scaling
  behavior. After tuning these factors, we observe that synthetic images
  demonstrate a scaling trend similar to, but slightly less effective than, real
  images in CLIP training, while they significantly underperform in scaling when
  training supervised image classifiers. Our analysis indicates that the main
  reason for this underperformance is the inability of off-the-shelf
  text-to-image models to generate certain concepts, a limitation that
  significantly impairs the training of image classifiers. Our findings also
  suggest that scaling synthetic data can be particularly effective in scenarios
  such as: (1) when there is a limited supply of real images for a supervised
  problem (e.g., fewer than 0.5 million images in ImageNet), (2) when the
  evaluation dataset diverges significantly from the training data, indicating
  the out-of-distribution scenario, or (3) when synthetic data is used in
  conjunction with real images, as demonstrated in the training of CLIP models.
paper_authors._id: 65729776abb25ed8aef5aee0
paper_authors.name: Lijie Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T11:16:13.355Z
paper_authors._id: 63f87e56df053017d1acbaa1
paper_authors.avatarUrl: /avatars/afef72d7ee6eb005a040850006b36115.svg
paper_authors.fullname: lijiefan
paper_authors.type: user
paper_authors.user: ljf02
paper_authors._id: 65729776abb25ed8aef5aee1
paper_authors.name: Kaifeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T11:14:45.164Z
paper_authors._id: 64e4a56bebeb049d7a742f46
paper_authors.avatarUrl: /avatars/06a4d73092631cc10e6cda5c99fd28c8.svg
paper_authors.fullname: KAIFENG CHEN
paper_authors.type: user
paper_authors.user: CKF0623
paper_authors._id: 65729776abb25ed8aef5aee2
paper_authors.name: Dilip Krishnan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T11:15:11.325Z
paper_authors._id: 6501603c12c1442d99618aa5
paper_authors.avatarUrl: /avatars/9c3d04da7fdb8b13b00b4486b75c9788.svg
paper_authors.fullname: Dilip Krishnan
paper_authors.type: user
paper_authors.user: dilipk8
paper_authors._id: 65729776abb25ed8aef5aee3
paper_authors.name: Dina Katabi
paper_authors._id: 65729776abb25ed8aef5aee4
paper_authors.name: Phillip Isola
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T11:15:30.164Z
paper_authors._id: 623cb1f3ca87989bbf797eee
paper_authors.avatarUrl: /avatars/13fbd5b123b8676561751f820317600c.svg
paper_authors.fullname: Phillip Isola
paper_authors.type: user
paper_authors.user: phillipi
paper_authors._id: 65729776abb25ed8aef5aee5
paper_authors.name: Yonglong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T11:15:37.711Z
paper_authors._id: 64793086ac77bb7603842456
paper_authors.avatarUrl: /avatars/5891292138d372a9cc063f86c08b8174.svg
paper_authors.fullname: Yonglong Tian
paper_authors.type: user
paper_authors.user: tyl5566
paper_title: Scaling Laws of Synthetic Images for Model Training ... for Now
paper_summary: Recent significant advances in text-to-image models unlock the possibility of
  training vision systems using synthetic images, potentially overcoming the
  difficulty of collecting curated data at scale. It is unclear, however, how
  these models behave at scale, as more synthetic data is added to the training
  set. In this paper we study the scaling laws of synthetic images generated by
  state of the art text-to-image models, for the training of supervised models:
  image classifiers with label supervision, and CLIP with language supervision.
  We identify several factors, including text prompts, classifier-free guidance
  scale, and types of text-to-image models, that significantly affect scaling
  behavior. After tuning these factors, we observe that synthetic images
  demonstrate a scaling trend similar to, but slightly less effective than, real
  images in CLIP training, while they significantly underperform in scaling when
  training supervised image classifiers. Our analysis indicates that the main
  reason for this underperformance is the inability of off-the-shelf
  text-to-image models to generate certain concepts, a limitation that
  significantly impairs the training of image classifiers. Our findings also
  suggest that scaling synthetic data can be particularly effective in scenarios
  such as: (1) when there is a limited supply of real images for a supervised
  problem (e.g., fewer than 0.5 million images in ImageNet), (2) when the
  evaluation dataset diverges significantly from the training data, indicating
  the out-of-distribution scenario, or (3) when synthetic data is used in
  conjunction with real images, as demonstrated in the training of CLIP models.
paper_ai_summary: Synthetic images from text-to-image models scale differently than real images, performing well in CLIP training but underperforming in supervised image classifiers due to limitations in generating certain concepts.
paper_ai_keywords: text-to-image models scaling laws image classifiers label supervision CLIP language supervision classifier-free guidance synthetic images off-the-shelf text-to-image models out-of-distribution scenario