Scaling Laws of Synthetic Images for Model Training ... for Now
Recent significant advances in text-to-image models unlock the possibility of
training vision systems using synthetic images, potentially overcoming the
difficulty of collecting curated data at scale. It is unclear, however, how
these models behave at scale, as more synthetic data is added to the training
set. In this paper we study the scaling laws of synthetic images generated by
state of the art text-to-image models, for the training of supervised models:
image classifiers with label supervision, and CLIP with language supervision.
We identify several factors, including text prompts, classifier-free guidance
scale, and types of text-to-image models, that significantly affect scaling
behavior. After tuning these factors, we observe that synthetic images
demonstrate a scaling trend similar to, but slightly less effective than, real
images in CLIP training, while they significantly underperform in scaling when
training supervised image classifiers. Our analysis indicates that the main
reason for this underperformance is the inability of off-the-shelf
text-to-image models to generate certain concepts, a limitation that
significantly impairs the training of image classifiers. Our findings also
suggest that scaling synthetic data can be particularly effective in scenarios
such as: (1) when there is a limited supply of real images for a supervised
problem (e.g., fewer than 0.5 million images in ImageNet), (2) when the
evaluation dataset diverges significantly from the training data, indicating
the out-of-distribution scenario, or (3) when synthetic data is used in
conjunction with real images, as demonstrated in the training of CLIP models.
65729776abb25ed8aef5aee0
Lijie Fan
admin_assigned
2023-12-08T11:16:13.355Z
63f87e56df053017d1acbaa1
/avatars/afef72d7ee6eb005a040850006b36115.svg
lijiefan
user
ljf02
65729776abb25ed8aef5aee1
Kaifeng Chen
admin_assigned
2023-12-08T11:14:45.164Z
64e4a56bebeb049d7a742f46
/avatars/06a4d73092631cc10e6cda5c99fd28c8.svg
KAIFENG CHEN
user
CKF0623
65729776abb25ed8aef5aee2
Dilip Krishnan
admin_assigned
2023-12-08T11:15:11.325Z
6501603c12c1442d99618aa5
/avatars/9c3d04da7fdb8b13b00b4486b75c9788.svg
Dilip Krishnan
user
dilipk8
65729776abb25ed8aef5aee3
Dina Katabi
65729776abb25ed8aef5aee4
Phillip Isola
admin_assigned
2023-12-08T11:15:30.164Z
623cb1f3ca87989bbf797eee
/avatars/13fbd5b123b8676561751f820317600c.svg
Phillip Isola
user
phillipi
65729776abb25ed8aef5aee5
Yonglong Tian
admin_assigned
2023-12-08T11:15:37.711Z
64793086ac77bb7603842456
/avatars/5891292138d372a9cc063f86c08b8174.svg
Yonglong Tian
user
tyl5566
Scaling Laws of Synthetic Images for Model Training ... for Now
Recent significant advances in text-to-image models unlock the possibility of
training vision systems using synthetic images, potentially overcoming the
difficulty of collecting curated data at scale. It is unclear, however, how
these models behave at scale, as more synthetic data is added to the training
set. In this paper we study the scaling laws of synthetic images generated by
state of the art text-to-image models, for the training of supervised models:
image classifiers with label supervision, and CLIP with language supervision.
We identify several factors, including text prompts, classifier-free guidance
scale, and types of text-to-image models, that significantly affect scaling
behavior. After tuning these factors, we observe that synthetic images
demonstrate a scaling trend similar to, but slightly less effective than, real
images in CLIP training, while they significantly underperform in scaling when
training supervised image classifiers. Our analysis indicates that the main
reason for this underperformance is the inability of off-the-shelf
text-to-image models to generate certain concepts, a limitation that
significantly impairs the training of image classifiers. Our findings also
suggest that scaling synthetic data can be particularly effective in scenarios
such as: (1) when there is a limited supply of real images for a supervised
problem (e.g., fewer than 0.5 million images in ImageNet), (2) when the
evaluation dataset diverges significantly from the training data, indicating
the out-of-distribution scenario, or (3) when synthetic data is used in
conjunction with real images, as demonstrated in the training of CLIP models.
Synthetic images from text-to-image models scale differently than real images, performing well in CLIP training but underperforming in supervised image classifiers due to limitations in generating certain concepts.
text-to-image models scaling laws image classifiers label supervision CLIP language supervision classifier-free guidance synthetic images off-the-shelf text-to-image models out-of-distribution scenario