Learning Vision from Models Rivals Learning Vision from Data
We introduce SynCLR, a novel approach for learning visual representations
exclusively from synthetic images and synthetic captions, without any real
data. We synthesize a large dataset of image captions using LLMs, then use an
off-the-shelf text-to-image model to generate multiple images corresponding to
each synthetic caption. We perform visual representation learning on these
synthetic images via contrastive learning, treating images sharing the same
caption as positive pairs. The resulting representations transfer well to many
downstream tasks, competing favorably with other general-purpose visual
representation learners such as CLIP and DINO v2 in image classification tasks.
Furthermore, in dense prediction tasks such as semantic segmentation, SynCLR
outperforms previous self-supervised methods by a significant margin, e.g.,
improving over MAE and iBOT by 6.2 and 4.3 mIoU on ADE20k for ViT-B/16.
659235e3c0b1372b2e0827c8
Yonglong Tian
claimed_verified
2024-01-03T12:35:04.367Z
64793086ac77bb7603842456
/avatars/5891292138d372a9cc063f86c08b8174.svg
Yonglong Tian
user
tyl5566
659235e3c0b1372b2e0827c9
Lijie Fan
659235e3c0b1372b2e0827ca
Kaifeng Chen
659235e3c0b1372b2e0827cb
Dina Katabi
659235e3c0b1372b2e0827cc
Dilip Krishnan
659235e3c0b1372b2e0827cd
Phillip Isola
Learning Vision from Models Rivals Learning Vision from Data
We introduce SynCLR, a novel approach for learning visual representations
exclusively from synthetic images and synthetic captions, without any real
data. We synthesize a large dataset of image captions using LLMs, then use an
off-the-shelf text-to-image model to generate multiple images corresponding to
each synthetic caption. We perform visual representation learning on these
synthetic images via contrastive learning, treating images sharing the same
caption as positive pairs. The resulting representations transfer well to many
downstream tasks, competing favorably with other general-purpose visual
representation learners such as CLIP and DINO v2 in image classification tasks.
Furthermore, in dense prediction tasks such as semantic segmentation, SynCLR
outperforms previous self-supervised methods by a significant margin, e.g.,
improving over MAE and iBOT by 6.2 and 4.3 mIoU on ADE20k for ViT-B/16.
SynCLR learns visual representations from synthetic images and captions using contrastive learning, outperforming other methods in image classification and semantic segmentation tasks.
SynCLR contrastive learning visual representation learning synthetic images synthetic captions CLIP DINO v2 semantic segmentation MAE iBOT ADE20k ViT-B/16