title: Learning Vision from Models Rivals Learning Vision from Data
summary: We introduce SynCLR, a novel approach for learning visual representations
  exclusively from synthetic images and synthetic captions, without any real
  data. We synthesize a large dataset of image captions using LLMs, then use an
  off-the-shelf text-to-image model to generate multiple images corresponding to
  each synthetic caption. We perform visual representation learning on these
  synthetic images via contrastive learning, treating images sharing the same
  caption as positive pairs. The resulting representations transfer well to many
  downstream tasks, competing favorably with other general-purpose visual
  representation learners such as CLIP and DINO v2 in image classification tasks.
  Furthermore, in dense prediction tasks such as semantic segmentation, SynCLR
  outperforms previous self-supervised methods by a significant margin, e.g.,
  improving over MAE and iBOT by 6.2 and 4.3 mIoU on ADE20k for ViT-B/16.
paper_authors._id: 659235e3c0b1372b2e0827c8
paper_authors.name: Yonglong Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-03T12:35:04.367Z
paper_authors._id: 64793086ac77bb7603842456
paper_authors.avatarUrl: /avatars/5891292138d372a9cc063f86c08b8174.svg
paper_authors.fullname: Yonglong Tian
paper_authors.type: user
paper_authors.user: tyl5566
paper_authors._id: 659235e3c0b1372b2e0827c9
paper_authors.name: Lijie Fan
paper_authors._id: 659235e3c0b1372b2e0827ca
paper_authors.name: Kaifeng Chen
paper_authors._id: 659235e3c0b1372b2e0827cb
paper_authors.name: Dina Katabi
paper_authors._id: 659235e3c0b1372b2e0827cc
paper_authors.name: Dilip Krishnan
paper_authors._id: 659235e3c0b1372b2e0827cd
paper_authors.name: Phillip Isola
paper_title: Learning Vision from Models Rivals Learning Vision from Data
paper_summary: We introduce SynCLR, a novel approach for learning visual representations
  exclusively from synthetic images and synthetic captions, without any real
  data. We synthesize a large dataset of image captions using LLMs, then use an
  off-the-shelf text-to-image model to generate multiple images corresponding to
  each synthetic caption. We perform visual representation learning on these
  synthetic images via contrastive learning, treating images sharing the same
  caption as positive pairs. The resulting representations transfer well to many
  downstream tasks, competing favorably with other general-purpose visual
  representation learners such as CLIP and DINO v2 in image classification tasks.
  Furthermore, in dense prediction tasks such as semantic segmentation, SynCLR
  outperforms previous self-supervised methods by a significant margin, e.g.,
  improving over MAE and iBOT by 6.2 and 4.3 mIoU on ADE20k for ViT-B/16.
paper_ai_summary: SynCLR learns visual representations from synthetic images and captions using contrastive learning, outperforming other methods in image classification and semantic segmentation tasks.
paper_ai_keywords: SynCLR contrastive learning visual representation learning synthetic images synthetic captions CLIP DINO v2 semantic segmentation MAE iBOT ADE20k ViT-B/16