title: CommonCanvas: An Open Diffusion Model Trained with Creative-Commons
    Images
summary: We assemble a dataset of Creative-Commons-licensed (CC) images, which we use
  to train a set of open diffusion models that are qualitatively competitive with
  Stable Diffusion 2 (SD2). This task presents two challenges: (1)
  high-resolution CC images lack the captions necessary to train text-to-image
  generative models; (2) CC images are relatively scarce. In turn, to address
  these challenges, we use an intuitive transfer learning technique to produce a
  set of high-quality synthetic captions paired with curated CC images. We then
  develop a data- and compute-efficient training recipe that requires as little
  as 3% of the LAION-2B data needed to train existing SD2 models, but obtains
  comparable quality. These results indicate that we have a sufficient number of
  CC images (~70 million) for training high-quality models. Our training recipe
  also implements a variety of optimizations that achieve ~3X training speed-ups,
  enabling rapid model iteration. We leverage this recipe to train several
  high-quality text-to-image models, which we dub the CommonCanvas family. Our
  largest model achieves comparable performance to SD2 on a human evaluation,
  despite being trained on our CC dataset that is significantly smaller than
  LAION and using synthetic captions for training. We release our models, data,
  and code at
  https://github.com/mosaicml/diffusion/blob/main/assets/common-canvas.md
paper_authors._id: 6539e114f186c8b4a8b2525c
paper_authors.name: Aaron Gokaslan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:18:55.314Z
paper_authors._id: 620573d0522e40b4a18d8763
paper_authors.avatarUrl: /avatars/9353c064ef8ccac84d0397411d38fa90.svg
paper_authors.fullname: Aaron Gokaslan
paper_authors.type: user
paper_authors.user: Skylion007
paper_authors._id: 6539e114f186c8b4a8b2525d
paper_authors.name: A. Feder Cooper
paper_authors._id: 6539e114f186c8b4a8b2525e
paper_authors.name: Jasmine Collins
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:19:12.906Z
paper_authors._id: 64b780de324038715946a2bc
paper_authors.avatarUrl: /avatars/4c3f695d7d0232a8d2be70b2ca222910.svg
paper_authors.fullname: Jasmine Collins
paper_authors.type: user
paper_authors.user: 3coepugOsubcba
paper_authors._id: 6539e114f186c8b4a8b2525f
paper_authors.name: Landan Seguin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-27T09:12:46.966Z
paper_authors._id: 63278bbc28396be9bfec8dbb
paper_authors.avatarUrl: /avatars/75ff28af26bbee4e243ed33ba4856ddb.svg
paper_authors.fullname: Landan Seguin
paper_authors.type: user
paper_authors.user: Landan
paper_authors._id: 6539e114f186c8b4a8b25260
paper_authors.name: Austin Jacobson
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-05T11:31:54.810Z
paper_authors._id: 630410a1fc783bfc7444aca2
paper_authors.avatarUrl: /avatars/d393466560bff8ca45c420a1315b75b6.svg
paper_authors.fullname: austin
paper_authors.type: user
paper_authors.user: austinj
paper_authors._id: 6539e114f186c8b4a8b25261
paper_authors.name: Mihir Patel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:20:25.947Z
paper_authors._id: 6453f6f9bc2ca42503ecf4d8
paper_authors.avatarUrl: /avatars/988f0264dc8e34aa28cae9b77336b0f1.svg
paper_authors.fullname: Mihir Patel
paper_authors.type: user
paper_authors.user: mihir-mosaic
paper_authors._id: 6539e114f186c8b4a8b25262
paper_authors.name: Jonathan Frankle
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:20:33.713Z
paper_authors._id: 62b0b7f9a596c50b1783fe9e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b0b7f9a596c50b1783fe9e/nzctJXoyiEwCnOHVJEx3_.png
paper_authors.fullname: Jonathan Frankle
paper_authors.type: user
paper_authors.user: jfrankle
paper_authors._id: 6539e114f186c8b4a8b25263
paper_authors.name: Cory Stephenson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:20:40.592Z
paper_authors._id: 638e714750a4e4beef99c802
paper_authors.avatarUrl: /avatars/73006dc9b196986511bcc9c5d7cb0fc8.svg
paper_authors.fullname: Cory Stephenson
paper_authors.type: user
paper_authors.user: CoryMosaicML
paper_authors._id: 6539e114f186c8b4a8b25264
paper_authors.name: Volodymyr Kuleshov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:20:46.782Z
paper_authors._id: 640ea1c43c82bd463ee80e19
paper_authors.avatarUrl: /avatars/3213d2e8a433a207dfb96a35f0f52a92.svg
paper_authors.fullname: Volodymyr Kuleshov
paper_authors.type: user
paper_authors.user: kuleshov
paper_title: CommonCanvas: An Open Diffusion Model Trained with Creative-Commons
    Images
paper_summary: We assemble a dataset of Creative-Commons-licensed (CC) images, which we use
  to train a set of open diffusion models that are qualitatively competitive with
  Stable Diffusion 2 (SD2). This task presents two challenges: (1)
  high-resolution CC images lack the captions necessary to train text-to-image
  generative models; (2) CC images are relatively scarce. In turn, to address
  these challenges, we use an intuitive transfer learning technique to produce a
  set of high-quality synthetic captions paired with curated CC images. We then
  develop a data- and compute-efficient training recipe that requires as little
  as 3% of the LAION-2B data needed to train existing SD2 models, but obtains
  comparable quality. These results indicate that we have a sufficient number of
  CC images (~70 million) for training high-quality models. Our training recipe
  also implements a variety of optimizations that achieve ~3X training speed-ups,
  enabling rapid model iteration. We leverage this recipe to train several
  high-quality text-to-image models, which we dub the CommonCanvas family. Our
  largest model achieves comparable performance to SD2 on a human evaluation,
  despite being trained on our CC dataset that is significantly smaller than
  LAION and using synthetic captions for training. We release our models, data,
  and code at
  https://github.com/mosaicml/diffusion/blob/main/assets/common-canvas.md
paper_ai_summary: A transfer learning technique and efficient training recipe using Creative Commons images enable high-quality text-to-image models with synthetic captions and reduced computational requirements.
paper_ai_keywords: transfer learning synthetic captions data- and compute-efficient training LAION-2B text-to-image models CommonCanvas family human evaluation