title: Understanding and Mitigating Copying in Diffusion Models
summary: Images generated by diffusion models like Stable Diffusion are increasingly
  widespread. Recent works and even lawsuits have shown that these models are
  prone to replicating their training data, unbeknownst to the user. In this
  paper, we first analyze this memorization problem in text-to-image diffusion
  models. While it is widely believed that duplicated images in the training set
  are responsible for content replication at inference time, we observe that the
  text conditioning of the model plays a similarly important role. In fact, we
  see in our experiments that data replication often does not happen for
  unconditional models, while it is common in the text-conditional case.
  Motivated by our findings, we then propose several techniques for reducing data
  replication at both training and inference time by randomizing and augmenting
  image captions in the training set.
paper_authors._id: 6477f76f6e6c7ac608c503de
paper_authors.name: Gowthami Somepalli
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-07T11:21:04.559Z
paper_authors._id: 62ae853a4b5d6165083a12f8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62ae853a4b5d6165083a12f8/3_Q1yB3Qfpd9PngzzBlgZ.jpeg
paper_authors.fullname: Gowthami Somepalli
paper_authors.type: user
paper_authors.user: somepago
paper_authors._id: 6477f76f6e6c7ac608c503df
paper_authors.name: Vasu Singla
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-01T16:29:20.857Z
paper_authors._id: 6352f7f90422304c3e7bf098
paper_authors.avatarUrl: /avatars/90bff66d85240415bb43d02c329ffbca.svg
paper_authors.fullname: Vasu Singla
paper_authors.type: user
paper_authors.user: vasusingla
paper_authors._id: 6477f76f6e6c7ac608c503e0
paper_authors.name: Micah Goldblum
paper_authors._id: 6477f76f6e6c7ac608c503e1
paper_authors.name: Jonas Geiping
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T12:27:32.150Z
paper_authors._id: 63d86dbf3130cadcaf8bdd11
paper_authors.avatarUrl: /avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
paper_authors.fullname: Jonas Geiping
paper_authors.type: user
paper_authors.user: JonasGeiping
paper_authors._id: 6477f76f6e6c7ac608c503e2
paper_authors.name: Tom Goldstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T12:27:05.749Z
paper_authors._id: 6381ca7d65dc156aba0b933d
paper_authors.avatarUrl: /avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
paper_authors.fullname: Tom Goldstein
paper_authors.type: user
paper_authors.user: tomgoldstein
paper_title: Understanding and Mitigating Copying in Diffusion Models
paper_summary: Images generated by diffusion models like Stable Diffusion are increasingly
  widespread. Recent works and even lawsuits have shown that these models are
  prone to replicating their training data, unbeknownst to the user. In this
  paper, we first analyze this memorization problem in text-to-image diffusion
  models. While it is widely believed that duplicated images in the training set
  are responsible for content replication at inference time, we observe that the
  text conditioning of the model plays a similarly important role. In fact, we
  see in our experiments that data replication often does not happen for
  unconditional models, while it is common in the text-conditional case.
  Motivated by our findings, we then propose several techniques for reducing data
  replication at both training and inference time by randomizing and augmenting
  image captions in the training set.
paper_ai_summary: Text-to-image diffusion models, such as Stable Diffusion, frequently replicate training data during inference, especially with text conditioning; proposed techniques address this by randomizing and augmenting image captions.
paper_ai_keywords: diffusion models text-to-image training data replication inference text conditioning unconditional models image captions randomization augmentation