Understanding and Mitigating Copying in Diffusion Models
Images generated by diffusion models like Stable Diffusion are increasingly
widespread. Recent works and even lawsuits have shown that these models are
prone to replicating their training data, unbeknownst to the user. In this
paper, we first analyze this memorization problem in text-to-image diffusion
models. While it is widely believed that duplicated images in the training set
are responsible for content replication at inference time, we observe that the
text conditioning of the model plays a similarly important role. In fact, we
see in our experiments that data replication often does not happen for
unconditional models, while it is common in the text-conditional case.
Motivated by our findings, we then propose several techniques for reducing data
replication at both training and inference time by randomizing and augmenting
image captions in the training set.
6477f76f6e6c7ac608c503de
Gowthami Somepalli
claimed_verified
2023-11-07T11:21:04.559Z
62ae853a4b5d6165083a12f8
https://cdn-avatars.huggingface.co/v1/production/uploads/62ae853a4b5d6165083a12f8/3_Q1yB3Qfpd9PngzzBlgZ.jpeg
Gowthami Somepalli
user
somepago
6477f76f6e6c7ac608c503df
Vasu Singla
claimed_verified
2023-06-01T16:29:20.857Z
6352f7f90422304c3e7bf098
/avatars/90bff66d85240415bb43d02c329ffbca.svg
Vasu Singla
user
vasusingla
6477f76f6e6c7ac608c503e0
Micah Goldblum
6477f76f6e6c7ac608c503e1
Jonas Geiping
admin_assigned
2023-06-02T12:27:32.150Z
63d86dbf3130cadcaf8bdd11
/avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
Jonas Geiping
user
JonasGeiping
6477f76f6e6c7ac608c503e2
Tom Goldstein
admin_assigned
2023-06-02T12:27:05.749Z
6381ca7d65dc156aba0b933d
/avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
Tom Goldstein
user
tomgoldstein
Understanding and Mitigating Copying in Diffusion Models
Images generated by diffusion models like Stable Diffusion are increasingly
widespread. Recent works and even lawsuits have shown that these models are
prone to replicating their training data, unbeknownst to the user. In this
paper, we first analyze this memorization problem in text-to-image diffusion
models. While it is widely believed that duplicated images in the training set
are responsible for content replication at inference time, we observe that the
text conditioning of the model plays a similarly important role. In fact, we
see in our experiments that data replication often does not happen for
unconditional models, while it is common in the text-conditional case.
Motivated by our findings, we then propose several techniques for reducing data
replication at both training and inference time by randomizing and augmenting
image captions in the training set.
Text-to-image diffusion models, such as Stable Diffusion, frequently replicate training data during inference, especially with text conditioning; proposed techniques address this by randomizing and augmenting image captions.
diffusion models text-to-image training data replication inference text conditioning unconditional models image captions randomization augmentation