  Recently, diffusion models have emerged as powerful deep generative models,
showcasing cutting-edge performance across various applications such as image
generation, solving inverse problems, and text-to-image synthesis. These models
generate new data (e.g., images) by transforming random noise inputs through a
reverse diffusion process. In this work, we uncover a distinct and prevalent
phenomenon within diffusion models in contrast to most other generative models,
which we refer to as ``consistent model reproducibility''. To elaborate, our
extensive experiments have consistently shown that when starting with the same
initial noise input and sampling with a deterministic solver, diffusion models
tend to produce nearly identical output content. This consistency holds true
regardless of the choices of model architectures and training procedures.
Additionally, our research has unveiled that this exceptional model
reproducibility manifests in two distinct training regimes: (i) ``memorization
regime,'' characterized by a significantly overparameterized model which
attains reproducibility mainly by memorizing the training data; (ii)
``generalization regime,'' in which the model is trained on an extensive
dataset, and its reproducibility emerges with the model's generalization
capabilities. Our analysis provides theoretical justification for the model
reproducibility in ``memorization regime''. Moreover, our research reveals that
this valuable property generalizes to many variants of diffusion models,
including conditional diffusion models, diffusion models for solving inverse
problems, and fine-tuned diffusion models. A deeper understanding of this
phenomenon has the potential to yield more interpretable and controllable data
generative processes based on diffusion models.

Machine Learning;Computer Vision and Pattern Recognition