title: Analyzing and Improving the Training Dynamics of Diffusion Models
summary: Diffusion models currently dominate the field of data-driven image synthesis
  with their unparalleled scaling to large datasets. In this paper, we identify
  and rectify several causes for uneven and ineffective training in the popular
  ADM diffusion model architecture, without altering its high-level structure.
  Observing uncontrolled magnitude changes and imbalances in both the network
  activations and weights over the course of training, we redesign the network
  layers to preserve activation, weight, and update magnitudes on expectation. We
  find that systematic application of this philosophy eliminates the observed
  drifts and imbalances, resulting in considerably better networks at equal
  computational complexity. Our modifications improve the previous record FID of
  2.41 in ImageNet-512 synthesis to 1.81, achieved using fast deterministic
  sampling.
    As an independent contribution, we present a method for setting the
  exponential moving average (EMA) parameters post-hoc, i.e., after completing
  the training run. This allows precise tuning of EMA length without the cost of
  performing several training runs, and reveals its surprising interactions with
  network architecture, training time, and guidance.
paper_authors._id: 656fe86f2f058b368c2b06e7
paper_authors.name: Tero Karras
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-30T13:22:25.068Z
paper_authors._id: 66587947b7d313010028ac02
paper_authors.avatarUrl: /avatars/cb95f524dd1c80aa6ec8af977c5f2606.svg
paper_authors.fullname: Tero Karras
paper_authors.type: user
paper_authors.user: tkarras
paper_authors._id: 656fe86f2f058b368c2b06e8
paper_authors.name: Miika Aittala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T11:54:46.315Z
paper_authors._id: 630f80e2a119d49bc1e3836f
paper_authors.avatarUrl: /avatars/e0ffe1c19df64606cb30801ad20a9f43.svg
paper_authors.fullname: Miika Aittala
paper_authors.type: user
paper_authors.user: miika
paper_authors._id: 656fe86f2f058b368c2b06e9
paper_authors.name: Jaakko Lehtinen
paper_authors._id: 656fe86f2f058b368c2b06ea
paper_authors.name: Janne Hellsten
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T11:54:55.806Z
paper_authors._id: 63dc13a562a61013a7509c17
paper_authors.avatarUrl: /avatars/b537cf5459a452c4e61b2a8f404c18db.svg
paper_authors.fullname: Janne Hellsten
paper_authors.type: user
paper_authors.user: Jlou
paper_authors._id: 656fe86f2f058b368c2b06eb
paper_authors.name: Timo Aila
paper_authors._id: 656fe86f2f058b368c2b06ec
paper_authors.name: Samuli Laine
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-12T11:41:19.367Z
paper_authors._id: 67ac68cda78e0ee095322704
paper_authors.avatarUrl: /avatars/7cc9fbd7ba8c02e48b01e56bbccea01e.svg
paper_authors.fullname: Samuli Laine
paper_authors.type: user
paper_authors.user: slaine2
paper_title: Analyzing and Improving the Training Dynamics of Diffusion Models
paper_summary: Diffusion models currently dominate the field of data-driven image synthesis
  with their unparalleled scaling to large datasets. In this paper, we identify
  and rectify several causes for uneven and ineffective training in the popular
  ADM diffusion model architecture, without altering its high-level structure.
  Observing uncontrolled magnitude changes and imbalances in both the network
  activations and weights over the course of training, we redesign the network
  layers to preserve activation, weight, and update magnitudes on expectation. We
  find that systematic application of this philosophy eliminates the observed
  drifts and imbalances, resulting in considerably better networks at equal
  computational complexity. Our modifications improve the previous record FID of
  2.41 in ImageNet-512 synthesis to 1.81, achieved using fast deterministic
  sampling.
    As an independent contribution, we present a method for setting the
  exponential moving average (EMA) parameters post-hoc, i.e., after completing
  the training run. This allows precise tuning of EMA length without the cost of
  performing several training runs, and reveals its surprising interactions with
  network architecture, training time, and guidance.
paper_ai_summary: Modifications to network layers in the ADM diffusion model architecture improve training stability and synthesis quality, reducing FID from 2.41 to 1.81, and a novel method for post-hoc EMA parameter tuning is introduced.
paper_ai_keywords: diffusion models ADM diffusion model network activations weights fast deterministic sampling exponential moving average (EMA)