title: Improved Distribution Matching Distillation for Fast Image Synthesis
summary: Recent approaches have shown promises distilling diffusion models into
  efficient one-step generators. Among them, Distribution Matching Distillation
  (DMD) produces one-step generators that match their teacher in distribution,
  without enforcing a one-to-one correspondence with the sampling trajectories of
  their teachers. However, to ensure stable training, DMD requires an additional
  regression loss computed using a large set of noise-image pairs generated by
  the teacher with many steps of a deterministic sampler. This is costly for
  large-scale text-to-image synthesis and limits the student's quality, tying it
  too closely to the teacher's original sampling paths. We introduce DMD2, a set
  of techniques that lift this limitation and improve DMD training. First, we
  eliminate the regression loss and the need for expensive dataset construction.
  We show that the resulting instability is due to the fake critic not estimating
  the distribution of generated samples accurately and propose a two time-scale
  update rule as a remedy. Second, we integrate a GAN loss into the distillation
  procedure, discriminating between generated samples and real images. This lets
  us train the student model on real data, mitigating the imperfect real score
  estimation from the teacher model, and enhancing quality. Lastly, we modify the
  training procedure to enable multi-step sampling. We identify and address the
  training-inference input mismatch problem in this setting, by simulating
  inference-time generator samples during training time. Taken together, our
  improvements set new benchmarks in one-step image generation, with FID scores
  of 1.28 on ImageNet-64x64 and 8.35 on zero-shot COCO 2014, surpassing the
  original teacher despite a 500X reduction in inference cost. Further, we show
  our approach can generate megapixel images by distilling SDXL, demonstrating
  exceptional visual quality among few-step methods.
paper_authors._id: 665016c254eda33d5406cde7
paper_authors.name: Tianwei Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:28:18.431Z
paper_authors._id: 63363b864067f020756275b7
paper_authors.avatarUrl: /avatars/bf18f9f76183f81589a80b3d0215f1a5.svg
paper_authors.fullname: Tianwei Yin
paper_authors.type: user
paper_authors.user: tianweiy
paper_authors._id: 665016c254eda33d5406cde8
paper_authors.name: Michaël Gharbi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:28:27.247Z
paper_authors._id: 658161ad208369a9403ca9a1
paper_authors.avatarUrl: /avatars/e2324ab2a712100681b0b8127305212b.svg
paper_authors.fullname: Michael Gharbi
paper_authors.type: user
paper_authors.user: mgharbi
paper_authors._id: 665016c254eda33d5406cde9
paper_authors.name: Taesung Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:28:57.967Z
paper_authors._id: 637563c59ef4e3b87a60d503
paper_authors.avatarUrl: /avatars/8264de115e6305e4720c78fc7dce7f8d.svg
paper_authors.fullname: Taesung Park
paper_authors.type: user
paper_authors.user: taesungp
paper_authors._id: 665016c254eda33d5406cdea
paper_authors.name: Richard Zhang
paper_authors._id: 665016c254eda33d5406cdeb
paper_authors.name: Eli Shechtman
paper_authors._id: 665016c254eda33d5406cdec
paper_authors.name: Fredo Durand
paper_authors._id: 665016c254eda33d5406cded
paper_authors.name: William T. Freeman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:29:29.740Z
paper_authors._id: 654e866a85a5e608059a9b4f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/O0T97EupMHH_iXcv1xYFr.png
paper_authors.fullname: William Freeman
paper_authors.type: user
paper_authors.user: mrpuppt
paper_title: Improved Distribution Matching Distillation for Fast Image Synthesis
paper_summary: Recent approaches have shown promises distilling diffusion models into
  efficient one-step generators. Among them, Distribution Matching Distillation
  (DMD) produces one-step generators that match their teacher in distribution,
  without enforcing a one-to-one correspondence with the sampling trajectories of
  their teachers. However, to ensure stable training, DMD requires an additional
  regression loss computed using a large set of noise-image pairs generated by
  the teacher with many steps of a deterministic sampler. This is costly for
  large-scale text-to-image synthesis and limits the student's quality, tying it
  too closely to the teacher's original sampling paths. We introduce DMD2, a set
  of techniques that lift this limitation and improve DMD training. First, we
  eliminate the regression loss and the need for expensive dataset construction.
  We show that the resulting instability is due to the fake critic not estimating
  the distribution of generated samples accurately and propose a two time-scale
  update rule as a remedy. Second, we integrate a GAN loss into the distillation
  procedure, discriminating between generated samples and real images. This lets
  us train the student model on real data, mitigating the imperfect real score
  estimation from the teacher model, and enhancing quality. Lastly, we modify the
  training procedure to enable multi-step sampling. We identify and address the
  training-inference input mismatch problem in this setting, by simulating
  inference-time generator samples during training time. Taken together, our
  improvements set new benchmarks in one-step image generation, with FID scores
  of 1.28 on ImageNet-64x64 and 8.35 on zero-shot COCO 2014, surpassing the
  original teacher despite a 500X reduction in inference cost. Further, we show
  our approach can generate megapixel images by distilling SDXL, demonstrating
  exceptional visual quality among few-step methods.
paper_ai_summary: DMD2 enhances one-step image generation by removing regression loss, integrating GAN loss, and enabling multi-step sampling, setting new benchmarks and achieving exceptional visual quality.
paper_ai_keywords: diffusion models Distribution Matching Distillation (DMD) one-step generators noise-image pairs deterministic sampler two time-scale update rule GAN loss multi-step sampling training-inference input mismatch FID scores ImageNet-64x64 COCO 2014 megapixel images SDXL