title: Immiscible Diffusion: Accelerating Diffusion Training with Noise
    Assignment
summary: In this paper, we point out suboptimal noise-data mapping leads to slow
  training of diffusion models. During diffusion training, current methods
  diffuse each image across the entire noise space, resulting in a mixture of all
  images at every point in the noise layer. We emphasize that this random mixture
  of noise-data mapping complicates the optimization of the denoising function in
  diffusion models. Drawing inspiration from the immiscible phenomenon in
  physics, we propose Immiscible Diffusion, a simple and effective method to
  improve the random mixture of noise-data mapping. In physics, miscibility can
  vary according to various intermolecular forces. Thus, immiscibility means that
  the mixing of the molecular sources is distinguishable. Inspired by this, we
  propose an assignment-then-diffusion training strategy. Specifically, prior to
  diffusing the image data into noise, we assign diffusion target noise for the
  image data by minimizing the total image-noise pair distance in a mini-batch.
  The assignment functions analogously to external forces to separate the
  diffuse-able areas of images, thus mitigating the inherent difficulties in
  diffusion training. Our approach is remarkably simple, requiring only one line
  of code to restrict the diffuse-able area for each image while preserving the
  Gaussian distribution of noise. This ensures that each image is projected only
  to nearby noise. To address the high complexity of the assignment algorithm, we
  employ a quantized-assignment method to reduce the computational overhead to a
  negligible level. Experiments demonstrate that our method achieve up to 3x
  faster training for consistency models and DDIM on the CIFAR dataset, and up to
  1.3x faster on CelebA datasets for consistency models. Besides, we conduct
  thorough analysis about the Immiscible Diffusion, which sheds lights on how it
  improves diffusion training speed while improving the fidelity.
paper_authors._id: 66736978fad1f4ec2b3cb31d
paper_authors.name: Yiheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T07:00:34.345Z
paper_authors._id: 65de6eacbeffeb39ba45a189
paper_authors.avatarUrl: /avatars/fad81b9b9fadce96ba11d44d50366fb7.svg
paper_authors.fullname: Yiheng Li
paper_authors.type: user
paper_authors.user: yiheng-li
paper_authors._id: 66736978fad1f4ec2b3cb31e
paper_authors.name: Heyang Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T09:47:11.977Z
paper_authors._id: 65e0905b05caf76026a622f9
paper_authors.avatarUrl: /avatars/d0ad6c94edd499645087e09583c8e41e.svg
paper_authors.fullname: Heyang Jiang
paper_authors.type: user
paper_authors.user: Firesheep
paper_authors._id: 66736978fad1f4ec2b3cb31f
paper_authors.name: Akio Kodaira
paper_authors._id: 66736978fad1f4ec2b3cb320
paper_authors.name: Masayoshi Tomizuka
paper_authors._id: 66736978fad1f4ec2b3cb321
paper_authors.name: Kurt Keutzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T09:46:59.171Z
paper_authors._id: 6251bf4b183aa4266924ad91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
paper_authors.fullname: Kurt Keutzer
paper_authors.type: user
paper_authors.user: kurtkeutzer
paper_authors._id: 66736978fad1f4ec2b3cb322
paper_authors.name: Chenfeng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T09:46:47.304Z
paper_authors._id: 654ca71d5255ee86711b52c5
paper_authors.avatarUrl: /avatars/52bf00fd74c8db5643c4daa185c678e6.svg
paper_authors.fullname: Chenfeng Xu
paper_authors.type: user
paper_authors.user: chenfengx
paper_title: Immiscible Diffusion: Accelerating Diffusion Training with Noise
    Assignment
paper_summary: In this paper, we point out suboptimal noise-data mapping leads to slow
  training of diffusion models. During diffusion training, current methods
  diffuse each image across the entire noise space, resulting in a mixture of all
  images at every point in the noise layer. We emphasize that this random mixture
  of noise-data mapping complicates the optimization of the denoising function in
  diffusion models. Drawing inspiration from the immiscible phenomenon in
  physics, we propose Immiscible Diffusion, a simple and effective method to
  improve the random mixture of noise-data mapping. In physics, miscibility can
  vary according to various intermolecular forces. Thus, immiscibility means that
  the mixing of the molecular sources is distinguishable. Inspired by this, we
  propose an assignment-then-diffusion training strategy. Specifically, prior to
  diffusing the image data into noise, we assign diffusion target noise for the
  image data by minimizing the total image-noise pair distance in a mini-batch.
  The assignment functions analogously to external forces to separate the
  diffuse-able areas of images, thus mitigating the inherent difficulties in
  diffusion training. Our approach is remarkably simple, requiring only one line
  of code to restrict the diffuse-able area for each image while preserving the
  Gaussian distribution of noise. This ensures that each image is projected only
  to nearby noise. To address the high complexity of the assignment algorithm, we
  employ a quantized-assignment method to reduce the computational overhead to a
  negligible level. Experiments demonstrate that our method achieve up to 3x
  faster training for consistency models and DDIM on the CIFAR dataset, and up to
  1.3x faster on CelebA datasets for consistency models. Besides, we conduct
  thorough analysis about the Immiscible Diffusion, which sheds lights on how it
  improves diffusion training speed while improving the fidelity.
paper_ai_summary: Immiscible Diffusion, which assigns diffusion target noise to images based on minimizing image-noise pair distances, improves the training speed and fidelity of diffusion models.
paper_ai_keywords: noise-data mapping diffusion training denoising function immiscible assignment-then-diffusion diffusion target noise total image-noise pair distance mini-batch Gaussian distribution consistency models DDIM CIFAR dataset CelebA datasets diffusion training speed fidelity