One-step Diffusion Models with f-Divergence Distribution Matching
Sampling from diffusion models involves a slow iterative process that hinders
their practical deployment, especially for interactive applications. To
accelerate generation speed, recent approaches distill a multi-step diffusion
model into a single-step student generator via variational score distillation,
which matches the distribution of samples generated by the student to the
teacher's distribution. However, these approaches use the reverse
Kullback-Leibler (KL) divergence for distribution matching which is known to be
mode seeking. In this paper, we generalize the distribution matching approach
using a novel f-divergence minimization framework, termed f-distill, that
covers different divergences with different trade-offs in terms of mode
coverage and training variance. We derive the gradient of the f-divergence
between the teacher and student distributions and show that it is expressed as
the product of their score differences and a weighting function determined by
their density ratio. This weighting function naturally emphasizes samples with
higher density in the teacher distribution, when using a less mode-seeking
divergence. We observe that the popular variational score distillation approach
using the reverse-KL divergence is a special case within our framework.
Empirically, we demonstrate that alternative f-divergences, such as
forward-KL and Jensen-Shannon divergences, outperform the current best
variational score distillation methods across image generation tasks. In
particular, when using Jensen-Shannon divergence, f-distill achieves current
state-of-the-art one-step generation performance on ImageNet64 and zero-shot
text-to-image generation on MS-COCO. Project page:
https://research.nvidia.com/labs/genair/f-distill
67bbe67c7727595ca5979d2a
Yilun Xu
admin_assigned
2025-02-24T15:54:56.422Z
649c37de5ffe05267a105fe8
/avatars/4f262272e6222f879c6c0fedfa2e5861.svg
Yilun Xu
user
AaronXyl
67bbe67c7727595ca5979d2b
Weili Nie
admin_assigned
2025-02-24T15:55:03.232Z
64c1a69e226e016da8450ae2
/avatars/54c161e8b8543244ed13cbe47017624e.svg
Weili Nie
user
xiaoli08
67bbe67c7727595ca5979d2c
Arash Vahdat
admin_assigned
2025-02-24T15:55:08.446Z
66727eb8f1bef807e01c6164
/avatars/ba8edee083cfd6124b82e2a776f0fb43.svg
Arash vahdat
user
ArashAVN
One-step Diffusion Models with f-Divergence Distribution Matching
Sampling from diffusion models involves a slow iterative process that hinders
their practical deployment, especially for interactive applications. To
accelerate generation speed, recent approaches distill a multi-step diffusion
model into a single-step student generator via variational score distillation,
which matches the distribution of samples generated by the student to the
teacher's distribution. However, these approaches use the reverse
Kullback-Leibler (KL) divergence for distribution matching which is known to be
mode seeking. In this paper, we generalize the distribution matching approach
using a novel f-divergence minimization framework, termed f-distill, that
covers different divergences with different trade-offs in terms of mode
coverage and training variance. We derive the gradient of the f-divergence
between the teacher and student distributions and show that it is expressed as
the product of their score differences and a weighting function determined by
their density ratio. This weighting function naturally emphasizes samples with
higher density in the teacher distribution, when using a less mode-seeking
divergence. We observe that the popular variational score distillation approach
using the reverse-KL divergence is a special case within our framework.
Empirically, we demonstrate that alternative f-divergences, such as
forward-KL and Jensen-Shannon divergences, outperform the current best
variational score distillation methods across image generation tasks. In
particular, when using Jensen-Shannon divergence, f-distill achieves current
state-of-the-art one-step generation performance on ImageNet64 and zero-shot
text-to-image generation on MS-COCO. Project page:
https://research.nvidia.com/labs/genair/f-distill
A novel distribution matching framework, $f$-distill, accelerates diffusion model sampling by using $f$-divergence minimization, improving mode coverage and reducing training variance.
diffusion models variational score distillation reverse-Kullback-Leibler divergence $f$-divergence mode seeking density ratio score differences forward-Kullback-Leibler divergence Jensen-Shannon divergence ImageNet64 MS-COCO one-step generation performance zero-shot text-to-image generation $f$-distill