Restart Sampling for Improving Generative Processes
Generative processes that involve solving differential equations, such as
diffusion models, frequently necessitate balancing speed and quality. ODE-based
samplers are fast but plateau in performance while SDE-based samplers deliver
higher sample quality at the cost of increased sampling time. We attribute this
difference to sampling errors: ODE-samplers involve smaller discretization
errors while stochasticity in SDE contracts accumulated errors. Based on these
findings, we propose a novel sampling algorithm called Restart in order to
better balance discretization errors and contraction. The sampling method
alternates between adding substantial noise in additional forward steps and
strictly following a backward ODE. Empirically, Restart sampler surpasses
previous SDE and ODE samplers in both speed and accuracy. Restart not only
outperforms the previous best SDE results, but also accelerates the sampling
speed by 10-fold / 2-fold on CIFAR-10 / ImageNet 64 times 64. In addition,
it attains significantly better sample quality than ODE samplers within
comparable sampling times. Moreover, Restart better balances text-image
alignment/visual quality versus diversity than previous samplers in the
large-scale text-to-image Stable Diffusion model pre-trained on LAION 512
times 512. Code is available at
https://github.com/Newbeeer/diffusion_restart_sampling
649a7e617a119fd611f7d9c7
Yilun Xu
claimed_verified
2023-06-28T20:15:10.199Z
649c37de5ffe05267a105fe8
/avatars/4f262272e6222f879c6c0fedfa2e5861.svg
Yilun Xu
user
AaronXyl
649a7e617a119fd611f7d9c8
Mingyang Deng
extracted_pending
2023-06-27T06:15:02.374Z
6449d8fa86e837e3d58718a7
/avatars/fb6ef807151d250105222b401f6bcb09.svg
Mingyang Deng
user
Goodeat
649a7e617a119fd611f7d9c9
Xiang Cheng
649a7e617a119fd611f7d9ca
Yonglong Tian
claimed_verified
2023-06-28T20:15:13.359Z
64793086ac77bb7603842456
/avatars/5891292138d372a9cc063f86c08b8174.svg
Yonglong Tian
user
tyl5566
649a7e617a119fd611f7d9cb
Ziming Liu
extracted_pending
2023-08-11T02:29:32.621Z
64d59cfa8767727dffa382de
/avatars/21611131f5f9281ff9eb954939449124.svg
Ziming Liu
user
zmliu
649a7e617a119fd611f7d9cc
Tommi Jaakkola
Restart Sampling for Improving Generative Processes
Generative processes that involve solving differential equations, such as
diffusion models, frequently necessitate balancing speed and quality. ODE-based
samplers are fast but plateau in performance while SDE-based samplers deliver
higher sample quality at the cost of increased sampling time. We attribute this
difference to sampling errors: ODE-samplers involve smaller discretization
errors while stochasticity in SDE contracts accumulated errors. Based on these
findings, we propose a novel sampling algorithm called Restart in order to
better balance discretization errors and contraction. The sampling method
alternates between adding substantial noise in additional forward steps and
strictly following a backward ODE. Empirically, Restart sampler surpasses
previous SDE and ODE samplers in both speed and accuracy. Restart not only
outperforms the previous best SDE results, but also accelerates the sampling
speed by 10-fold / 2-fold on CIFAR-10 / ImageNet 64 times 64. In addition,
it attains significantly better sample quality than ODE samplers within
comparable sampling times. Moreover, Restart better balances text-image
alignment/visual quality versus diversity than previous samplers in the
large-scale text-to-image Stable Diffusion model pre-trained on LAION 512
times 512. Code is available at
https://github.com/Newbeeer/diffusion_restart_sampling
A new Restart sampling algorithm enhances both speed and quality for generative processes involving differential equations, outperforming previous ODE and SDE samplers in various tasks.
diffusion models ODE-based samplers SDE-based samplers discretization errors sampling errors stochasticity Restart sampler CIFAR-10 ImageNet large-scale text-to-image Stable Diffusion model LAION