title: Restart Sampling for Improving Generative Processes
summary: Generative processes that involve solving differential equations, such as
  diffusion models, frequently necessitate balancing speed and quality. ODE-based
  samplers are fast but plateau in performance while SDE-based samplers deliver
  higher sample quality at the cost of increased sampling time. We attribute this
  difference to sampling errors: ODE-samplers involve smaller discretization
  errors while stochasticity in SDE contracts accumulated errors. Based on these
  findings, we propose a novel sampling algorithm called Restart in order to
  better balance discretization errors and contraction. The sampling method
  alternates between adding substantial noise in additional forward steps and
  strictly following a backward ODE. Empirically, Restart sampler surpasses
  previous SDE and ODE samplers in both speed and accuracy. Restart not only
  outperforms the previous best SDE results, but also accelerates the sampling
  speed by 10-fold / 2-fold on CIFAR-10 / ImageNet 64 times 64. In addition,
  it attains significantly better sample quality than ODE samplers within
  comparable sampling times. Moreover, Restart better balances text-image
  alignment/visual quality versus diversity than previous samplers in the
  large-scale text-to-image Stable Diffusion model pre-trained on LAION 512
  times 512. Code is available at
  https://github.com/Newbeeer/diffusion_restart_sampling
paper_authors._id: 649a7e617a119fd611f7d9c7
paper_authors.name: Yilun Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:15:10.199Z
paper_authors._id: 649c37de5ffe05267a105fe8
paper_authors.avatarUrl: /avatars/4f262272e6222f879c6c0fedfa2e5861.svg
paper_authors.fullname: Yilun Xu
paper_authors.type: user
paper_authors.user: AaronXyl
paper_authors._id: 649a7e617a119fd611f7d9c8
paper_authors.name: Mingyang Deng
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-06-27T06:15:02.374Z
paper_authors._id: 6449d8fa86e837e3d58718a7
paper_authors.avatarUrl: /avatars/fb6ef807151d250105222b401f6bcb09.svg
paper_authors.fullname: Mingyang Deng
paper_authors.type: user
paper_authors.user: Goodeat
paper_authors._id: 649a7e617a119fd611f7d9c9
paper_authors.name: Xiang Cheng
paper_authors._id: 649a7e617a119fd611f7d9ca
paper_authors.name: Yonglong Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:15:13.359Z
paper_authors._id: 64793086ac77bb7603842456
paper_authors.avatarUrl: /avatars/5891292138d372a9cc063f86c08b8174.svg
paper_authors.fullname: Yonglong Tian
paper_authors.type: user
paper_authors.user: tyl5566
paper_authors._id: 649a7e617a119fd611f7d9cb
paper_authors.name: Ziming Liu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-08-11T02:29:32.621Z
paper_authors._id: 64d59cfa8767727dffa382de
paper_authors.avatarUrl: /avatars/21611131f5f9281ff9eb954939449124.svg
paper_authors.fullname: Ziming Liu
paper_authors.type: user
paper_authors.user: zmliu
paper_authors._id: 649a7e617a119fd611f7d9cc
paper_authors.name: Tommi Jaakkola
paper_title: Restart Sampling for Improving Generative Processes
paper_summary: Generative processes that involve solving differential equations, such as
  diffusion models, frequently necessitate balancing speed and quality. ODE-based
  samplers are fast but plateau in performance while SDE-based samplers deliver
  higher sample quality at the cost of increased sampling time. We attribute this
  difference to sampling errors: ODE-samplers involve smaller discretization
  errors while stochasticity in SDE contracts accumulated errors. Based on these
  findings, we propose a novel sampling algorithm called Restart in order to
  better balance discretization errors and contraction. The sampling method
  alternates between adding substantial noise in additional forward steps and
  strictly following a backward ODE. Empirically, Restart sampler surpasses
  previous SDE and ODE samplers in both speed and accuracy. Restart not only
  outperforms the previous best SDE results, but also accelerates the sampling
  speed by 10-fold / 2-fold on CIFAR-10 / ImageNet 64 times 64. In addition,
  it attains significantly better sample quality than ODE samplers within
  comparable sampling times. Moreover, Restart better balances text-image
  alignment/visual quality versus diversity than previous samplers in the
  large-scale text-to-image Stable Diffusion model pre-trained on LAION 512
  times 512. Code is available at
  https://github.com/Newbeeer/diffusion_restart_sampling
paper_ai_keywords: diffusion models ODE-based samplers SDE-based samplers sampling errors discretization errors stochasticity Restart sampler forward steps backward ODE sample quality sampling speed stiffness sample diversity text-to-image Stable Diffusion model LAION