title: Distribution Backtracking Builds A Faster Convergence Trajectory for
    One-step Diffusion Distillation
summary: Accelerating the sampling speed of diffusion models remains a significant
  challenge. Recent score distillation methods distill a heavy teacher model into
  an one-step student generator, which is optimized by calculating the difference
  between the two score functions on the samples generated by the student model.
  However, there is a score mismatch issue in the early stage of the distillation
  process, because existing methods mainly focus on using the endpoint of
  pre-trained diffusion models as teacher models, overlooking the importance of
  the convergence trajectory between the student generator and the teacher model.
  To address this issue, we extend the score distillation process by introducing
  the entire convergence trajectory of teacher models and propose Distribution
  Backtracking Distillation (DisBack) for distilling student generators. DisBask
  is composed of two stages: Degradation Recording and Distribution Backtracking.
  Degradation Recording is designed to obtain the convergence trajectory of
  teacher models, which records the degradation path from the trained teacher
  model to the untrained initial student generator. The degradation path
  implicitly represents the intermediate distributions of teacher models. Then
  Distribution Backtracking trains a student generator to backtrack the
  intermediate distributions for approximating the convergence trajectory of
  teacher models. Extensive experiments show that DisBack achieves faster and
  better convergence than the existing distillation method and accomplishes
  comparable generation performance. Notably, DisBack is easy to implement and
  can be generalized to existing distillation methods to boost performance. Our
  code is publicly available on https://github.com/SYZhang0805/DisBack.
paper_authors._id: 66cffb5bd58394b62bf24301
paper_authors.name: Shengyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T07:22:58.760Z
paper_authors._id: 63943c882b9483beb473ec25
paper_authors.avatarUrl: /avatars/abd2aae43e68c34770159c15a01c8297.svg
paper_authors.fullname: Shengyuan Zhang
paper_authors.type: user
paper_authors.user: SYZhang0805
paper_authors._id: 66cffb5bd58394b62bf24302
paper_authors.name: Ling Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T15:34:30.378Z
paper_authors._id: 64fde4e252e82dd432b74ce9
paper_authors.avatarUrl: /avatars/061a69d858b86d1600be916122cae7fc.svg
paper_authors.fullname: Ling Yang
paper_authors.type: user
paper_authors.user: Lingaaaaaaa
paper_authors._id: 66cffb5bd58394b62bf24303
paper_authors.name: Zejian Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T08:09:16.254Z
paper_authors._id: 63a06cebe648d425374f3ea3
paper_authors.avatarUrl: /avatars/2e15a761aac16bfdc3da46dd5eef1adf.svg
paper_authors.fullname: Zejian Li
paper_authors.type: user
paper_authors.user: hyllbd
paper_authors._id: 66cffb5bd58394b62bf24304
paper_authors.name: An Zhao
paper_authors._id: 66cffb5bd58394b62bf24305
paper_authors.name: Chenye Meng
paper_authors._id: 66cffb5bd58394b62bf24306
paper_authors.name: Changyuan Yang
paper_authors._id: 66cffb5bd58394b62bf24307
paper_authors.name: Guang Yang
paper_authors._id: 66cffb5bd58394b62bf24308
paper_authors.name: Zhiyuan Yang
paper_authors._id: 66cffb5bd58394b62bf24309
paper_authors.name: Lingyun Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T08:07:24.113Z
paper_authors._id: 662a5c0649e03000ed1adab4
paper_authors.avatarUrl: /avatars/961e4b9bf923923fcd871672d156f50c.svg
paper_authors.fullname: Lingyun Sun
paper_authors.type: user
paper_authors.user: slysun
paper_title: Distribution Backtracking Builds A Faster Convergence Trajectory for
    One-step Diffusion Distillation
paper_summary: Accelerating the sampling speed of diffusion models remains a significant
  challenge. Recent score distillation methods distill a heavy teacher model into
  an one-step student generator, which is optimized by calculating the difference
  between the two score functions on the samples generated by the student model.
  However, there is a score mismatch issue in the early stage of the distillation
  process, because existing methods mainly focus on using the endpoint of
  pre-trained diffusion models as teacher models, overlooking the importance of
  the convergence trajectory between the student generator and the teacher model.
  To address this issue, we extend the score distillation process by introducing
  the entire convergence trajectory of teacher models and propose Distribution
  Backtracking Distillation (DisBack) for distilling student generators. DisBask
  is composed of two stages: Degradation Recording and Distribution Backtracking.
  Degradation Recording is designed to obtain the convergence trajectory of
  teacher models, which records the degradation path from the trained teacher
  model to the untrained initial student generator. The degradation path
  implicitly represents the intermediate distributions of teacher models. Then
  Distribution Backtracking trains a student generator to backtrack the
  intermediate distributions for approximating the convergence trajectory of
  teacher models. Extensive experiments show that DisBack achieves faster and
  better convergence than the existing distillation method and accomplishes
  comparable generation performance. Notably, DisBack is easy to implement and
  can be generalized to existing distillation methods to boost performance. Our
  code is publicly available on https://github.com/SYZhang0805/DisBack.
paper_ai_keywords: diffusion models score distillation score function pre-trained diffusion models Distribution Backtracking Distance (DisBack) Degradation Recording Distribution Backtracking convergence trajectory intermediate distributions generation performance