title: Make a Cheap Scaling: A Self-Cascade Diffusion Model for
    Higher-Resolution Adaptation
summary: Diffusion models have proven to be highly effective in image and video
  generation; however, they still face composition challenges when generating
  images of varying sizes due to single-scale training data. Adapting large
  pre-trained diffusion models for higher resolution demands substantial
  computational and optimization resources, yet achieving a generation capability
  comparable to low-resolution models remains elusive. This paper proposes a
  novel self-cascade diffusion model that leverages the rich knowledge gained
  from a well-trained low-resolution model for rapid adaptation to
  higher-resolution image and video generation, employing either tuning-free or
  cheap upsampler tuning paradigms. Integrating a sequence of multi-scale
  upsampler modules, the self-cascade diffusion model can efficiently adapt to a
  higher resolution, preserving the original composition and generation
  capabilities. We further propose a pivot-guided noise re-schedule strategy to
  speed up the inference process and improve local structural details. Compared
  to full fine-tuning, our approach achieves a 5X training speed-up and requires
  only an additional 0.002M tuning parameters. Extensive experiments demonstrate
  that our approach can quickly adapt to higher resolution image and video
  synthesis by fine-tuning for just 10k steps, with virtually no additional
  inference time.
paper_authors._id: 65d2c60b71ca8e594f7a0eac
paper_authors.name: Lanqing Guo
paper_authors._id: 65d2c60b71ca8e594f7a0ead
paper_authors.name: Yingqing He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:29:50.604Z
paper_authors._id: 630231bf7e137e3d6b3b0645
paper_authors.avatarUrl: /avatars/dad52f8955110f0a2caeb613d6aa3ea2.svg
paper_authors.fullname: He
paper_authors.type: user
paper_authors.user: Yingqing
paper_authors._id: 65d2c60b71ca8e594f7a0eae
paper_authors.name: Haoxin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:30:19.045Z
paper_authors._id: 6305cb2f435ec751b723d80a
paper_authors.avatarUrl: /avatars/5e8519d9814038726bf98f4104f74568.svg
paper_authors.fullname: Haoxin Chen
paper_authors.type: user
paper_authors.user: paulai
paper_authors._id: 65d2c60b71ca8e594f7a0eaf
paper_authors.name: Menghan Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:30:36.830Z
paper_authors._id: 63401c89f81b9d101361f712
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
paper_authors.fullname: Richard
paper_authors.type: user
paper_authors.user: menghanxia
paper_authors._id: 65d2c60b71ca8e594f7a0eb0
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:30:49.206Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 65d2c60b71ca8e594f7a0eb1
paper_authors.name: Yufei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:31:55.148Z
paper_authors._id: 64a48f758b961fd6fa385737
paper_authors.avatarUrl: /avatars/328fdf1a4dca5f2bafbedf1d4af83f7c.svg
paper_authors.fullname: Yufei Wang
paper_authors.type: user
paper_authors.user: lomo-1
paper_authors._id: 65d2c60b71ca8e594f7a0eb2
paper_authors.name: Siyu Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:32:02.611Z
paper_authors._id: 647060e37fd7ecdbd0e982cd
paper_authors.avatarUrl: /avatars/9bb1b61182f837e45722931695c0b8d0.svg
paper_authors.fullname: siyu huang
paper_authors.type: user
paper_authors.user: hsyyy
paper_authors._id: 65d2c60b71ca8e594f7a0eb3
paper_authors.name: Yong Zhang
paper_authors._id: 65d2c60b71ca8e594f7a0eb4
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:32:41.481Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 65d2c60b71ca8e594f7a0eb5
paper_authors.name: Qifeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:32:48.176Z
paper_authors._id: 6467b121e7a6a374fd19b44b
paper_authors.avatarUrl: /avatars/3f2874d58986d651aef55e3408b05700.svg
paper_authors.fullname: Qifeng Chen
paper_authors.type: user
paper_authors.user: cqf
paper_authors._id: 65d2c60b71ca8e594f7a0eb6
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T11:33:09.709Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_authors._id: 65d2c60b71ca8e594f7a0eb7
paper_authors.name: Bihan Wen
paper_title: Make a Cheap Scaling: A Self-Cascade Diffusion Model for
    Higher-Resolution Adaptation
paper_summary: Diffusion models have proven to be highly effective in image and video
  generation; however, they still face composition challenges when generating
  images of varying sizes due to single-scale training data. Adapting large
  pre-trained diffusion models for higher resolution demands substantial
  computational and optimization resources, yet achieving a generation capability
  comparable to low-resolution models remains elusive. This paper proposes a
  novel self-cascade diffusion model that leverages the rich knowledge gained
  from a well-trained low-resolution model for rapid adaptation to
  higher-resolution image and video generation, employing either tuning-free or
  cheap upsampler tuning paradigms. Integrating a sequence of multi-scale
  upsampler modules, the self-cascade diffusion model can efficiently adapt to a
  higher resolution, preserving the original composition and generation
  capabilities. We further propose a pivot-guided noise re-schedule strategy to
  speed up the inference process and improve local structural details. Compared
  to full fine-tuning, our approach achieves a 5X training speed-up and requires
  only an additional 0.002M tuning parameters. Extensive experiments demonstrate
  that our approach can quickly adapt to higher resolution image and video
  synthesis by fine-tuning for just 10k steps, with virtually no additional
  inference time.
paper_ai_keywords: self-cascade diffusion model rich knowledge low-resolution model single-scale training data multi-scale upsample upsampler modules pivot-guided noise reschedule strategy full fine-tuning tuning parameters