title: TinyFusion: Diffusion Transformers Learned Shallow
summary: Diffusion Transformers have demonstrated remarkable capabilities in image
  generation but often come with excessive parameterization, resulting in
  considerable inference overhead in real-world applications. In this work, we
  present TinyFusion, a depth pruning method designed to remove redundant layers
  from diffusion transformers via end-to-end learning. The core principle of our
  approach is to create a pruned model with high recoverability, allowing it to
  regain strong performance after fine-tuning. To accomplish this, we introduce a
  differentiable sampling technique to make pruning learnable, paired with a
  co-optimized parameter to simulate future fine-tuning. While prior works focus
  on minimizing loss or error after pruning, our method explicitly models and
  optimizes the post-fine-tuning performance of pruned models. Experimental
  results indicate that this learnable paradigm offers substantial benefits for
  layer pruning of diffusion transformers, surpassing existing importance-based
  and error-based methods. Additionally, TinyFusion exhibits strong
  generalization across diverse architectures, such as DiTs, MARs, and SiTs.
  Experiments with DiT-XL show that TinyFusion can craft a shallow diffusion
  transformer at less than 7% of the pre-training cost, achieving a 2times
  speedup with an FID score of 2.86, outperforming competitors with comparable
  efficiency. Code is available at https://github.com/VainF/TinyFusion.
paper_authors._id: 674e94d36e8d5661331c9344
paper_authors.name: Gongfan Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:22:25.108Z
paper_authors._id: 646a1939c37ca1e12308fe81
paper_authors.avatarUrl: /avatars/752e9d86018e7d33ad8bcd741203fd86.svg
paper_authors.fullname: Gongfan Fang
paper_authors.type: user
paper_authors.user: Vinnnf
paper_authors._id: 674e94d36e8d5661331c9345
paper_authors.name: Kunjun Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:44:50.033Z
paper_authors._id: 6540ef0e733c1ce6a6fc989a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6540ef0e733c1ce6a6fc989a/lyDLbmJ-h4nUmkWZCvWtg.jpeg
paper_authors.fullname: Kunjun Li
paper_authors.type: user
paper_authors.user: stargazerx0
paper_authors._id: 674e94d36e8d5661331c9346
paper_authors.name: Xinyin Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:22:04.107Z
paper_authors._id: 64396ebc21221ac7411852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64396ebc21221ac7411852b3/SR0dC8N0bdj9tZFxYPpSf.jpeg
paper_authors.fullname: Xinyin Ma
paper_authors.type: user
paper_authors.user: horseee
paper_authors._id: 674e94d36e8d5661331c9347
paper_authors.name: Xinchao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:21:57.737Z
paper_authors._id: 63fc03a50aab060792ffef39
paper_authors.avatarUrl: /avatars/9d5b1bb2a41928e08176b703935133ab.svg
paper_authors.fullname: Wangxinchao
paper_authors.type: user
paper_authors.user: wxcTest
paper_title: TinyFusion: Diffusion Transformers Learned Shallow
paper_summary: Diffusion Transformers have demonstrated remarkable capabilities in image
  generation but often come with excessive parameterization, resulting in
  considerable inference overhead in real-world applications. In this work, we
  present TinyFusion, a depth pruning method designed to remove redundant layers
  from diffusion transformers via end-to-end learning. The core principle of our
  approach is to create a pruned model with high recoverability, allowing it to
  regain strong performance after fine-tuning. To accomplish this, we introduce a
  differentiable sampling technique to make pruning learnable, paired with a
  co-optimized parameter to simulate future fine-tuning. While prior works focus
  on minimizing loss or error after pruning, our method explicitly models and
  optimizes the post-fine-tuning performance of pruned models. Experimental
  results indicate that this learnable paradigm offers substantial benefits for
  layer pruning of diffusion transformers, surpassing existing importance-based
  and error-based methods. Additionally, TinyFusion exhibits strong
  generalization across diverse architectures, such as DiTs, MARs, and SiTs.
  Experiments with DiT-XL show that TinyFusion can craft a shallow diffusion
  transformer at less than 7% of the pre-training cost, achieving a 2times
  speedup with an FID score of 2.86, outperforming competitors with comparable
  efficiency. Code is available at https://github.com/VainF/TinyFusion.
paper_ai_summary: TinyFusion, a depth pruning method using end-to-end learning, improves diffusion transformers' inference efficiency without significant performance loss after fine-tuning.
paper_ai_keywords: diffusion transformers depth pruning end-to-end learning differentiable sampling co-optimized parameter post-fine-tuning performance layer pruning DiTs MARs SiTs FID score pre-training cost inference speedup