TinyFusion: Diffusion Transformers Learned Shallow
Diffusion Transformers have demonstrated remarkable capabilities in image
generation but often come with excessive parameterization, resulting in
considerable inference overhead in real-world applications. In this work, we
present TinyFusion, a depth pruning method designed to remove redundant layers
from diffusion transformers via end-to-end learning. The core principle of our
approach is to create a pruned model with high recoverability, allowing it to
regain strong performance after fine-tuning. To accomplish this, we introduce a
differentiable sampling technique to make pruning learnable, paired with a
co-optimized parameter to simulate future fine-tuning. While prior works focus
on minimizing loss or error after pruning, our method explicitly models and
optimizes the post-fine-tuning performance of pruned models. Experimental
results indicate that this learnable paradigm offers substantial benefits for
layer pruning of diffusion transformers, surpassing existing importance-based
and error-based methods. Additionally, TinyFusion exhibits strong
generalization across diverse architectures, such as DiTs, MARs, and SiTs.
Experiments with DiT-XL show that TinyFusion can craft a shallow diffusion
transformer at less than 7% of the pre-training cost, achieving a 2times
speedup with an FID score of 2.86, outperforming competitors with comparable
efficiency. Code is available at https://github.com/VainF/TinyFusion.
674e94d36e8d5661331c9344
Gongfan Fang
admin_assigned
2024-12-03T15:22:25.108Z
646a1939c37ca1e12308fe81
/avatars/752e9d86018e7d33ad8bcd741203fd86.svg
Gongfan Fang
user
Vinnnf
674e94d36e8d5661331c9345
Kunjun Li
claimed_verified
2025-05-26T08:44:50.033Z
6540ef0e733c1ce6a6fc989a
https://cdn-avatars.huggingface.co/v1/production/uploads/6540ef0e733c1ce6a6fc989a/lyDLbmJ-h4nUmkWZCvWtg.jpeg
Kunjun Li
user
stargazerx0
674e94d36e8d5661331c9346
Xinyin Ma
admin_assigned
2024-12-03T15:22:04.107Z
64396ebc21221ac7411852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/64396ebc21221ac7411852b3/SR0dC8N0bdj9tZFxYPpSf.jpeg
Xinyin Ma
user
horseee
674e94d36e8d5661331c9347
Xinchao Wang
admin_assigned
2024-12-03T15:21:57.737Z
63fc03a50aab060792ffef39
/avatars/9d5b1bb2a41928e08176b703935133ab.svg
Wangxinchao
user
wxcTest
TinyFusion: Diffusion Transformers Learned Shallow
Diffusion Transformers have demonstrated remarkable capabilities in image
generation but often come with excessive parameterization, resulting in
considerable inference overhead in real-world applications. In this work, we
present TinyFusion, a depth pruning method designed to remove redundant layers
from diffusion transformers via end-to-end learning. The core principle of our
approach is to create a pruned model with high recoverability, allowing it to
regain strong performance after fine-tuning. To accomplish this, we introduce a
differentiable sampling technique to make pruning learnable, paired with a
co-optimized parameter to simulate future fine-tuning. While prior works focus
on minimizing loss or error after pruning, our method explicitly models and
optimizes the post-fine-tuning performance of pruned models. Experimental
results indicate that this learnable paradigm offers substantial benefits for
layer pruning of diffusion transformers, surpassing existing importance-based
and error-based methods. Additionally, TinyFusion exhibits strong
generalization across diverse architectures, such as DiTs, MARs, and SiTs.
Experiments with DiT-XL show that TinyFusion can craft a shallow diffusion
transformer at less than 7% of the pre-training cost, achieving a 2times
speedup with an FID score of 2.86, outperforming competitors with comparable
efficiency. Code is available at https://github.com/VainF/TinyFusion.
TinyFusion, a depth pruning method using end-to-end learning, improves diffusion transformers' inference efficiency without significant performance loss after fine-tuning.
diffusion transformers depth pruning end-to-end learning differentiable sampling co-optimized parameter post-fine-tuning performance layer pruning DiTs MARs SiTs FID score pre-training cost inference speedup