SparseDM: Toward Sparse Efficient Diffusion Models
<|reference_start|>SparseDM: Toward Sparse Efficient Diffusion Models: Diffusion models have been extensively used in data generation tasks and are recognized as one of the best generative models. However, their time-consuming deployment, long inference time, and requirements on large memory limit their application on mobile devices. In this paper, we propose a method based on the improved Straight-Through Estimator to improve the deployment efficiency of diffusion models. Specifically, we add sparse masks to the Convolution and Linear layers in a pre-trained diffusion model, then use design progressive sparsity for model training in the fine-tuning stage, and switch the inference mask on and off, which supports a flexible choice of sparsity during inference according to the FID and MACs requirements. Experiments on four datasets conducted on a state-of-the-art Transformer-based diffusion model demonstrate that our method reduces MACs by $50\%$ while increasing FID by only 1.5 on average. Under other MACs conditions, the FID is also lower than 1$\sim$137 compared to other methods.<|reference_end|>
@article{wang2024sparsedm:,
    title={SparseDM: Toward Sparse Efficient Diffusion Models},
    author={Kafeng Wang, Jianfei Chen, He Li, Zhenpeng Mi, Jun Zhu},
    journal={arXiv preprint arXiv:2404.10445},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.10445},
    primaryClass={cs.LG cs.AI}
}