Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting
<|reference_start|>Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting: Diffusion models have significantly advanced the field of generative modeling. However, training a diffusion model is computationally expensive, creating a pressing need to adapt off-the-shelf diffusion models for downstream generation tasks. Current fine-tuning methods focus on parameter-efficient transfer learning but overlook the fundamental transfer characteristics of diffusion models. In this paper, we investigate the transferability of diffusion models and observe a monotonous chain of forgetting trend of transferability along the reverse process. Based on this observation and novel theoretical insights, we present Diff-Tuning, a frustratingly simple transfer approach that leverages the chain of forgetting tendency. Diff-Tuning encourages the fine-tuned model to retain the pre-trained knowledge at the end of the denoising chain close to the generated data while discarding the other noise side. We conduct comprehensive experiments to evaluate Diff-Tuning, including the transfer of pre-trained Diffusion Transformer models to eight downstream generations and the adaptation of Stable Diffusion to five control conditions with ControlNet. Diff-Tuning achieves a 26% improvement over standard fine-tuning and enhances the convergence speed of ControlNet by 24%. Notably, parameter-efficient transfer learning techniques for diffusion models can also benefit from Diff-Tuning.<|reference_end|>
@article{zhong2024diffusion,
    title={Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting},
    author={Jincheng Zhong, Xingzhuo Guo, Jiaxiang Dong, Mingsheng Long},
    journal={arXiv preprint arXiv:2406.00773},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.00773},
    primaryClass={cs.LG cs.CV}
}