Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models
<|reference_start|>Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models: Adapting large-scale pre-trained generative models in a parameter-efficient manner is gaining traction. Traditional methods like low rank adaptation achieve parameter efficiency by imposing constraints but may not be optimal for tasks requiring high representation capacity. We propose a novel spectrum-aware adaptation framework for generative models. Our method adjusts both singular values and their basis vectors of pretrained weights. Using the Kronecker product and efficient Stiefel optimizers, we achieve parameter-efficient adaptation of orthogonal matrices. We introduce Spectral Orthogonal Decomposition Adaptation (SODA), which balances computational efficiency and representation capacity. Extensive evaluations on text-to-image diffusion models demonstrate SODA's effectiveness, offering a spectrum-aware alternative to existing fine-tuning methods.<|reference_end|>
@article{zhang2024spectrum-aware,
    title={Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models},
    author={Xinxi Zhang, Song Wen, Ligong Han, Felix Juefei-Xu, Akash Srivastava,
  Junzhou Huang, Hao Wang, Molei Tao, Dimitris N. Metaxas},
    journal={arXiv preprint arXiv:2405.21050},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.21050},
    primaryClass={cs.CV cs.LG}
}