SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse
  Low-Rank Adaptation
In recent years, the development of diffusion models has led to significant
progress in image and video generation tasks, with pre-trained models like the
Stable Diffusion series playing a crucial role. Inspired by model pruning which
lightens large pre-trained models by removing unimportant parameters, we
propose a novel model fine-tuning method to make full use of these ineffective
parameters and enable the pre-trained model with new task-specified
capabilities. In this work, we first investigate the importance of parameters
in pre-trained diffusion models, and discover that the smallest 10% to 20% of
parameters by absolute values do not contribute to the generation process.
Based on this observation, we propose a method termed SaRA that re-utilizes
these temporarily ineffective parameters, equating to optimizing a sparse
weight matrix to learn the task-specific knowledge. To mitigate overfitting, we
propose a nuclear-norm-based low-rank sparse training scheme for efficient
fine-tuning. Furthermore, we design a new progressive parameter adjustment
strategy to make full use of the re-trained/finetuned parameters. Finally, we
propose a novel unstructural backpropagation strategy, which significantly
reduces memory costs during fine-tuning. Our method enhances the generative
capabilities of pre-trained models in downstream applications and outperforms
traditional fine-tuning methods like LoRA in maintaining model's generalization
ability. We validate our approach through fine-tuning experiments on SD models,
demonstrating significant improvements. SaRA also offers a practical advantage
that requires only a single line of code modification for efficient
implementation and is seamlessly compatible with existing methods.
66e0fc05fb009ef59810c280
Teng Hu
66e0fc05fb009ef59810c281
Jiangning Zhang
66e0fc05fb009ef59810c282
Ran Yi
66e0fc05fb009ef59810c283
Hongrui Huang
admin_assigned
2024-09-11T10:48:54.042Z
664d79bf35e245d2ebe95540
/avatars/2087bfbcd82e5f43ff1cd32afbc0bc47.svg
hongrui huang
user
hongruihuang
66e0fc05fb009ef59810c284
Yabiao Wang
66e0fc05fb009ef59810c285
Lizhuang Ma
SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse
  Low-Rank Adaptation
In recent years, the development of diffusion models has led to significant
progress in image and video generation tasks, with pre-trained models like the
Stable Diffusion series playing a crucial role. Inspired by model pruning which
lightens large pre-trained models by removing unimportant parameters, we
propose a novel model fine-tuning method to make full use of these ineffective
parameters and enable the pre-trained model with new task-specified
capabilities. In this work, we first investigate the importance of parameters
in pre-trained diffusion models, and discover that the smallest 10% to 20% of
parameters by absolute values do not contribute to the generation process.
Based on this observation, we propose a method termed SaRA that re-utilizes
these temporarily ineffective parameters, equating to optimizing a sparse
weight matrix to learn the task-specific knowledge. To mitigate overfitting, we
propose a nuclear-norm-based low-rank sparse training scheme for efficient
fine-tuning. Furthermore, we design a new progressive parameter adjustment
strategy to make full use of the re-trained/finetuned parameters. Finally, we
propose a novel unstructural backpropagation strategy, which significantly
reduces memory costs during fine-tuning. Our method enhances the generative
capabilities of pre-trained models in downstream applications and outperforms
traditional fine-tuning methods like LoRA in maintaining model's generalization
ability. We validate our approach through fine-tuning experiments on SD models,
demonstrating significant improvements. SaRA also offers a practical advantage
that requires only a single line of code modification for efficient
implementation and is seamlessly compatible with existing methods.
A method called SaRA reutilizes ineffective parameters in pre-trained diffusion models to enhance fine-tuning, improving generative capabilities and memory efficiency.
diffusion models pre-trained models Stable Diffusion model pruning fine-tuning SaRA sparse weight matrix nuclear-norm-based low-rank sparse training progressive parameter adjustment unstructural backpropagation LoRA generalization ability