title: SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse
    Low-Rank Adaptation
summary: In recent years, the development of diffusion models has led to significant
  progress in image and video generation tasks, with pre-trained models like the
  Stable Diffusion series playing a crucial role. Inspired by model pruning which
  lightens large pre-trained models by removing unimportant parameters, we
  propose a novel model fine-tuning method to make full use of these ineffective
  parameters and enable the pre-trained model with new task-specified
  capabilities. In this work, we first investigate the importance of parameters
  in pre-trained diffusion models, and discover that the smallest 10% to 20% of
  parameters by absolute values do not contribute to the generation process.
  Based on this observation, we propose a method termed SaRA that re-utilizes
  these temporarily ineffective parameters, equating to optimizing a sparse
  weight matrix to learn the task-specific knowledge. To mitigate overfitting, we
  propose a nuclear-norm-based low-rank sparse training scheme for efficient
  fine-tuning. Furthermore, we design a new progressive parameter adjustment
  strategy to make full use of the re-trained/finetuned parameters. Finally, we
  propose a novel unstructural backpropagation strategy, which significantly
  reduces memory costs during fine-tuning. Our method enhances the generative
  capabilities of pre-trained models in downstream applications and outperforms
  traditional fine-tuning methods like LoRA in maintaining model's generalization
  ability. We validate our approach through fine-tuning experiments on SD models,
  demonstrating significant improvements. SaRA also offers a practical advantage
  that requires only a single line of code modification for efficient
  implementation and is seamlessly compatible with existing methods.
paper_authors._id: 66e0fc05fb009ef59810c280
paper_authors.name: Teng Hu
paper_authors._id: 66e0fc05fb009ef59810c281
paper_authors.name: Jiangning Zhang
paper_authors._id: 66e0fc05fb009ef59810c282
paper_authors.name: Ran Yi
paper_authors._id: 66e0fc05fb009ef59810c283
paper_authors.name: Hongrui Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-11T10:48:54.042Z
paper_authors._id: 664d79bf35e245d2ebe95540
paper_authors.avatarUrl: /avatars/2087bfbcd82e5f43ff1cd32afbc0bc47.svg
paper_authors.fullname: hongrui huang
paper_authors.type: user
paper_authors.user: hongruihuang
paper_authors._id: 66e0fc05fb009ef59810c284
paper_authors.name: Yabiao Wang
paper_authors._id: 66e0fc05fb009ef59810c285
paper_authors.name: Lizhuang Ma
paper_title: SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse
    Low-Rank Adaptation
paper_summary: In recent years, the development of diffusion models has led to significant
  progress in image and video generation tasks, with pre-trained models like the
  Stable Diffusion series playing a crucial role. Inspired by model pruning which
  lightens large pre-trained models by removing unimportant parameters, we
  propose a novel model fine-tuning method to make full use of these ineffective
  parameters and enable the pre-trained model with new task-specified
  capabilities. In this work, we first investigate the importance of parameters
  in pre-trained diffusion models, and discover that the smallest 10% to 20% of
  parameters by absolute values do not contribute to the generation process.
  Based on this observation, we propose a method termed SaRA that re-utilizes
  these temporarily ineffective parameters, equating to optimizing a sparse
  weight matrix to learn the task-specific knowledge. To mitigate overfitting, we
  propose a nuclear-norm-based low-rank sparse training scheme for efficient
  fine-tuning. Furthermore, we design a new progressive parameter adjustment
  strategy to make full use of the re-trained/finetuned parameters. Finally, we
  propose a novel unstructural backpropagation strategy, which significantly
  reduces memory costs during fine-tuning. Our method enhances the generative
  capabilities of pre-trained models in downstream applications and outperforms
  traditional fine-tuning methods like LoRA in maintaining model's generalization
  ability. We validate our approach through fine-tuning experiments on SD models,
  demonstrating significant improvements. SaRA also offers a practical advantage
  that requires only a single line of code modification for efficient
  implementation and is seamlessly compatible with existing methods.
paper_ai_summary: A method called SaRA reutilizes ineffective parameters in pre-trained diffusion models to enhance fine-tuning, improving generative capabilities and memory efficiency.
paper_ai_keywords: diffusion models pre-trained models Stable Diffusion model pruning fine-tuning SaRA sparse weight matrix nuclear-norm-based low-rank sparse training progressive parameter adjustment unstructural backpropagation LoRA generalization ability