title: CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting
    Mitigation
summary: This paper introduces CURLoRA, a novel approach to fine-tuning large language
  models (LLMs) that leverages CUR matrix decomposition in the context of
  Low-Rank Adaptation (LoRA). Our method addresses two critical challenges in LLM
  fine-tuning: mitigating catastrophic forgetting during continual learning and
  reducing the number of trainable parameters. We propose a unique modification
  to the CUR decomposition process, utilizing inverted probabilities for column
  and row selection which acts as an implicit regularization, and initializing
  the U matrix as a zero matrix, and only fine-tuning it. We demonstrate
  through experiments on multiple datasets that CURLoRA outperforms standard LoRA
  in mitigating catastrophic forgetting. It maintains model stability and
  performance across tasks while significantly reducing the number of trainable
  parameters. Our results show that CURLoRA achieves very good and stable task
  accuracy while maintaining base model's perplexity scores fixed compared to
  LoRA upon continual fine-tuning, particularly in scenarios with limited data.
paper_authors._id: 66d104cab66342762f60228a
paper_authors.name: Muhammad Fawi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-30T07:18:07.049Z
paper_authors._id: 6629743aee8fc0fa81e184b7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6629743aee8fc0fa81e184b7/pgjTipnBVt0uo5kwQzddx.png
paper_authors.fullname: Muhammad Fawi
paper_authors.type: user
paper_authors.user: mnoorfawi
paper_title: CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting
    Mitigation
paper_summary: This paper introduces CURLoRA, a novel approach to fine-tuning large language
  models (LLMs) that leverages CUR matrix decomposition in the context of
  Low-Rank Adaptation (LoRA). Our method addresses two critical challenges in LLM
  fine-tuning: mitigating catastrophic forgetting during continual learning and
  reducing the number of trainable parameters. We propose a unique modification
  to the CUR decomposition process, utilizing inverted probabilities for column
  and row selection which acts as an implicit regularization, and initializing
  the U matrix as a zero matrix, and only fine-tuning it. We demonstrate
  through experiments on multiple datasets that CURLoRA outperforms standard LoRA
  in mitigating catastrophic forgetting. It maintains model stability and
  performance across tasks while significantly reducing the number of trainable
  parameters. Our results show that CURLoRA achieves very good and stable task
  accuracy while maintaining base model's perplexity scores fixed compared to
  LoRA upon continual fine-tuning, particularly in scenarios with limited data.
paper_ai_keywords: CURLoRA CUR matrix decomposition Low-Rank Adaptation (LoRA) catastrophic forgetting continual learning inverted probabilities column and row selection implicit regularization $U$ matrix perplexity scores