title: SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in
    LLMs
summary: We propose SPARC, a lightweight continual learning framework for large
  language models (LLMs) that enables efficient task adaptation through prompt
  tuning in a lower-dimensional space. By leveraging principal component analysis
  (PCA), we identify a compact subspace of the training data. Optimizing prompts
  in this lower-dimensional space enhances training efficiency, as it focuses
  updates on the most relevant features while reducing computational overhead.
  Furthermore, since the model's internal structure remains unaltered, the
  extensive knowledge gained from pretraining is fully preserved, ensuring that
  previously learned information is not compromised during adaptation. Our method
  achieves high knowledge retention in both task-incremental and
  domain-incremental continual learning setups while fine-tuning only 0.04% of
  the model's parameters. Additionally, by integrating LoRA, we enhance
  adaptability to computational constraints, allowing for a tradeoff between
  accuracy and training cost. Experiments on the SuperGLUE benchmark demonstrate
  that our PCA-based prompt tuning combined with LoRA maintains full knowledge
  retention while improving accuracy, utilizing only 1% of the model's
  parameters. These results establish our approach as a scalable and
  resource-efficient solution for continual learning in LLMs.
paper_authors._id: 67aa91fd5f845ebfe01d7769
paper_authors.name: Dinithi Jayasuriya
paper_authors._id: 67aa91fd5f845ebfe01d776a
paper_authors.name: Sina Tayebati
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:25:43.230Z
paper_authors._id: 655ec30b12fb73960ceb048f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/655ec30b12fb73960ceb048f/q7zVSStJWBywrtPoL2ChO.png
paper_authors.fullname: Sina Tayebati
paper_authors.type: user
paper_authors.user: sinatayebati
paper_authors._id: 67aa91fd5f845ebfe01d776b
paper_authors.name: Davide Ettori
paper_authors._id: 67aa91fd5f845ebfe01d776c
paper_authors.name: Ranganath Krishnan
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-10T23:55:43.302Z
paper_authors._id: 647a45aeccb84c6180b41b54
paper_authors.avatarUrl: /avatars/cd0db59a1b7f49f53f65751a8efc1033.svg
paper_authors.fullname: Ranganath Krishnan
paper_authors.type: user
paper_authors.user: ranganathkrishnan
paper_authors._id: 67aa91fd5f845ebfe01d776d
paper_authors.name: Amit Ranjan Trivedi
paper_title: SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in
    LLMs
paper_summary: We propose SPARC, a lightweight continual learning framework for large
  language models (LLMs) that enables efficient task adaptation through prompt
  tuning in a lower-dimensional space. By leveraging principal component analysis
  (PCA), we identify a compact subspace of the training data. Optimizing prompts
  in this lower-dimensional space enhances training efficiency, as it focuses
  updates on the most relevant features while reducing computational overhead.
  Furthermore, since the model's internal structure remains unaltered, the
  extensive knowledge gained from pretraining is fully preserved, ensuring that
  previously learned information is not compromised during adaptation. Our method
  achieves high knowledge retention in both task-incremental and
  domain-incremental continual learning setups while fine-tuning only 0.04% of
  the model's parameters. Additionally, by integrating LoRA, we enhance
  adaptability to computational constraints, allowing for a tradeoff between
  accuracy and training cost. Experiments on the SuperGLUE benchmark demonstrate
  that our PCA-based prompt tuning combined with LoRA maintains full knowledge
  retention while improving accuracy, utilizing only 1% of the model's
  parameters. These results establish our approach as a scalable and
  resource-efficient solution for continual learning in LLMs.
paper_ai_summary: A lightweight continual learning framework for LLMs uses PCA-based prompt tuning in a lower-dimensional space to enhance efficiency and retain knowledge while adapting to new tasks.
paper_ai_keywords: SPARC continual learning large language models LLMs prompt tuning principal component analysis PCA task-incremental domain-incremental fine-tuning LoRA knowledge retention SuperGLUE