title: LIFT the Veil for the Truth: Principal Weights Emerge after Rank
    Reduction for Reasoning-Focused Supervised Fine-Tuning
summary: Recent studies have shown that supervised fine-tuning of LLMs on a small
  number of high-quality datasets can yield strong reasoning capabilities.
  However, full fine-tuning (Full FT), while powerful, is computationally
  expensive and susceptible to overfitting and catastrophic forgetting,
  particularly when data is limited. Sparse fine-tuning, which previously
  achieved notable success by updating only a small subset of model parameters,
  offers a promising trade-off between efficiency and effectiveness. Yet, it has
  lagged behind in the LLM era due to the difficulty of identifying parameters
  truly critical for reasoning. In this work, we state that weights with the
  largest magnitude after low-rank approximation are critical weights for
  fine-tuning, which we call Principal Weights. Surprisingly, while
  magnitude-based sparse fine-tuning performs poorly as a baseline on LLM
  fine-tuning, it becomes highly effective after rank reduction. These insights
  motivate our method: Low-rank Informed Sparse Fine-Tuning (LIFT). LIFT only
  updates the top 5% Principal Weights throughout training and consistently
  achieves better performance on reasoning tasks than Full FT, while maintaining
  memory efficiency on par with popular parameter-efficient fine-tuning methods.
  In addition to strong performance on target domains such as arithmetic
  reasoning, LIFT also retains up to 20% more source-domain knowledge, compared
  to Full FT and LoRA. Our code is available at:
  https://github.com/zihanghliu/LIFT.
paper_authors._id: 683ec2d53c81cc903bbb418c
paper_authors.name: Zihang Liu
paper_authors._id: 683ec2d53c81cc903bbb418d
paper_authors.name: Tianyu Pang
paper_authors._id: 683ec2d53c81cc903bbb418e
paper_authors.name: Oleg Balabanov
paper_authors._id: 683ec2d53c81cc903bbb418f
paper_authors.name: Chaoqun Yang
paper_authors._id: 683ec2d53c81cc903bbb4190
paper_authors.name: Tianjin Huang
paper_authors._id: 683ec2d53c81cc903bbb4191
paper_authors.name: Lu Yin
paper_authors._id: 683ec2d53c81cc903bbb4192
paper_authors.name: Yaoqing Yang
paper_authors._id: 683ec2d53c81cc903bbb4193
paper_authors.name: Shiwei Liu
paper_title: LIFT the Veil for the Truth: Principal Weights Emerge after Rank
    Reduction for Reasoning-Focused Supervised Fine-Tuning
paper_summary: Recent studies have shown that supervised fine-tuning of LLMs on a small
  number of high-quality datasets can yield strong reasoning capabilities.
  However, full fine-tuning (Full FT), while powerful, is computationally
  expensive and susceptible to overfitting and catastrophic forgetting,
  particularly when data is limited. Sparse fine-tuning, which previously
  achieved notable success by updating only a small subset of model parameters,
  offers a promising trade-off between efficiency and effectiveness. Yet, it has
  lagged behind in the LLM era due to the difficulty of identifying parameters
  truly critical for reasoning. In this work, we state that weights with the
  largest magnitude after low-rank approximation are critical weights for
  fine-tuning, which we call Principal Weights. Surprisingly, while
  magnitude-based sparse fine-tuning performs poorly as a baseline on LLM
  fine-tuning, it becomes highly effective after rank reduction. These insights
  motivate our method: Low-rank Informed Sparse Fine-Tuning (LIFT). LIFT only
  updates the top 5% Principal Weights throughout training and consistently
  achieves better performance on reasoning tasks than Full FT, while maintaining
  memory efficiency on par with popular parameter-efficient fine-tuning methods.
  In addition to strong performance on target domains such as arithmetic
  reasoning, LIFT also retains up to 20% more source-domain knowledge, compared
  to Full FT and LoRA. Our code is available at:
  https://github.com/zihanghliu/LIFT.
paper_ai_summary: Leveraging low-rank approximation to identify critical weights for sparse fine-tuning of large language models enhances performance and efficiency compared to full fine-tuning.
paper_ai_keywords: LLMs supervised fine-tuning full fine-tuning sparse fine-tuning low-rank approximation Principal Weights Low-rank Informed Sparse Fine-Tuning LIFT memory efficiency parameter-efficient fine-tuning reasoning tasks arithmetic reasoning source-domain knowledge LoRA