Fine-Tuning Linear Layers Only Is a Simple yet Effective Way for Task Arithmetic
<|reference_start|>Fine-Tuning Linear Layers Only Is a Simple yet Effective Way for Task Arithmetic: Task arithmetic has recently emerged as a cost-effective and scalable approach to edit pre-trained models directly in weight space, by adding the fine-tuned weights of different tasks. The performance has been further improved by a linear property which is illustrated by weight disentanglement. Yet, conventional linearization methods (e.g., NTK linearization) not only double the time and training cost but also have a disadvantage on single-task performance. We propose a simple yet effective and efficient method that only fine-tunes linear layers, which improves weight disentanglement and efficiency simultaneously. Specifically, our study reveals that only fine-tuning the linear layers in the attention modules makes the whole model occur in a linear regime, significantly improving weight disentanglement. To further understand how our method improves the disentanglement of task arithmetic, we present a comprehensive study of task arithmetic by differentiating the role of representation model and task-specific model. In particular, we find that the representation model plays an important role in improving weight disentanglement whereas the task-specific models such as the classification heads can degenerate the weight disentanglement performance. Overall, our work uncovers novel insights into the fundamental mechanisms of task arithmetic and offers a more reliable and effective approach to editing pre-trained models.<|reference_end|>
@article{jin2024fine-tuning,
    title={Fine-Tuning Linear Layers Only Is a Simple yet Effective Way for Task
  Arithmetic},
    author={Ruochen Jin, Bojian Hou, Jiancong Xiao, Weijie Su and Li Shen},
    journal={arXiv preprint arXiv:2407.07089},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.07089},
    primaryClass={cs.LG}
}