Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translation
<|reference_start|>Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translation: Multilingual neural machine translation models support fine-tuning hundreds of languages simultaneously. However, fine-tuning on full parameters solely is inefficient potentially leading to negative interactions among languages. In this work, we demonstrate that the fine-tuning for a language occurs in its intrinsic language-specific subspace with a tiny fraction of entire parameters. Thus, we propose language-specific LoRA to isolate intrinsic language-specific subspaces. Furthermore, we propose architecture learning techniques and introduce a gradual pruning schedule during fine-tuning to exhaustively explore the optimal setting and the minimal intrinsic subspaces for each language, resulting in a lightweight yet effective fine-tuning procedure. The experimental results on a 12-language subset and a 30-language subset of FLORES-101 show that our methods not only outperform full-parameter fine-tuning up to 2.25 spBLEU scores but also reduce trainable parameters to $0.4\%$ for high and medium-resource languages and $1.6\%$ for low-resource ones.<|reference_end|>
@article{cao2024exploring,
    title={Exploring Intrinsic Language-specific Subspaces in Fine-tuning
  Multilingual Neural Machine Translation},
    author={Zhe Cao and Zhi Qu and Hidetaka Kamigaito and Taro Watanabe},
    journal={arXiv preprint arXiv:2409.05224},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.05224},
    primaryClass={cs.CL}
}